Dovecot, PHP5, SA, and Postfix Upgrades; Planned Service Changes

Dovecot (IMAP/POP3 service), PHP5, SpamAssassin, and Postfix (mail delivery service) will be upgraded on all servers between the window of 1 AM – 3 AM EDT (-0400 GMT) on Sunday, September 2nd. Each server should take no longer than 20 minutes to finish the upgrade. During this window services may be intermittently unavailable for 2 – 3 minutes as old services are shutdown and replaced.

• PHP5 will be upgraded to 5.2.4 to address several high impact vulnerabilities
• Postfix is a performance upgrade from 2.3.x to 2.4.5. Several enhancements have been introduced between version changes including epoll support and better queue management both of which are geared at improving efficiency of message delivery.
• SpamAssassin is a general upgrade from 3.2.0 to 3.2.3.
• Dovecot is a general upgrade from 1.0 to 1.0.3. Likely this will be the last upgrade to Dovecot before 1.1 is released with major enhancements in scalability of large mailboxes.

Further, mod_fcgid is going through initial testing on the experimental server to assess impact on replacing mod_fastcgi with mod_fcgid. mod_fcgid improves on process management strategies that mod_fastcgi is sorely lacking in, specifically with restarting failed processes and avoiding orphans, both of which are major problems with mod_fastcgi. No ETA is available yet on when mod_fcgid will replace mod_fastcgi for serving FastCGI content or whether it will, but at this point I can confirm that alternatives to solve the two aforementioned problems are being researched. While not a noticeable problem for users, it is a burden to manage under the hood.

Secondly, another issue is with mitigating script exploitation. Two separate, custom-made scripts on different servers have been exploited this week with the sole intent of proliferating spam. This is a another large problem that is a time waster to manage for customers. First you have to determine the domain, then ascertain the exact file exploited through careful log inspection. Once that is determined, then stepping through the logic of why it happened and writing up an explanation e-mail of how it happened to the customer follows. Of course at the same time damage control becomes an issue, so there’s a bit of legwork involved with mitigating the effects of spam.

One possible solution would be per-sender quotas through a filter in Postfix. All known cases of spam have begun with local submissions through the sendmail binary (most notably the mail() function in PHP), so if quotas are considered, then it would affect mailing lists and e-mail generated from your Web site. Outbound e-mail sent through Outlook/Thunderbird, e-mail aliases, and forwarding wouldn’t be affected by the hypothetical quotas. Messages would be throttled with say 100 per hour per user threshold with the excess slipping back into the queue for later delivery and an alert filed to notify us that something suspicious might be up.

This idea is just on the white board. Whether it comes to fruition is uncertain, but steps need to be taken to avoid or minimize the damages incurred from sloppy coding.

1:48 AM EDT: upgrade window is now closed.  All servers have been upgraded at this time.