Layered Filesystems
Accounts now reside upon stacks of folders (”layers”); each layer beyond the first, which holds user files, provides a collection of system files.  Our new layout provides a very efficient method of adding new software to accounts by injecting files into a particular layer, which propagate to all inheritable accounts.  Further, by residing on the top-most layer, instead of selectively backing up /home and /var/www, we have a comprehensive method of tracking which files to backup for clients.

Layered filesystems decrease the amount of storage required to populate an account, and allow a true 1:1 replication of system files.  The old system, a Replicated Filesystem, utilized hardlinks, which are references to files, to populate system files.  This posed several problems:

1. very inefficient to create a reference for 50,000 files to each site
2. filesystem limitations prevent creation of more than 32,768 hardlinks to a file introducing scalability concerns.  This limit prevented clients access to all locales, severely hindering application capabilities
3. adding new software is quite cumbersome, which requires removing all references to the file (thankfully scripts are used to automate this exhaustive task), then copying the application to the filesystem template, then creating a reference for each file on each account (yikes!).  Any references that remain of the old file still count as occupied storage.
4. servers accumulate wasted storage over time from old software references resident on the server and by storage occupied from duplication of folders and file references (17 MB vs 1.3 MB baseline)

Control Groups
Control groups (”cgroups“) provide an accurate and fair means of accounting for memory and CPU usage generated by an account.  cgroups aggregate memory and CPU generated by applications grouped by account, which means for clients, we can permit any software to be run on an account up to x MB or y CPU seconds.  CouchDB?  MongoDB?  MySQL 5.5? (might want to read the next note!) Jetty?  You name it!  If it fits within the container limits, have fun!  Even better – unlike a VPS, crucial system processes do not count against your limits – only applications that you explicitly start.

Software Upgrades
Along with the latest applications provided by RHEL, additional software upgrades were selected:

• Percona MySQL 5.1  – clients now have privileges to create and run stored procedures/triggers.  InnoDB performance is greatly improved, on par with MySQL 5.5
• Mercurial + Bazaar- two revision control tools clients have asked for in the past
• mod_rails 2.2.15 – Rails and Rack functionality tested and working 100%
• Horde Groupware – enhanced webmail now includes a calendar with memo and task-creation capabilities
• Vim 7.2 – support for tabs
• Pyzor – additional spam filter, which will enhance filtering capabilities

Suexec Changes
Clients bemoaned a super-restrictive suexec wrapper that, in all honesty, was a little too restrictive:  (1) FastCGI/CGI applications could not be executed inside /home/user, but could inside /home/user/public_html; (2) 775 parent directory permissions would cause a failure regardless of who owns the folder complicating Rails dispatcher deployment (you should be using mod_rails now!); (3) best of all, suexec logs, which contained invaluable clues why a script was denied execution, were never logged to your account!  All three of these problems have been resolved with a new suexec wrapper and those error messages?  Check /var/log/httpd/suexec.log

Want to be a part of Apollo?  Open up a ticket beginning Monday for a server migration.  This process takes approximately 24 hours for a preparatory phase, followed by the actual move.  Moves take less than 5 minutes to complete and DNS is updated automatically.

Thank you everyone for your feedback,
Matt

2:44 AM Update: maintenance window is now closed.  RAID is rebuilding and should be completed by the morning (ETA ~5 hours)

Existing clients who would like to utilize Mercurial, Bazaar, or MySQL 5.1 (trigger/stored procedure support) can be migrated to the interim server beginning in July.  Until then, let me know of any other applications you would like to see.  “Apollo” will be deployed in 2 weeks.

- Matt (discussion)

New feature to permit/deny control panel access for users.

• NEW: IMAP/POP3 bandwidth logging
• NEW: control panel permit/deny (Manage Users, User Defaults)
• FIX: rename focuses on search box instead of active row (Manage Mailboxes)
• FIX: DNS Manager hostname canonicalization ignores subdomain resulting in former DNS records remaining in cache (DNS Manager)
• FIX: JS blur event fired on scheduling option “Custom” hides scheduling fields even if “Custom” is still checked (Crontab)
• FIX: $link property set for all translated paths (File)
• FIX: uconf-driver “add” operation returns Util_Process status, which should instead be the status code (Urchin)
• FIX: mailbox selection checkbox cancels click event (Manage Mailboxes)
• FIX: mysql user properties incorrectly displays last user (Manage MySQL)
• FIX: last page data overwritten following session re-initialization (Auth)
• FIX: parsing FTP records may exhaust memory limit with a sufficiently large file (Tabulate Bandwidth)
• FIX: update phpMyAdmin to login properly with phpMyAdmin 3.3 (phpMyAdmin)
• FIX: ensure no stale session data remains from a previous login when switching roles (Auth)
• FIX: corrupted MySQL stored password (phpMyAdmin)
• FIX: delete()- glob matches yielding no items treats the pattern as a real file (File)
• FIX: delete_mysql_user() cannot delete MySQL users (SQL)
• FIX: check if referent exists before accessing symlink in stat() (File)
• FIX: cache miss on stat() lookup for filesystem root generates error (File)
• FIX: corrupted backtraces (Error Reporter)
• FIX: IP-based HTTP requests fail site resolution (Bandwidth Logger)
• FIX: UTF-8 file encoding garbled (File Manager)
• FIX: process exit code regex conversion ignores integers (Process)
• FIX: exit code ignored (Process)
• FIX: master/worker premature input drain (apnscpd)
• FIX: page view statistics are not saved (UCard)
• FIX: user quota reported as account quota (Manage Users)
• FIX: scrollable div seek broken (apnscp JS)
• FIX: exit codes are ignored (Process)
• FIX: workers can block indefinitely during interrupted master/worker hand-off (apnscpd)
• FIX: extraction/listing fails for tar archives (File)
• CHG: is() and is_error() should adhere to method signature by returning bool (Error Reporter)
• CHG: permit mix-case mailboxes (Manage Mailboxes)
• CHG: all “Select All” field control to mailboxes (Manage Mailboxes)
• CHG: remove domain from http domain map before disassociating from account metadata (Aliases)
• CHG: skip /proc ACL calcuations (File)
• CHG: display insufficient information message if insufficient information for bandwidth projections (Bandwidth Statistics)
• CHG: ignore special _invoke() module method when fetching caller (Error Reporter)
• CHG: Update urchin to handle new http log format (ident field contains vhost)
• CHG: Upgrade FCKEditor to CKEditor (File Manager)
• CHG: properties action (File Manager)
• CHG: suppress repeated errors (Error Reporter)
• CHG: allow subdomain creation if subdomain directory exists and html symlink does not (Web)
• CHG: prevent catch-all creation on forwarded addresses (Manage Mailboxes)
• CHG: clarify select box for local user inclusion on forwards (Manage Mailboxes)
• CHG: support singular http bandwidth log format
• CHG: clarify unlimited quota by separate input control (Manage Users, User Defaults)
• CHG: prior to assuming apnscpd is running, verify process name matches executable (apnscpd)
• CHG: permit database deletion if db matches site prefix and admin user is not privileged (SQL)
• CHG: input focus changes border color in IE7 (CSS)

1:07 AM: KVM has been replaced and properly configured at this time.

1:10 AM: Maintenance window is closed.  All servers are online.

• Apache (http server): general security upgrade
• Dovecot (IMAP/POP3): performance and security improvements, major upgrade from 1.0 -> 1.2
• Postfix (SMTP): performance upgrade 2.5 -> 2.7
• SpamAssassin (spam filtering): improved spam filtering capabilities, major upgrade from 3.2 -> 3.3
• vsftpd (FTP): security upgrade, 2.1 -> 2.2

11:37 PM: all upgrades have been completed.

mod_rails

After several custom patches, mod_rails is a possibility for our clients.  mod_rails provides the same great, secure setup experienced with FastCGI, but without the nasty suexec headaches.  Perform an upgrade from Ruby 1.8 via Development > Code Frameworks.  Gems will be migrated to Ruby 1.9 during the upgrade process.  After upgrading add the necessary .htaccess directives found in the Rails Quickstart

Ruby 1.9 is here

Easy Task Scheduling (@token)

Another nice feature coming on the heels of a planned server outage is support for crontab @ time tokens.   @reboot executes a task on start-up, which obsoletes rc.d support, @hourly executes once per hour, @daily, @weekly, @monthly, and @yearly are additional special times recognized by crontab.  Each scheduling time is guaranteed to execute once during an interval, but the minute/hour vary.  Both minutes (@hourly/…/@yearly) and hours (@daily/…/@yearly) are evenly distributed among all accounts to prevent CPU bottlenecks caused, for example, by multiple hourly tasks occurring at 12:00 AM, 1:oo AM, 2:00 AM…

New Crontab Interface

apnscpd multi-tasking backend

Finally, the nicest improvement, based upon effort, includes a bevvy of performance optimizations culminating with a client-server replacement for the single-user privileged processor necessary for carrying out filesystem management of the account… that forked on every single privileged request.  For example, loading up the File Manager may generate as many as 5 separate forks of the privileged processor “lservice” [this name is about as old as the concept apnscp itself].  Now a master distributes work to one of several concurrent workers awaiting further instruction.  Best of all no additional forking required unless a worker dies on the job.   Another area of concern included sharing caches between the privileged processor and UI.  This was solved by implementing multi-tiered memcached framework.

As a comparative benchmark, requesting directory contents within the File Manager has decreased from 250 ms per request to 17 ms – a 14x improvement.  Speeding up code wasn’t enough.  Static content is now offloaded to a separate CDN (http://getap.is – clever?) to eliminate cookies, add caching, and reduce bandwidth overhead too.  Overall the latest set of changes should produce a noticeable improvement in responsiveness and allow you to get things done faster than ever before.

Assuming that spiel wasn’t enough to put you to sleep, here is your night cap in the form of a changelog:

• NEW: Ruby 1.9
• NEW: Util_Process_Tee – read from program standard input, write to external file (replaces built-in tee from Module Skeleton)
• NEW: account delete/edit hooks
• NEW: unified global/account/session cache support shared between UI and privileged processor
• NEW: multi-process privileged processor framework, 8x throughput improvement, handles multiple requests concurrently (apnscpd)
• NEW: delete Urchin profiles/log sources (Urchin)
• NEW: stream multiple commands to privileged processor (DataStream)
• NEW: recompose()- build regular expressions from multiple subexpressions (REGEX)
• NEW: user preferences (UCard)
• NEW: FTP bandwidth usage
• NEW: Styled, scrollable control with mousewheel support based upon jQuery Slider (Manage MySQL)
• NEW: raw PHP code interpreter (CLI)
• FIX: 201X backup rotation (Database Backup)
• FIX: printf named argument parsing (%(foo)s, %(bar)d) (Process)
• FIX: path transation corruption (File)
• FIX: bogus worker starvation response (apnscpd)
• FIX: Mode fixation on first auth handler triggered on start (Auth)
• FIX: non-interactive auth requests fail (Auth::CLI)
• FIX: session fixation (Auth)
• FIX: reset buffer status during worker initialization (apnscd)
• FIX: ftp usernames reported as site id (Bandwidth Logging)
• FIX: escape shell special characters in Urchin passwords
• FIX: apnscp version is not cached due to incorrect key (Page Template)
• FIX: backtrace argument corruption (Error Reporter)
• FIX: purge file metadata cache on chmod/chown (File)
• FIX: Modifying DNS cache duplicates parameter values consisting of multiple, space-delimited tokens (DNS)
• FIX: interrupt user creation process if usermod fails (User)
• FIX: multiple billing invoices result in malformed query (Billing)
• FIX: Profile changes result in “invalid token” error (Client Referrals)
• FIX: DNS fails to propagate on multi-homed accounts (DNS)
• FIX: uninitialized class reference in logfile callback (Add Subdomain)
• FIX: IE6 modal header formatting (Modal)
• FIX: refer to forms explicitly by name rather than position (SPF Wizard, Addon Domains)
• FIX: action icons are hidden in IE7 (CSS)
• CHG: result from apnscpd query returns return value instead of an object consisting of the return value among other unused flags (DataStream)
• CHG: support multiple domains (FrontPage)
• CHG: size-based rotation defaults to MB (Logrotate)
• CHG: expand crontab times to accept hourly/daily/weekly/monthly/yearly/startup task times (Cronjob)
• CHG: RESET flag flushes worker cache before processing a job (apnscpd)
• CHG: sudo uid no longer mandatory, defaults to current uid of worker (Process::Sudo)
• CHG: increase socket read buffer from 4k to 8k (apnscpd)
• CHG: permit multiple A/AAAA DNS records for a given subdomain/RR (DNS)
• CHG: Add reset flag to force apnscpd session reinitialization (DataStream)
• CHG: dump auth profile instead of session data since class data is now extensively cached within sessions generating enormous bug reports (Error Reporter)
• CHG: move archive extensions out of the constructor to enable archive recongition between cached module deserialization routines (File)
• CHG: Performance: remove call_user_func_array() and invoke afi::__call() directly
• CHG: Performance: save loaded modules between page requests (afi)
• CHG: add deferrable/non-deferrable Javascript (Template)
• CHG: notify if upgrade for Rails is available (Code Frameworks)
• CHG: Reduce module skeleton footprint.  Migrate call_proc* functions to Util_Process_* classes, pam_* to Util_Pam, break inheritance from DataStream (Module Skeleton)
• CHG: Page load improvements: use CDN for static assets, move /libjs to /js, flush HEAD immediately (Template)
• CHG: Rename file_create_symlink to file_symlink (File)
• CHG: adding a DNS record with no parameter raises an error (DNS)
• CHG: Performance: lazy load class name changes (Conf::Util, Process)
• CHG: Performance: optimize filesystem stat cache flushing frequency (File)
• CHG: Performance: determine account info freshness from domain info mtime (Account Info)
• CHG: Performance: cache get_users() response (User)
• CHG: Performance: reset account metadata on login if current/ mtime does not match saved value (Auth::Account Info)
• CHG: migrate user/account information to separate objects (User_Info, Account_Info)
• CHG: check if database exists before creation (SQL)
• CHG: sort search results based upon page view frequency (Search)
• CHG: clarify formatting of password reset e-mail
• CHG: show full help on first page view (Page Template)
• CHG: Performance: stat()- cache all directory entries on first run to improve response of related directory queries (File)
• CHG: cleanup aliases module
• CHG: obsolete common_*news* news functions, use http://updates.apisnetworks.com (Common)
• CHG: extract()- skip chown operation on empty archives (File)
• CHG: Ignore deprecation notice in Net/Whois.php (DNS)
• CHG: Separate MySQL Manager into 2 distinct views, populate only active user/database editor metadata (MySQL Manager)
• CHG: cacheless session switching between clients (CLI)
• CHG: redirect /favicon.ico to /images/favicon.ico for broken browsers
• CHG: update upload progress bar to apnscp 2 (File Upload)
• CHG: disable postback indicator triggered on login via session id (Dashboard)
• CHG: remove tooltip MySQL cache as it is now part of the short-term session cache
• REM: italicized property from hinted input elements (CSS)
• REM: get_php_version()- PHP4 information (Common)