Introducing Apollo: Our Next Generation Platform

After a few months of strenuous development, testing, and problem-solving Apollo has touched down sporting our latest hosting platform.  Based on RedHat Enterprise Linux 5.5, this platform encompasses several key elements from the proposed Fifth Generation architecture.

Layered Filesystems
Accounts now reside upon stacks of folders (“layers”); each layer beyond the first, which holds user files, provides a collection of system files.  Our new layout provides a very efficient method of adding new software to accounts by injecting files into a particular layer, which propagate to all inheritable accounts.  Further, by residing on the top-most layer, instead of selectively backing up /home and /var/www, we have a comprehensive method of tracking which files to backup for clients.

Layered filesystems decrease the amount of storage required to populate an account, and allow a true 1:1 replication of system files.  The old system, a Replicated Filesystem, utilized hardlinks, which are references to files, to populate system files.  This posed several problems:

1. very inefficient to create a reference for 50,000 files to each site
2. filesystem limitations prevent creation of more than 32,768 hardlinks to a file introducing scalability concerns.  This limit prevented clients access to all locales, severely hindering application capabilities
3. adding new software is quite cumbersome, which requires removing all references to the file (thankfully scripts are used to automate this exhaustive task), then copying the application to the filesystem template, then creating a reference for each file on each account (yikes!).  Any references that remain of the old file still count as occupied storage.
4. servers accumulate wasted storage over time from old software references resident on the server and by storage occupied from duplication of folders and file references (17 MB vs 1.3 MB baseline)

Control Groups
Control groups (“cgroups“) provide an accurate and fair means of accounting for memory and CPU usage generated by an account.  cgroups aggregate memory and CPU generated by applications grouped by account, which means for clients, we can permit any software to be run on an account up to x MB or y CPU seconds.  CouchDB?  MongoDB?  MySQL 5.5? (might want to read the next note!) Jetty?  You name it!  If it fits within the container limits, have fun!  Even better – unlike a VPS, crucial system processes do not count against your limits – only applications that you explicitly start.

Software Upgrades
Along with the latest applications provided by RHEL, additional software upgrades were selected:

• Percona MySQL 5.1  – clients now have privileges to create and run stored procedures/triggers.  InnoDB performance is greatly improved, on par with MySQL 5.5
• Mercurial + Bazaar– two revision control tools clients have asked for in the past
• mod_rails 2.2.15 – Rails and Rack functionality tested and working 100%
• Horde Groupware – enhanced webmail now includes a calendar with memo and task-creation capabilities
• Vim 7.2 – support for tabs
• Pyzor – additional spam filter, which will enhance filtering capabilities

Suexec Changes
Clients bemoaned a super-restrictive suexec wrapper that, in all honesty, was a little too restrictive:  (1) FastCGI/CGI applications could not be executed inside /home/user, but could inside /home/user/public_html; (2) 775 parent directory permissions would cause a failure regardless of who owns the folder complicating Rails dispatcher deployment (you should be using mod_rails now!); (3) best of all, suexec logs, which contained invaluable clues why a script was denied execution, were never logged to your account!  All three of these problems have been resolved with a new suexec wrapper and those error messages?  Check /var/log/httpd/suexec.log 😉

Want to be a part of Apollo?  Open up a ticket beginning Monday for a server migration.  This process takes approximately 24 hours for a preparatory phase, followed by the actual move.  Moves take less than 5 minutes to complete and DNS is updated automatically.

Thank you everyone for your feedback,
Matt