Innovation is an important component of apis. Roughly 75% of my workweek is allocated towards research, design, and implementation strategies, which has enabled apis to rollout its own filesystem, control panel, and amenities. Several topics have been brought up through the forums, trouble tickets, and e-mail that bring up several interesting changes for everyone within the community – ext4, new backup system, MySQL 5.1, esprit -> apnscp 1.0 development, privatized Tomcat containers, RAM containers. I would like to take some time today to briefly discuss the state of these changes and bring everyone up to pace with what’s going on around apis.
ext4
ext4 is the filesystem successor to ext3 highlighting improved I/O throughput and journaling capabilities among other benefits. Most importantly, disk consistency check times have significantly decreased, which means quicker server reboots every 6+ months.
An ext4 upgrade had been planned for the end of January, but some users brought up concern with an in-place filesystem upgrade. While I can reassure everyone that the in-place upgrade is seamless – both development and satellite servers are running ext4 – long-term real-world usage still has limited research data. Fedora Core, Red Hat’s desktop Linux distribution, will default to ext4 beginning with release 11 in May. This major distribution will provide ample data on ext4 stability and help assuage fears of filesystem corruption. Not withholding, deferment also provides me with time to coalesce the crude external backup drives into a single PE2950 server… and spend some time with Dell’s PERC 6i RAID controller.
Backup system
As touched on previously, servers now have an external SATA drive attached for backups in the near future. Scalability problems have plagued old backup system, but help is on the way with a distributed filesystem template scheduled after esprit 0.99 is closed out sometime next month. I will spend the brief holiday restructuring filesystem layouts to provide nightly, incremental backups that are also scalable. Filesystem hardlinks, which allow smart jails to operate, have become an impediment as server architecture becomes more efficient and account density grows. Maintaining 45,000+ hardlink relations across 500+ sites through rsync will bring even the stoutest server to its knees. I have visualized the new system as skipping known invariant distribution points, e.g. /usr/bin, /etc, while selectively pulling site-specific configuration files from those directories, then calculating changes through ddsnap in Zumastor. Sounds great on paper, but we’ll wait and see if this provides a space and time-efficient backup system.
MySQL 5.1
5.1 introduces database-level grants for triggers and stored procedures providing users with the ability to write routines and utilize triggers. The first recommended release (GA) introduced a slew of nasty bugs that preclude it from being deployed on the servers. MySQL 5.1’s first GA, 5.1.30, was released on November 14th. Prior to 5.1.30, releases came roughly 30 – 45 days apart; when 5.1.31 comes out is anyone’s guess. Until 5.1’s shortcomings are addressed, 5.0 will remain the active distribution.
apnscp 1.0
apnscp esprit 0.99 is slowly approaching the finish line by replacing the few remaining Ensim applications with its very own version. An htpasswd driver (Manage Users/Manage Groups/Protect Directories under “Web Server”), Site Summary, and SSL Certificates are the few applications that remain. apnscp 0.99 has had one goal from the start, to free itself from Ensim and provide a standalone UI, which is why I never used version “1” to denote the current iteration of the control panel.
While esprit’s focus has been separation and getting its feet off the ground, apnscp 1.0 will focus on usability and design. I know the current design is terrible, no need to remind me. A new reporting system is the first planned feature for apnscp 1.0, designed to facilitate user feedback and bug reporting. I want to encourage everyone ahead of time to think of tie-ins, such as adding a domain in MDM includes the option to setup logfiles, or minor changes like moving the remove/delete operations to the far-right. Shoot me an e-mail at matt+feedback@apisnetworks.com whenever something comes to mind on how it can be made better and easier to use. All of the suggestions will be factored into apnscp 1.0’s design… the dashboard, a particular application, application titles, application grouping— you name it. If you feel it’ll improve the experience for everyone, voice it regardless of how over the top it sounds. Nit-pick as much as you would like! I nit-pick my own design (this is possibly why I’m so slow). Don’t be afraid to step on my toes by saying x sucks or y looks ugly, because I will likely agree. Send your feedback on the control panel to matt+feedback@apisnetworks.com.
RAM Containers (Tomcat, JBoss, MySQL 6, anything)
Touched on briefly a few months ago, privatized Tomcat containers are coming, but up until recently how was still being researched. More specifically, how to partition process RAM utilization in an efficient manner was the stumbling block. Plenty of italicized words I know, but for good reason— emphasis. I stumbled across control groups (cgroups), which can be leveraged through a custom PAM module to attach shell processes to a site’s cgroup RAM container. An inotify daemon can in turn monitor the cgroups for memory overages and either warn or kill offending processes that exceed the memory quotas. RAM accounting allows carte blanche with services, including the option to run your own Tomcat/jBoss servers with the option to upgrade RAM on your plan. This feature will be launched on the Basic and higher packages once the bugs have been ironed out. I have a solid process of how to implement RAM accounting, which is a big step towards manifesting this new feature. Next, a daemon needs to be written and tied into the control panel. I am looking at an early March deployment for RAM containers.
– Matt