Apis Networks

Coming Soon: Atlas the v7 Platform

Atlas, a new hosting platform, will be launching soon. It’s built off technology introduced by Luna, a v6.5 platform with a stronger focus on hardening – specifically reducing unauthorized activity. The server will be activated in a couple weeks and go live in mid or late December for early migration. As technology becomes more widespread so too do opportunities to misuse and abuse technology. We must take proactive steps to limit not only the risk that our clients face but too the external consequences they leave behind in the event of a hack. These situations have shifted from “what-if” to “when” scenarios, and nothing is as rewarding as relaxing on a beach sipping a Mai Tai during Summer. Until we achieve that goal ourselves, we’ll continue to refine the platform and introduce an iteratively stronger platform so that you can have good company on a beachfront on a warm Summer day.

With any major platform release comes great care in downselecting hardware. Nothing leaves me with more disgrace than signing off on a failed platform. Apollo was the first and last of that class (RAID5 on early-gen 2.5″ 10k SAS), which leads into SSD suitability for high volume, high reliability scenarios.

SSD has matured significantly since its introduction in the 2000s so much so that it is the de facto for consumer-grade storage, but consumer grade pales to the needs of 24/7/365 enterprise applications. Hardware fails. In 14 years of running Apis, I’ve seen everything fail from power supplies to planar boards to capacitors to SCSI ribbon cables. Regardless of statistics, we’re all governed by physics and in the physical world, shit breaks.

We’re left in a predicament: speed over reliability and generally it takes years for an equilibrium to meet with an advancement in either field. SSD is broken down to a variety of formats on a microscopic level. Single, tri, multi, and enterprise-multi that determine how many states a bit may be stored as. More layers confer more storage but also more volatility and higher wear rates. Wear is bad; it’s what separates a recoverable CRC soft failure with a catastrophic, 100% data loss hard failure. I believe that in no situation that can be avoided should data safety be sacrificed for data performance. Overall performance can be tweaked elsewhere and storage should be long-term storage, not short-term data retrieval. Short-term data retrieval has always been and should always be RAM, not disk storage. Mechanical, so long as you’re not in an earthquake-prone zone, will continue for the best option in the near future in terms of cost and benefit.

For our situations, read/write is split 55%/45% and the majority of which is log files. Every situation is unique and should be evaluated carefully. Of this, the worst-case scenario server pushes around 250 IOPS. It’s split between 2 VMs each hosting ~750 domains, so each VM churning out websites, handling email, and delivering data over MySQL + PostgreSQL still fail to make a mark on 1 SSD drive in terms of its theoretical minimum (5,000 IOPS). Even more interesting, the theoretical maximum of an 8x RAID10 15k SAS arrangement is 1,000.

But there’s a nasty downside to SSD beyond performance overkill. Take a comparable 512 GB Samsung EVO 850 Pro, an SSD, which is approximately $225. Compare with a 600 GB Dell-certified 15k SAS drive at $233 ($990 via Dell, yeowch!). SSD requires overprovisioning to avoid overusing cells that results in premature drive failure. In enterprise applications, the rule of thumb is 20% leaving 410 GB usable. Now, SSD is $0.5487/MB versus $0.375/MB for an entry-level nearline SSD. This excludes consumer-grade MLC drives with far lower endurance rates and exponentially higher failure rates at 24/7/365 duty cycles. SSD has improved since its introduction, but the laws of probability still dictate that increasing the number of events will greatly amplify the outcome of at least 1 event happening. The Birthday Paradox illustrates this risk of multiplicity so well and so much so that often it’s taught at an introductory Algebra class.

Samsung officially rates is EVO 850 Pro good for 300 TBW. Augend, the most recent server retired, sported 2 300 GB 3.5″ 15k SAS drives since its upgrade in 2010 and in 6 years it processed over 723 TB of data on each drive (RAID1). Compared to other servers, it’s rather quiet. Helios, which is the largest monolithic server, has processed at least 1.3 PB on each drive since 2012. Based upon Samsung’s numbers, in 6 years at a life expectancy of 300 terabytes, Augend would have gone through 4 drives and Helios 18 drives. I can improve IOPS at the expense of reliability; yet, I would sacrifice long-term reliability to improve something that isn’t limited based upon architecture.

Data loss is scary. It’s more scary than a DoS, because at the end of it all, you have to tell your clients, “I failed. I failed to uphold the faith you instilled upon my company.” It goes beyond that: it alters your trust. Yes, there are backups just like there is insurance, but if your engine fails resulting in a catastrophic loss of your vehicle, you’d lose faith in the manufacturer. Data loss is no different. It drives your business. If I were in your shoes and suffered a catastrophic loss, I too would switch providers.

I love SSD. I use it for my desktop. But, I would never put the needs of marketing nor the opportunity to improve upon client density at the risk of jeopardizing client data. We all work and we deserve an environment to thrive. Based upon my research, I am not 100% comfortable yet with SSD handling rigorous workloads. I’m not OK with the reduction in storage nor the workarounds to preserve performance over long hauls. I am, however, OK with grabbing a Mai Tai with you on the beach without a worry in the world.

Atlas will not be a SSD platform, but I hope that the SSD improvements afford a chance to its successor when the time comes in 1-2 years.

PS: Atlas will be open for migrations in December. Got a request? Let me know on the forums!

Matt Saladna
Owner, Lead Platform Architect




Our packages are changing…

Effective June 22, 2016 our packages have changed for new clients. Technology changes as does what is considered the “bare minimum” utility set. Applications built on Symfony assume Composer support. Laravel requires Artisan. Even static sites use SCSS/SASS preprocessors to change look and feel on the fly. Everything is tied back to the terminal, which is now standard on every package. Every package also includes requisite support to run Node, Ruby, or Python utilities from the command-line limited only by available package memory (begins at 256 MB). All accounts prior to June 22, 2016 will be grandfathered into the old rates and specifications.

Along with standardizing terminal support, the baseline $5/month package has been retired in favor of a $10/month Essential package including a doubling of storage to 2 GB. All packages have had monthly transfer doubled as well.

Moving forward, including terminal support on all accounts provides an exciting opportunity to streamline, within the control panel, the development pipeline for clients from source to release of web applications.

Oh, and we’re turning 14 in July, so stay tuned!

Matt Saladna
Founder & Lead CP Developer

Comments (1)


apnscp Update + New Fortification Mode: Learning

A new control panel update has been pushed to the servers incorporating a “Learning Mode” to Fortification available under Web > Web Apps. Learning bridges the gap between known and unknown web apps by taking a filesystem snapshot, opening permissions, then taking another snapshot after a predetermined amount of time (60 minutes). Changes are calculated three ways: creation time, modification time, and size to determine what files the web application has modified during this window. Once a changeset has been calculated, the control panel locks down all files except those files (or directories) required by the web server.

Learning Mode is only available on unknown apps. Common apps (WordPress, Joomla!, Drupal, and Magento) have prescribed Fortification Modes available to each (and Web App Write Mode is always available to allow 100% write access for 10 minutes). Learning Mode is relevant only when an application is installed that does not fit into the big four.


Other changes

Beyond Fortification enhancements, there is a new API call for unattended commands, pman_schedule_api_cmd(), which powers a significant share of new CP enhancements; scheduled task tagging to prevent duplicate tasks; initial webhook framework; localization (Account > Change Information); and 125 miscellaneous changes and fixes to the control panel!


  • NEW: NO_WAIT flag, send command to backend without waiting for response (DataStream)
  • NEW: localization. Feature does not affect CP language at this time, only one-clicks if support
  • NEW: Joomla! one-click, still need to implement update + extension management (Web Apps)
  • NEW: Web App Write Mode, a fortification mode that temporarily grants write-access to the web server for 10 minutes, then seizes those files created rendering it secure from future modification (Web Apps)
  • NEW: add support for non-numeric array keys (Error Reporter)
  • NEW: add umask support to Util_Process::Sudo()
  • NEW: schedule_api_cmd_admin()- run an api command as any arbitrary account and any user under that account (Pman)
  • NEW: schedule_api_cmd()- run an apnscp command as the current user at a given time (Pman)
  • NEW: add duplicate atd process detection with ID tags (Util_Process::Schedule)
  • NEW: Magento 1.x one-click
  • NEW: $.ajaxWait()- add ajax spinner icon + disable button on interaction (apnscp)
  • NEW: Drupal one-click
  • NEW: set_acls()- glob() support (File)
  • NEW: renew()- renew LE certificates (LetsEncrypt)
  • NEW: add WordPress install option to Add Subdomain/Addon Domains
  • NEW: include recommended preloaded WP stacks
  • NEW: WordPress one-click module (WordPress)
  • NEW: normalize_path()- take a hostname + optional path component and return its corresponding docroot (Web)
  • NEW: support array data types using the format [item1,item2,…] or key-value as [item1:val1,…] (cmd)
  • NEW: empty_mysql_database() – retain schema structure, but purge all records in db (SQL)
  • NEW: export databases (MySQL Manager)
  • NEW: EOL selector on file edit (File Manager)
  • NEW: Let’s Encrypt multi-domain (SAN) support (SSL)
  • FIX: typo on path to log files (File Manager)
  • FIX: allow deletion of directories (File Manager)
  • FIX: fortify(max) called immediately after web app write mode enabled (Web Apps)
  • FIX: truncated parameters include markup language (DNS)
  • FIX: strip quotes if key passed to CLI quoted (cmd)
  • FIX: bring back addon domain editor (Addon Domains)
  • FIX: error once if domain -> site mapping fails instead of on each record (tabulateBandwidth)
  • FIX: email on v4 platforms delivered using LF EOL, parser expects CRLF (parseTicket.php)
  • FIX: trim randomly generated name if name + prefix exceeds user column max length (Module_Support::Webapps)
  • FIX: get_directory_contents()- call stat() instead of stat_backend() to take advantage of stat cache (File)
  • FIX: killing shellinabox left behind children (Service::Terminal)
  • FIX: encode seen server cookie to prevent cookie spanning multiple lines that would break the Set-Cookie header (Auth::Redirect)
  • FIX: get_acls()- incorrect caching (File)
  • FIX: when changing domain, on logout, force an internal proxy redirect to the server. Logging out will clear the temporary proxy location cookie rendering the client unable to login following a domain change (Change Information)
  • FIX: extract inline attachments (parseTicket)
  • FIX: get_user_settings() must return array at all times (Manage Users)
  • FIX: require domains to be selected when creating a subdomain (Subdomains)
  • FIX: get_address_from_domain()- handle addon domains (Admin)
  • FIX: heredoc syntax on v6.5+ platforms (WordPress)
  • FIX: global subdomain url formatting fails (Web Apps)
  • FIX: PHP 5.3 compatibility, using self within a closure (Pman)
  • FIX: match foo.domain-name.com (Regex)
  • FIX: set_acls()- parser fixes (File)
  • FIX: reorder su parameters (Util_Process::Sudo)
  • FIX: new directory form control injection (FileTree)
  • FIX: Uncaught Error: Syntax error, unrecognized expression: option[value=] (Manage Mailboxes)
  • FIX: mysql_database_exists()- if mysql database exists, is not located under site root, and has privileges by dbaseadmin, return true (SQL)
  • FIX: only directories in xferlist are copied (Transfer)
  • FIX: missing __state variable (PostgreSQL Manager)
  • FIX: AJAX indicator does not show (CSS)
  • FIX: multiple stale cache bugs relating to recently-deleted files (File)
  • FIX: TypeError: g.load is not a function (Dashboard)
  • FIX: preserve custom mailbox on user rename (Mail)
  • FIX: unable to remove GA key (Change Information)
  • FIX: fallthrough subdomain improperly configured (Subdomains)
  • FIX: add_virtual_transport()- accessing dns_get_records() as if non-multidimensional array, doh (Email)
  • FIX: broken mailbox editor
  • FIX: escape chroot argument (Util_Process::Chroot)
  • FIX: hostname param missing on user edit when remote hosts not displayed (MySQL Manager)
  • FIX: login option alignment on smaller screens
  • FIX: row improperly cleared (Summary)
  • FIX: support TXT records > 255 characters (DNS)
  • FIX: destination not set on edit (Manage Mailboxes)
  • FIX: install()- SSLCertificateChainFile is not set when replacing a certificate if custom HTTP config present from previous SSL install (SSL)
  • FIX: build CSS assets under subdirectories
  • FIX: get_processes() – Fatal error: Cannot access self:: when no class scope is active on PHP 5.3 (Pman)
  • CHG: add .htaccess to minimum fortification (wordpress)
  • CHG: tag active fortification mode (Web Apps)
  • CHG: load page with database/user list expanded upon deletion (MySQL Manager)
  • CHG: add system status to login page (Login)
  • CHG: chown()- use optimized shadow assertions, ~70x speed up (file)
  • CHG: chown()- prefer tomcat_system_user() for proper Tomcat user (file)
  • CHG: set_acls()- always treat flags as array (file)
  • CHG: renewals that fail due to zero valid hostnames (account moved on) should not elicit a renewal failure (Letsencrypt)
  • CHG: trim parameter before ajax callback (DNS Manager)
  • CHG: auto-install Composer on 4.5+ platforms if missing (php)
  • CHG: cap Magento to 1.x if MySQL < 5.6 (magento)
  • CHG: kick out Magento 2 for now…
  • CHG: on func cache miss, expire func table (apnscpFunctionInterceptor)
  • CHG: db_config()- export table prefix (Web Apps)
  • CHG: remove no recalc mask (n) flag to default ACL settings (file)
  • CHG: proper Magento version tracking (Magento)
  • CHG: add uid support to -u (Auth::CLI)
  • CHG: Add DNS form formatting (DNS Manager)
  • CHG: avoid globbing non-existent paths (File)
  • CHG: set_acls()- retain effective rights mask on acl adjustment (file)
  • CHG: run()- force umask to override system default (0002) for site admins (pman)
  • CHG: fail request if error produced (letsencrypt)
  • CHG: delay HTTP reload for SSL installation by 2 minutes, reload when load permits, and tag by ID to prevent duplicate reload requests (SSL)
  • CHG: fully implement automated Let’s Encrypt SSL renewal
  • CHG: consolidate Util_Process::Batch into Util_Process::Schedule
  • CHG: escape the parameter rather than the components (Util_Account::Editor)
  • CHG: unconditionally enable openssl on certificate install (SSL)
  • CHG: force SSL if certificate setup OK (Magento)
  • CHG: get_acls()- only pull ACL information from shadow layer where appropriate (File)
  • CHG: require fortify()/unfortify() implementations; reflection will not pick these methods up unless explicitly defined in the derived class (WordPress, Drupal)
  • CHG: set trusted_host_patterns on 8.x+ Drupal installations (Drupal)
  • CHG: limit ticket data height to at most 650px (Trouble Tickets)
  • CHG: strip reply from Outlook-generated messages that merge reply + original message in one mime part (parseTicket)
  • CHG: add bypass support for changing primary domain on an account (Change Information)
  • CHG: if detection fails, ensure map entry is removed (Webapps)
  • CHG: get_docroot()- truncate www. if present (Web)
  • CHG: stat()- return empty array instead of null if stat path does not exist (File)
  • CHG: breakout webapp common function to separate support structure (Module_Support::Webapps)
  • CHG: create docroot directory if it does not already exist (WordPress)
  • CHG: open some method privileges to administrator (SSL)
  • CHG: take ceil used, floor free in calculating storage/bandwidth usage (UCard)
  • CHG: take ceil used, floor free in calculating storage/bandwidth usage (UCard)
  • CHG: integrate htaccess manager + app manager into Addon Domains and Subdomains
  • CHG: enable recovery mode
  • CHG: perform correct redirection if account resides on proxy endpoint and is suspended (Auth::Redirect)
  • CHG: add namespace autoloading (apnscpFunctionInterceptor)
  • CHG: support flushing user preferences if preferences written through common_save_preferences() instead of UCard::setPref()
  • CHG: create_temp_mysql_user()- MariaDB 10.1 no longer implicitly creates user on GRANT (SQL)
  • CHG: handle case where all_domains/ exists and is directory gracefully
  • CHG: reject record if its hostname contains spaces (DNS)
  • CHG: add support methods getParameterByName(), getUrl, allow ucwords to extend String object (apnscp)
  • CHG: verify proc pid exists before performing stat (Pman)
  • CHG: move()- only perform symlink chown if uid >= minimum uid or apache; excludes changing to root if symlink was root:root, e.g. mainwebsite_html (File)
  • CHG: bypass systemd when reloading (and config rebuilding) HTTP server (Web)
  • CHG: only invoke g.load if it is really loaded (Dashboard)
  • CHG: mysql_schema_column_maxlen()- cache results (SQL)
  • CHG: set_mysql_privileges()- add support for $privileges to be a two-element array consisting of “read” and “write” privilege simplification (SQL)
  • CHG: purge page cache on file upload on v6.5+ platforms (File Manager)
  • CHG: disable button + replace with “Processing” on account wipe (Change Information)
  • CHG: initialize_download()- remove fifo deletion callback on fork, since code can never logically reach (File)
  • CHG: make recursive directory deletion implied (File Manager)
  • CHG: flush page template buffer before app logic evaluation (Page Template)
  • CHG: unoptimized codepath if calling stat() from UI (File)
  • CHG: remove .php exposure on soap endpoint (Net_WSDL)
  • CHG: display amnesty request if storage within 98.5% capacity, instead of 99.5% (Dashboard)
  • CHG: modify_mailbox() make parameter list less cumbersome, drop catchall ability entirely (Email)
  • CHG: cache payments for only 24 hours (Billing)
  • CHG: send HUP, not USR2 to flush Dovecot auth cache on user creation (User)
  • CHG: move auth flush from user to user creation hook in email module
  • CHG: record_exists() drop timeout from 5 to 1 second (DNS)
  • CHG: expand upload block if previous action is upload (File Manager)
  • CHG: copy()- use non-shadow path on copy for overlayfs-backed platforms, which have issues invalidating the page cache on direct branc manipulation without performing a costly remount (File)
  • CHG: convert backup routines to Util_Process (backup DBs)
  • CHG: install()- ensure http config is rebuilt (SSL)
  • CHG: make postcss available on v6.5+ platforms
  • CHG: use postcss for CSS minification
  • CHG: switch JS build from YUI to UglifyJS
  • CHG: upgrade TinyMCE to 4.3.2
  • CHG: add PEAR5.php compatibility library for Util_HTML_BBCode
  • CHG: split_host()- include hostname that failed splitting (Web)
  • CHG: general JS fixups (DNS Manager)
  • CHG: jail_user()- use the referent if jailed path is a symlink (FTP)
  • CHG: kill SIGUSR2 dovecot/auth on v6.5+ platforms to flush dovecot auth cache (User)
  • CHG: minor analytics appearance tweaks
  • CHG: move open ticket position for admin (Trouble Tickets)
  • CHG: Dovecot on 6.5+ platforms use systemd, which lacks non-LSB commands. Instead of dovecot flush/reload send SIGUSR2 directly to auth process to flush cache on user addition (User)
  • CHG: delete_user()- remove user-specific subdomain during user deletion if only subdomain present (User)
  • CHG: include server name in X-Server header
  • REM: tomcat4/tomcat name conversion from get_service_value() (Module_Skeleton)
  • REM: jquery.ui.css dependency (Terminal)

Comments off