Apis Networks

Welcome to another edition of apnscp changes!  Important changes include Rails infrastructure changes from Ruby 1.8 + FastCGI to Ruby 1.9 + mod_rails, new multiple worker backend (apnscpd), CDN to distribute static content with minimal bandwidth overhead (http://getap.is), and an aggressive caching  framework to reduce bottlenecks.

mod_rails

After several custom patches, mod_rails is a possibility for our clients.  mod_rails provides the same great, secure setup experienced with FastCGI, but without the nasty suexec headaches.  Perform an upgrade from Ruby 1.8 via Development > Code Frameworks.  Gems will be migrated to Ruby 1.9 during the upgrade process.  After upgrading add the necessary .htaccess directives found in the Rails Quickstart

Ruby 1.9 is here

Easy Task Scheduling (@token)

Another nice feature coming on the heels of a planned server outage is support for crontab @ time tokens.   @reboot executes a task on start-up, which obsoletes rc.d support, @hourly executes once per hour, @daily, @weekly, @monthly, and @yearly are additional special times recognized by crontab.  Each scheduling time is guaranteed to execute once during an interval, but the minute/hour vary.  Both minutes (@hourly/…/@yearly) and hours (@daily/…/@yearly) are evenly distributed among all accounts to prevent CPU bottlenecks caused, for example, by multiple hourly tasks occurring at 12:00 AM, 1:oo AM, 2:00 AM…

New Crontab Interface

apnscpd multi-tasking backend

Finally, the nicest improvement, based upon effort, includes a bevvy of performance optimizations culminating with a client-server replacement for the single-user privileged processor necessary for carrying out filesystem management of the account… that forked on every single privileged request.  For example, loading up the File Manager may generate as many as 5 separate forks of the privileged processor “lservice” [this name is about as old as the concept apnscp itself].  Now a master distributes work to one of several concurrent workers awaiting further instruction.  Best of all no additional forking required unless a worker dies on the job.   Another area of concern included sharing caches between the privileged processor and UI.  This was solved by implementing multi-tiered memcached framework.

As a comparative benchmark, requesting directory contents within the File Manager has decreased from 250 ms per request to 17 ms – a 14x improvement.  Speeding up code wasn’t enough.  Static content is now offloaded to a separate CDN (http://getap.is – clever?) to eliminate cookies, add caching, and reduce bandwidth overhead too.  Overall the latest set of changes should produce a noticeable improvement in responsiveness and allow you to get things done faster than ever before.

Assuming that spiel wasn’t enough to put you to sleep, here is your night cap in the form of a changelog:

• NEW: Ruby 1.9
• NEW: Util_Process_Tee – read from program standard input, write to external file (replaces built-in tee from Module Skeleton)
• NEW: account delete/edit hooks
• NEW: unified global/account/session cache support shared between UI and privileged processor
• NEW: multi-process privileged processor framework, 8x throughput improvement, handles multiple requests concurrently (apnscpd)
• NEW: delete Urchin profiles/log sources (Urchin)
• NEW: stream multiple commands to privileged processor (DataStream)
• NEW: recompose()- build regular expressions from multiple subexpressions (REGEX)
• NEW: user preferences (UCard)
• NEW: FTP bandwidth usage
• NEW: Styled, scrollable control with mousewheel support based upon jQuery Slider (Manage MySQL)
• NEW: raw PHP code interpreter (CLI)
• FIX: 201X backup rotation (Database Backup)
• FIX: printf named argument parsing (%(foo)s, %(bar)d) (Process)
• FIX: path transation corruption (File)
• FIX: bogus worker starvation response (apnscpd)
• FIX: Mode fixation on first auth handler triggered on start (Auth)
• FIX: non-interactive auth requests fail (Auth::CLI)
• FIX: session fixation (Auth)
• FIX: reset buffer status during worker initialization (apnscd)
• FIX: ftp usernames reported as site id (Bandwidth Logging)
• FIX: escape shell special characters in Urchin passwords
• FIX: apnscp version is not cached due to incorrect key (Page Template)
• FIX: backtrace argument corruption (Error Reporter)
• FIX: purge file metadata cache on chmod/chown (File)
• FIX: Modifying DNS cache duplicates parameter values consisting of multiple, space-delimited tokens (DNS)
• FIX: interrupt user creation process if usermod fails (User)
• FIX: multiple billing invoices result in malformed query (Billing)
• FIX: Profile changes result in “invalid token” error (Client Referrals)
• FIX: DNS fails to propagate on multi-homed accounts (DNS)
• FIX: uninitialized class reference in logfile callback (Add Subdomain)
• FIX: IE6 modal header formatting (Modal)
• FIX: refer to forms explicitly by name rather than position (SPF Wizard, Addon Domains)
• FIX: action icons are hidden in IE7 (CSS)
• CHG: result from apnscpd query returns return value instead of an object consisting of the return value among other unused flags (DataStream)
• CHG: support multiple domains (FrontPage)
• CHG: size-based rotation defaults to MB (Logrotate)
• CHG: expand crontab times to accept hourly/daily/weekly/monthly/yearly/startup task times (Cronjob)
• CHG: RESET flag flushes worker cache before processing a job (apnscpd)
• CHG: sudo uid no longer mandatory, defaults to current uid of worker (Process::Sudo)
• CHG: increase socket read buffer from 4k to 8k (apnscpd)
• CHG: permit multiple A/AAAA DNS records for a given subdomain/RR (DNS)
• CHG: Add reset flag to force apnscpd session reinitialization (DataStream)
• CHG: dump auth profile instead of session data since class data is now extensively cached within sessions generating enormous bug reports (Error Reporter)
• CHG: move archive extensions out of the constructor to enable archive recongition between cached module deserialization routines (File)
• CHG: Performance: remove call_user_func_array() and invoke afi::__call() directly
• CHG: Performance: save loaded modules between page requests (afi)
• CHG: add deferrable/non-deferrable Javascript (Template)
• CHG: notify if upgrade for Rails is available (Code Frameworks)
• CHG: Reduce module skeleton footprint.  Migrate call_proc* functions to Util_Process_* classes, pam_* to Util_Pam, break inheritance from DataStream (Module Skeleton)
• CHG: Page load improvements: use CDN for static assets, move /libjs to /js, flush HEAD immediately (Template)
• CHG: Rename file_create_symlink to file_symlink (File)
• CHG: adding a DNS record with no parameter raises an error (DNS)
• CHG: Performance: lazy load class name changes (Conf::Util, Process)
• CHG: Performance: optimize filesystem stat cache flushing frequency (File)
• CHG: Performance: determine account info freshness from domain info mtime (Account Info)
• CHG: Performance: cache get_users() response (User)
• CHG: Performance: reset account metadata on login if current/ mtime does not match saved value (Auth::Account Info)
• CHG: migrate user/account information to separate objects (User_Info, Account_Info)
• CHG: check if database exists before creation (SQL)
• CHG: sort search results based upon page view frequency (Search)
• CHG: clarify formatting of password reset e-mail
• CHG: show full help on first page view (Page Template)
• CHG: Performance: stat()- cache all directory entries on first run to improve response of related directory queries (File)
• CHG: cleanup aliases module
• CHG: obsolete common_*news* news functions, use http://updates.apisnetworks.com (Common)
• CHG: extract()- skip chown operation on empty archives (File)
• CHG: Ignore deprecation notice in Net/Whois.php (DNS)
• CHG: Separate MySQL Manager into 2 distinct views, populate only active user/database editor metadata (MySQL Manager)
• CHG: cacheless session switching between clients (CLI)
• CHG: redirect /favicon.ico to /images/favicon.ico for broken browsers
• CHG: update upload progress bar to apnscp 2 (File Upload)
• CHG: disable postback indicator triggered on login via session id (Dashboard)
• CHG: remove tooltip MySQL cache as it is now part of the short-term session cache
• REM: italicized property from hinted input elements (CSS)
• REM: get_php_version()- PHP4 information (Common)

(and by hefty I mean 2 months of unposted changes…)

There is good news to report though – a significant chunk of the changes were minor fixes to resolve regressions that appeared during rapid development of apnscp 2.   Even better – apnscp now has a very limited application lookup tool.

Access the search box either by mouse or as a shortcut junkie with the S access key, type in a few letters, hit enter to confirm the choice, and voila!  Zee app is loaded!

A notable mention, a long-standing bug preventing RubyGems from being upgraded via gem update --system has been patched, but unofficial until today.

Initial release of the inline search feature

ADD: inline search against application titles and taglines (Template)
ADD: proper 404/550 error pages (Template)
ADD: AJAX app method requestor, queries a public method of an application (apnscp)

FIX: rename operation does not work for subsequent subdomains (Subdomains)
FIX: tooltip displays undefined for “Administrative E-Mail” (Dashboard)
FIX: domain is not forgotten if omitted (Login)
FIX: login redirects only to dashboard (Auth UI)
FIX: updating credit card did not commit to database (Change Billing)
FIX: removing subdomain through Delete action fails (List Subdomains)
FIX: create_referral_profile()- missing invoice (Billing)
FIX: changing credit card details always results in failure (Billing)
FIX: bandwidth displays 0% (Template)
FIX: undefined Majordomo variables elicit warning (Majordomo)
FIX: referral profile is not synchronized to slave in time for reading (Billing)
FIX: footer is positioned at bottom
FIX: various IE6 rendering fixes – footer, login, postback “Hide Details” (UI)
FIX: prevent Overview event listener from handling propagated events (Page Template)
FIX: editing a DNS entry in an RR group triggers the edit for the respective position in the first RR group (DNS)
FIX: SPF Wizard does not warn of overwriting records (SPF Wizard)
FIX: add link to display tickets older than 90 days (Tickets)
FIX: browser detection returns spurious results (Page Container)
FIX: function permissions are not exported (SOAP)
FIX: WSDL generation exports uncallable methods (SOAP)
FIX: spurious free space report in storage gauge for users over quota (Template)
FIX: File Tree cancels first directory selection (File Tree)
FIX: methods reported missing within a module signature are not verified (afi)

CHG: parse “REFUSED” nsupdate response (DNS)
CHG: include default rbconfig.rb to allow gem update –system to complete sucessfully.  Make a symlink from /usr/bin/ruby to /usr/local/bin/ruby as well to clear up pathing by specifying a bindir of /usr/local/bin
CHG: referral system provides a one-time lump sum of $5, $10, $20, or $30 (Minimalist -> Super)
CHG: rename get_commissions() to get_referrals() (Billing)
CHG: omit 4 KB directory size from File Manager
CHG: omit parsing of @replies in Twitter feed (Dashboard)
CHG: consolidate frontend/backend functions (Majordomo)
CHG: increase base font size from 13px to 14px (UI)
CHG: remove extra service configuration information, update gauges (Manage Users)
CHG: remove unnecessary billing information from referral profile (Client Referrals)
CHG: display an excerpt of the latest news (Dashboard)
CHG: pulse postback status once for success, twice for warning, and thrice for error (Page Template)
CHG: lessen intensity of green table header for long tables (head1 pseudoclass) (CSS)
CHG: refactor delete() to stat just the parent directory (File)
CHG: improve stat() response for multiple lookups of the same file by statically caching before offloading to APC (File)
CHG: cache template configuration
CHG: fire postback status pulse on postback only (Template)
CHG: update disk space gauges (Quota Tracker)
CHG: use local session cache instead of global cache for tooltips (Tooltip)
CHG: rewrite user behavior logger for applications (Stats Logger)
CHG: promote referral statistics to dashboard (Dashboard)

REM: Legacy Help is irrelevant

(phew! nearly 50 changes)

apnscp 2, the successor to our control panel, apnscp esprit, has been released.  apnscp 2 is an enormous milestone that represents a shift in focus from backend to UI development – Instead of devising a process to execute a task, how can the process be simplified to the user.  The announcement will be abbreviated to give a glimpse of earlier WIPs, including the first run of apnscp 2 that thankfully never was.  The next step is a backend move from the proprietary gobbledygook MVC to Zend Framework, then back to UI improvements.  Enjoy!

First draft for apnscp 2 in May

Initial dashboard design (July 29)

Revised dashboard (July 31)

Tag cloud still present - a future tie-in (August 6)

Menu expansion background (July 29)

Full app example (August 3)

Second rendition of menu expansions (August 4)

Graduated gauges - unknown/normal/warn/critical (August 6)

Notification presentation for postbacks (August 6)

Latest dashboard (September 17)

Changes:

• Added: file upload queuing (File Manager)
• Added: setCursorPosition() – position cursor in text boxes, optionally inversing selection and highlighting file name in file.  Used in file rename dialog in File Manager (apnscp JS)
• Fixed: get_next_payment()- subscriptions without a payment are ignored (Billing)
• Fixed: saved passwords become corrupt when login exceeds 32 characters (Login)
• Fixed: list_mysql_databases()- old MySQL database grants escape “-_” in the database name (SQL)
• Fixed: destination removed after retrieval by “Download & Extract” utility if current directory is /tmp (File Manager)
• Fixed: site storage usage misreported (Site Summary)
• Fixed: get_debug_bt()- uncaught exceptions result in off-by-one error while enumerating the callstack (Error Reporter)
• Fixed: garbled display for browsers masking User-Agent header (Page Template)
• Fixed: get_invoice()- Using $this when not in object context (Billing)
• Fixed: literal percent (%) treated as format specifier (Process)
• Fixed: query_params() does not instantiate PGSQL database connection
  apnscp 2 (2009/08/08)
• Fixed: text indent doesn’t work under Webkit
• Fixed: Granular Logging does not capture primary domain if www is omitted
• Fixed: Feedback mailto is invalid, encode + as %2B
• Fixed: domains unassociated with MDM are removed upon Activate Changes (Multiple Domain Manager)
• Fixed: various IE6, IE7, WebKit rendering fixes
• Fixed: bogus upload speed (File Manager)
• Fixed: duplicate file upload queue entries under IE7 (File Manager)
• Fixed: WSDL generation ignores commands not explicitly exported (SOAP)
• Fixed: clean-up temp file on duplicate file uploads (File Manager)
• Fixed: privilege escalation (SOAP)
• Fixed: empty argument list passed incorrectly as null (afi)
• Fixed: PHP 5.3 incompatible changes: AddPoint() casts empty string as array (pChart), split() deprecation (ftp, file)
• Fixed: unset booleans raise parser notice (Majordomo)
• Fixed: empty args treated as string (AJAX Engine)
• Changed: order ticket notes in reverse chronological order, word wrap (Trouble Tickets)
• Changed: expandable sections (MySQL Manager, Referrals)
• Changed: rename Referral Tree to Referrals, clean-up
• Changed: improve post-upload speed with large files (File Manager)
• Changed: apply default module permissions thereby consolidating the exported function table
• Changed: append_trouble_ticket()- clean-up ticket formatting (CRM)
• Changed: increase verbosity of unsynchroniZed domains (Multiple Domain Manager)
• Changed: hide TCP connections by default (MySQL Manager)
• Changed: clean-up (phpMyAdmin, phpPgAdmin)
• Changed: ticket reply-to address to support+tickets@apisnetworks.com (CRM)
• Changed: apnscp 2 initial release
• Changed: rewrite Granular Logging matching to use env variables from subdomain/domain rewrites instead of introducing a separate set of SetEnvIf directives
• Changed: filter PHP functions from generating error reports (Error Reporter)
• Changed: disable multiselect auto-expand on mouseover behavior (Multiselect)
• Changed: auto-load news if visible when switching news sources (Dashboard)
• Changed: file rename automatically selects file name (File Manager)
• Changed: PHP 5.3 compatibility
• Change: split() deprecation
• Changed: rename application container from /pages to /apps
• Removed: mysql packet compression with mysqlnd interface (http://bugs.php.net/bug.php?id=47017)