Apis Networks

Time for another periodic listing of apnscp updates over the last few months.  The following changes are mostly fixes or minor enhancements to improve usability of the interface.  Two new notable features found their way into apnscp in March: IMAP/POP3 bandwidth logging (Reports > Bandwidth Breakdown) and control panel access restrictions (Users > Manage Users).  New features are in the pipeline now that apnscp is bug-free

New feature to permit/deny control panel access for users.

• NEW: IMAP/POP3 bandwidth logging
• NEW: control panel permit/deny (Manage Users, User Defaults)
• FIX: rename focuses on search box instead of active row (Manage Mailboxes)
• FIX: DNS Manager hostname canonicalization ignores subdomain resulting in former DNS records remaining in cache (DNS Manager)
• FIX: JS blur event fired on scheduling option “Custom” hides scheduling fields even if “Custom” is still checked (Crontab)
• FIX: $link property set for all translated paths (File)
• FIX: uconf-driver “add” operation returns Util_Process status, which should instead be the status code (Urchin)
• FIX: mailbox selection checkbox cancels click event (Manage Mailboxes)
• FIX: mysql user properties incorrectly displays last user (Manage MySQL)
• FIX: last page data overwritten following session re-initialization (Auth)
• FIX: parsing FTP records may exhaust memory limit with a sufficiently large file (Tabulate Bandwidth)
• FIX: update phpMyAdmin to login properly with phpMyAdmin 3.3 (phpMyAdmin)
• FIX: ensure no stale session data remains from a previous login when switching roles (Auth)
• FIX: corrupted MySQL stored password (phpMyAdmin)
• FIX: delete()- glob matches yielding no items treats the pattern as a real file (File)
• FIX: delete_mysql_user() cannot delete MySQL users (SQL)
• FIX: check if referent exists before accessing symlink in stat() (File)
• FIX: cache miss on stat() lookup for filesystem root generates error (File)
• FIX: corrupted backtraces (Error Reporter)
• FIX: IP-based HTTP requests fail site resolution (Bandwidth Logger)
• FIX: UTF-8 file encoding garbled (File Manager)
• FIX: process exit code regex conversion ignores integers (Process)
• FIX: exit code ignored (Process)
• FIX: master/worker premature input drain (apnscpd)
• FIX: page view statistics are not saved (UCard)
• FIX: user quota reported as account quota (Manage Users)
• FIX: scrollable div seek broken (apnscp JS)
• FIX: exit codes are ignored (Process)
• FIX: workers can block indefinitely during interrupted master/worker hand-off (apnscpd)
• FIX: extraction/listing fails for tar archives (File)
• CHG: is() and is_error() should adhere to method signature by returning bool (Error Reporter)
• CHG: permit mix-case mailboxes (Manage Mailboxes)
• CHG: all “Select All” field control to mailboxes (Manage Mailboxes)
• CHG: remove domain from http domain map before disassociating from account metadata (Aliases)
• CHG: skip /proc ACL calcuations (File)
• CHG: display insufficient information message if insufficient information for bandwidth projections (Bandwidth Statistics)
• CHG: ignore special _invoke() module method when fetching caller (Error Reporter)
• CHG: Update urchin to handle new http log format (ident field contains vhost)
• CHG: Upgrade FCKEditor to CKEditor (File Manager)
• CHG: properties action (File Manager)
• CHG: suppress repeated errors (Error Reporter)
• CHG: allow subdomain creation if subdomain directory exists and html symlink does not (Web)
• CHG: prevent catch-all creation on forwarded addresses (Manage Mailboxes)
• CHG: clarify select box for local user inclusion on forwards (Manage Mailboxes)
• CHG: support singular http bandwidth log format
• CHG: clarify unlimited quota by separate input control (Manage Users, User Defaults)
• CHG: prior to assuming apnscpd is running, verify process name matches executable (apnscpd)
• CHG: permit database deletion if db matches site prefix and admin user is not privileged (SQL)
• CHG: input focus changes border color in IE7 (CSS)

Welcome to another edition of apnscp changes!  Important changes include Rails infrastructure changes from Ruby 1.8 + FastCGI to Ruby 1.9 + mod_rails, new multiple worker backend (apnscpd), CDN to distribute static content with minimal bandwidth overhead (http://getap.is), and an aggressive caching  framework to reduce bottlenecks.

mod_rails

After several custom patches, mod_rails is a possibility for our clients.  mod_rails provides the same great, secure setup experienced with FastCGI, but without the nasty suexec headaches.  Perform an upgrade from Ruby 1.8 via Development > Code Frameworks.  Gems will be migrated to Ruby 1.9 during the upgrade process.  After upgrading add the necessary .htaccess directives found in the Rails Quickstart

Ruby 1.9 is here

Easy Task Scheduling (@token)

Another nice feature coming on the heels of a planned server outage is support for crontab @ time tokens.   @reboot executes a task on start-up, which obsoletes rc.d support, @hourly executes once per hour, @daily, @weekly, @monthly, and @yearly are additional special times recognized by crontab.  Each scheduling time is guaranteed to execute once during an interval, but the minute/hour vary.  Both minutes (@hourly/…/@yearly) and hours (@daily/…/@yearly) are evenly distributed among all accounts to prevent CPU bottlenecks caused, for example, by multiple hourly tasks occurring at 12:00 AM, 1:oo AM, 2:00 AM…

New Crontab Interface

apnscpd multi-tasking backend

Finally, the nicest improvement, based upon effort, includes a bevvy of performance optimizations culminating with a client-server replacement for the single-user privileged processor necessary for carrying out filesystem management of the account… that forked on every single privileged request.  For example, loading up the File Manager may generate as many as 5 separate forks of the privileged processor “lservice” [this name is about as old as the concept apnscp itself].  Now a master distributes work to one of several concurrent workers awaiting further instruction.  Best of all no additional forking required unless a worker dies on the job.   Another area of concern included sharing caches between the privileged processor and UI.  This was solved by implementing multi-tiered memcached framework.

As a comparative benchmark, requesting directory contents within the File Manager has decreased from 250 ms per request to 17 ms – a 14x improvement.  Speeding up code wasn’t enough.  Static content is now offloaded to a separate CDN (http://getap.is – clever?) to eliminate cookies, add caching, and reduce bandwidth overhead too.  Overall the latest set of changes should produce a noticeable improvement in responsiveness and allow you to get things done faster than ever before.

Assuming that spiel wasn’t enough to put you to sleep, here is your night cap in the form of a changelog:

• NEW: Ruby 1.9
• NEW: Util_Process_Tee – read from program standard input, write to external file (replaces built-in tee from Module Skeleton)
• NEW: account delete/edit hooks
• NEW: unified global/account/session cache support shared between UI and privileged processor
• NEW: multi-process privileged processor framework, 8x throughput improvement, handles multiple requests concurrently (apnscpd)
• NEW: delete Urchin profiles/log sources (Urchin)
• NEW: stream multiple commands to privileged processor (DataStream)
• NEW: recompose()- build regular expressions from multiple subexpressions (REGEX)
• NEW: user preferences (UCard)
• NEW: FTP bandwidth usage
• NEW: Styled, scrollable control with mousewheel support based upon jQuery Slider (Manage MySQL)
• NEW: raw PHP code interpreter (CLI)
• FIX: 201X backup rotation (Database Backup)
• FIX: printf named argument parsing (%(foo)s, %(bar)d) (Process)
• FIX: path transation corruption (File)
• FIX: bogus worker starvation response (apnscpd)
• FIX: Mode fixation on first auth handler triggered on start (Auth)
• FIX: non-interactive auth requests fail (Auth::CLI)
• FIX: session fixation (Auth)
• FIX: reset buffer status during worker initialization (apnscd)
• FIX: ftp usernames reported as site id (Bandwidth Logging)
• FIX: escape shell special characters in Urchin passwords
• FIX: apnscp version is not cached due to incorrect key (Page Template)
• FIX: backtrace argument corruption (Error Reporter)
• FIX: purge file metadata cache on chmod/chown (File)
• FIX: Modifying DNS cache duplicates parameter values consisting of multiple, space-delimited tokens (DNS)
• FIX: interrupt user creation process if usermod fails (User)
• FIX: multiple billing invoices result in malformed query (Billing)
• FIX: Profile changes result in “invalid token” error (Client Referrals)
• FIX: DNS fails to propagate on multi-homed accounts (DNS)
• FIX: uninitialized class reference in logfile callback (Add Subdomain)
• FIX: IE6 modal header formatting (Modal)
• FIX: refer to forms explicitly by name rather than position (SPF Wizard, Addon Domains)
• FIX: action icons are hidden in IE7 (CSS)
• CHG: result from apnscpd query returns return value instead of an object consisting of the return value among other unused flags (DataStream)
• CHG: support multiple domains (FrontPage)
• CHG: size-based rotation defaults to MB (Logrotate)
• CHG: expand crontab times to accept hourly/daily/weekly/monthly/yearly/startup task times (Cronjob)
• CHG: RESET flag flushes worker cache before processing a job (apnscpd)
• CHG: sudo uid no longer mandatory, defaults to current uid of worker (Process::Sudo)
• CHG: increase socket read buffer from 4k to 8k (apnscpd)
• CHG: permit multiple A/AAAA DNS records for a given subdomain/RR (DNS)
• CHG: Add reset flag to force apnscpd session reinitialization (DataStream)
• CHG: dump auth profile instead of session data since class data is now extensively cached within sessions generating enormous bug reports (Error Reporter)
• CHG: move archive extensions out of the constructor to enable archive recongition between cached module deserialization routines (File)
• CHG: Performance: remove call_user_func_array() and invoke afi::__call() directly
• CHG: Performance: save loaded modules between page requests (afi)
• CHG: add deferrable/non-deferrable Javascript (Template)
• CHG: notify if upgrade for Rails is available (Code Frameworks)
• CHG: Reduce module skeleton footprint.  Migrate call_proc* functions to Util_Process_* classes, pam_* to Util_Pam, break inheritance from DataStream (Module Skeleton)
• CHG: Page load improvements: use CDN for static assets, move /libjs to /js, flush HEAD immediately (Template)
• CHG: Rename file_create_symlink to file_symlink (File)
• CHG: adding a DNS record with no parameter raises an error (DNS)
• CHG: Performance: lazy load class name changes (Conf::Util, Process)
• CHG: Performance: optimize filesystem stat cache flushing frequency (File)
• CHG: Performance: determine account info freshness from domain info mtime (Account Info)
• CHG: Performance: cache get_users() response (User)
• CHG: Performance: reset account metadata on login if current/ mtime does not match saved value (Auth::Account Info)
• CHG: migrate user/account information to separate objects (User_Info, Account_Info)
• CHG: check if database exists before creation (SQL)
• CHG: sort search results based upon page view frequency (Search)
• CHG: clarify formatting of password reset e-mail
• CHG: show full help on first page view (Page Template)
• CHG: Performance: stat()- cache all directory entries on first run to improve response of related directory queries (File)
• CHG: cleanup aliases module
• CHG: obsolete common_*news* news functions, use http://updates.apisnetworks.com (Common)
• CHG: extract()- skip chown operation on empty archives (File)
• CHG: Ignore deprecation notice in Net/Whois.php (DNS)
• CHG: Separate MySQL Manager into 2 distinct views, populate only active user/database editor metadata (MySQL Manager)
• CHG: cacheless session switching between clients (CLI)
• CHG: redirect /favicon.ico to /images/favicon.ico for broken browsers
• CHG: update upload progress bar to apnscp 2 (File Upload)
• CHG: disable postback indicator triggered on login via session id (Dashboard)
• CHG: remove tooltip MySQL cache as it is now part of the short-term session cache
• REM: italicized property from hinted input elements (CSS)
• REM: get_php_version()- PHP4 information (Common)

(and by hefty I mean 2 months of unposted changes…)

There is good news to report though – a significant chunk of the changes were minor fixes to resolve regressions that appeared during rapid development of apnscp 2.   Even better – apnscp now has a very limited application lookup tool.

Access the search box either by mouse or as a shortcut junkie with the S access key, type in a few letters, hit enter to confirm the choice, and voila!  Zee app is loaded!

A notable mention, a long-standing bug preventing RubyGems from being upgraded via gem update --system has been patched, but unofficial until today.

Initial release of the inline search feature

ADD: inline search against application titles and taglines (Template)
ADD: proper 404/550 error pages (Template)
ADD: AJAX app method requestor, queries a public method of an application (apnscp)

FIX: rename operation does not work for subsequent subdomains (Subdomains)
FIX: tooltip displays undefined for “Administrative E-Mail” (Dashboard)
FIX: domain is not forgotten if omitted (Login)
FIX: login redirects only to dashboard (Auth UI)
FIX: updating credit card did not commit to database (Change Billing)
FIX: removing subdomain through Delete action fails (List Subdomains)
FIX: create_referral_profile()- missing invoice (Billing)
FIX: changing credit card details always results in failure (Billing)
FIX: bandwidth displays 0% (Template)
FIX: undefined Majordomo variables elicit warning (Majordomo)
FIX: referral profile is not synchronized to slave in time for reading (Billing)
FIX: footer is positioned at bottom
FIX: various IE6 rendering fixes – footer, login, postback “Hide Details” (UI)
FIX: prevent Overview event listener from handling propagated events (Page Template)
FIX: editing a DNS entry in an RR group triggers the edit for the respective position in the first RR group (DNS)
FIX: SPF Wizard does not warn of overwriting records (SPF Wizard)
FIX: add link to display tickets older than 90 days (Tickets)
FIX: browser detection returns spurious results (Page Container)
FIX: function permissions are not exported (SOAP)
FIX: WSDL generation exports uncallable methods (SOAP)
FIX: spurious free space report in storage gauge for users over quota (Template)
FIX: File Tree cancels first directory selection (File Tree)
FIX: methods reported missing within a module signature are not verified (afi)

CHG: parse “REFUSED” nsupdate response (DNS)
CHG: include default rbconfig.rb to allow gem update –system to complete sucessfully.  Make a symlink from /usr/bin/ruby to /usr/local/bin/ruby as well to clear up pathing by specifying a bindir of /usr/local/bin
CHG: referral system provides a one-time lump sum of $5, $10, $20, or $30 (Minimalist -> Super)
CHG: rename get_commissions() to get_referrals() (Billing)
CHG: omit 4 KB directory size from File Manager
CHG: omit parsing of @replies in Twitter feed (Dashboard)
CHG: consolidate frontend/backend functions (Majordomo)
CHG: increase base font size from 13px to 14px (UI)
CHG: remove extra service configuration information, update gauges (Manage Users)
CHG: remove unnecessary billing information from referral profile (Client Referrals)
CHG: display an excerpt of the latest news (Dashboard)
CHG: pulse postback status once for success, twice for warning, and thrice for error (Page Template)
CHG: lessen intensity of green table header for long tables (head1 pseudoclass) (CSS)
CHG: refactor delete() to stat just the parent directory (File)
CHG: improve stat() response for multiple lookups of the same file by statically caching before offloading to APC (File)
CHG: cache template configuration
CHG: fire postback status pulse on postback only (Template)
CHG: update disk space gauges (Quota Tracker)
CHG: use local session cache instead of global cache for tooltips (Tooltip)
CHG: rewrite user behavior logger for applications (Stats Logger)
CHG: promote referral statistics to dashboard (Dashboard)

REM: Legacy Help is irrelevant

(phew! nearly 50 changes)