Zend Optimizer removal, switch to ionCube ASAP if used

After 24 hours of tweaking and logging, the cause looks to be directly attributable to Zend Optimizer.  As a result, we are recommending all customers at this time to switch their encoded applications from Zend Encoder to ionCube.  ionCube does not exhibit the same issues and after a few hours of exclusively running ionCube + eAccelerator and ionCube + APC, both have performed admirably.

Zend Optimizer tends to result in crashes in clumps, e.g.:

[Thu Jun 12 13:16:24 2008] [notice] child pid 17624 exit signal Segmentation fault (11)
[Thu Jun 12 13:18:05 2008] [notice] child pid 27997 exit signal Segmentation fault (11)
[Thu Jun 12 13:18:05 2008] [notice] child pid 27999 exit signal Segmentation fault (11)


[Thu Jun 12 14:04:08 2008] [notice] child pid 30838 exit signal Segmentation fault (11)
[Thu Jun 12 14:04:09 2008] [notice] child pid 6332 exit signal Segmentation fault (11)


[Thu Jun 12 15:30:18 2008] [notice] child pid 17317 exit signal Segmentation fault (11)
[Thu Jun 12 15:30:18 2008] [notice] child pid 18367 exit signal Segmentation fault (11)


[Thu Jun 12 15:45:19 2008] [notice] child pid 28589 exit signal Segmentation fault (11)
[Thu Jun 12 15:45:42 2008] [notice] child pid 15373 exit signal Segmentation fault (11)


[Thu Jun 12 16:45:03 2008] [notice] child pid 31388 exit signal Segmentation fault (11)
[Thu Jun 12 16:45:19 2008] [notice] child pid 19022 exit signal Segmentation fault (11)
[Thu Jun 12 16:45:19 2008] [notice] child pid 22038 exit signal Segmentation fault (11)
[Thu Jun 12 16:45:45 2008] [notice] child pid 19016 exit signal Segmentation fault (11)
[Thu Jun 12 16:45:45 2008] [notice] child pid 5576 exit signal Segmentation fault (11)

And so on.  This is a snippet from Borel, which appears to be all over the map in terms of crashes.  Other servers aren’t as frequent in the rate of crashes (2 every 4 – 6 hours).  There is likely some suspect code that is evaluated by Zend causing the issues, but having noted long-standing reliability issues with Zend in the past, it’s safe to say we need to put that extension to rest.

Now, as an added perk, we will be able to deploy APC, which has upload byte logging support.  Previously, APC and Zend Optimizer cannot co-exist (well they could, but every request segfaults at the end…)  eAccelerator and Zend Optimizer can, which is why I originally chose that solution.  Not withholding, APC was also extremely buggy during the initial round of software evaluation in November ’06.

If these results remain as promising as they have been during the first few hours of testing, the servers will be running ionCube and APC by Monday morning.  We can finally put the erratic restarts to rest on Augend and Borel.

– Matt

Update: Oh, and reminds me — if my thought process is right, then it is also likely the cause of the odd phenomenon of blank PHP pages that hit less frequently than lottery winners.  Assume that eAccelerator needs to compile the page (exceeded TTL or new code), finishes servicing the request, then reserves an inode on the filesystem to store the compiled code.  After the inode is reserved, or block in memory, or whatever to signal the code is about to be stored, the shutdown hook calls Zend Optimizer which in turn crashes.  This results in an opened inode or section in shared memory with no data written by a buffer flush to disk/memory as Apache has prematurely died thus leading to… the blank page.

(Purely speculation, but a possibility.)