Setting up APC and Memcache on Dreamhost to support Drupal performance improvements

 

I've been working on the performance of my websites, all of them Drupal powered, and all hosted on Dreamhost VPS. The latest step was to move over to PHP 5.3, use APC, use memcache, use the Drupal module for memcache, and oh I also installed the "upload progress" PECL module along the way to get a status report entry to shut up. There are two ways to get better performance, right? One is to buy more servers and install a distributed infrastructure. The other is optimizing the software. A recent podcast from Lullabot said "APC, Memcache, and Varnish". I've done the first two and not seen much performance gain, and unfortunately installing Varnish looks to be too complicated (on Dreamhost).

Installing PECL extensions like APC on Dreamhost

Out of the box a Dreamhost VPS can be configured pretty readily to support Xcache. Actually the Lullabot people in their podcast did say maybe it didn't matter which opcode cache you use, but that it's APC they had the most experience with. The point is to use an opcode cache because out of the box PHP reparses and recompiles into opcodes every file on every page load. This is a lot of CPU power that could be put to better purposes such as rendering pages. Opcode cache's save the time of parsing and compiling PHP, by saving away the compiled state of each PHP file.

The first thing to do is switch your domains over to PHP 5.3 FastCGI. The reason is apparent on their Wiki, http://wiki.dreamhost.com/Custom_PHP.ini .. just look at the size of the sections on customizing PHP 5.2 and 5.3. Customizing PHP 5.3 is trivial versus the process to customize PHP 5.2.

The method is

  • in the Configure Server control panel click the "disable mod_php" checkbox,
  • in each domain hosted on the server, edit the domain and choose "PHP 5.3 FastCGI"
  • restart the server

We disable mod_php because we're instead using FastCGI and as it says on the control panel, it'll save a lot of memory.

Then to install APC itself, carefully follow these instructions: http://wiki.dreamhost.com/APC

The procedure will be followed for the other PECL modules, so it's worth going over it as a generalized procedure.

The following steps must be performed as the admin user of your server. So, ssh into your server as the admin user and type the following to get a root shell: sudo bash

Go to http://pecl.php.net and search for the PECL module you desire. (in this case APC, but later we'll do it for memcache and uploadprogress) Go to the PECL module page and you'll see a list of tarballs. What I do is right-click on the link to the tarball, the popup menu has a choice reading something like "Copy Menu Location". That will copy the tarball's URL onto the clipboard. Then do this:

 

# cd /usr/local
# wget ...past in the URL...
# tar xvfz .... the name of the file just downloaded

 

I'm thinking it would be better to unpack and build the PECL extensions in /usr/local rather than in a home directory. Reading Dreamhosts wiki page for APC suggests you should unpack the PECL extension in a home directory. It's up to you where you put the module source.

Now, prepare the module source to be built.

 

# cd ... the directory created when the tarball was unpacked
# /usr/local/php53/bin/phpize
# ./configure --with-php-config=/usr/local/php53/bin/php-config

 

Notice we have to be careful to use the PHP 5.3 versions of these commands. I tried once building without the PHP 5.3 versions and the server failed with this message in error.log:

 

# tail -60 error.log
[Mon Mar 28 07:47:33 2011] [notice] mod_fcgid: call /home/robogeek5/visforvoltage.org/index.php with wrapper /dh/cgi-system/php53.cgi
PHP Warning: PHP Startup: apc: Unable to initialize module
Module compiled with module API=20060613
PHP compiled with module API=20090626
These options need to match
in Unknown on line 0
PHP Warning: PHP Startup: memcache: Unable to initialize module
Module compiled with module API=20060613
PHP compiled with module API=20090626
These options need to match
in Unknown on line 0

 

It took awhile for it to sink in what that was saying. Basically it said the PECL extensions had been compiled with the PHP 5.2 toolchain, rather than the PHP 5.3 version.

The next step is to build the module:

 

# make

 

Do not run "make install". It doesn't need to install anything, it's fine right where it is.

The next step is to create a file named (in this case) /etc/php53/conf.d/apc.ini reading

 

$ cat /etc/php53/conf.d/apc.ini
extension = /usr/local/APC-3.1.6/modules/apc.so
apc.rfc1867 = 1

 

There are a lot of options you can configure for APC, I'm sure, but that's the basic idea. The extension= line tells PHP where to go. That is, where to go to get the extension.

This also demonstrates just how nice PHP 5.3 is to configure. The way Dreamhost had us modifying the configuration on PHP 5.2 was nothing short of maddenly confusing. Now, we just put a file in the conf.d directory and it's automagically recognized. Way cool.

Finally, restart the web server which you can do from the control panel or since you're already in a root shell you can do it the old fashioned way:

 

# /etc/init.d/httpd2 restart
Stopping apache2-ps47533 webserver....done.
Starting apache2-ps47533 webserver...starting...ok......done.

 

Installing APC and memcache Debian packages on Dreamhost VPS

The Dreamhost VPS uses a modified version of Debian. There are many Debian packages you can just install using apt-get.

For this project install php-apc, memcached, and php-memcache. You should result in somehting like this:

 

[psXXXXX]$ dpkg -l | grep apc
ii php-apc 3.0.19-2 APC (Alternative PHP Cache) module for PHP 5
[psXXXXX]$ dpkg -l | grep upload
[psXXXXX]$ dpkg -l | grep memcache
ii memcached 1.2.2-1+lenny1 A high-performance memory object caching system
ii php5-memcache 3.0.1-1 memcache extension module for PHP5
[psXXXXX]$

 

Installing APC, memcache, uploadprogress PECL extensions on Dreamhost

With the above instructions it should be pretty straightforward to end up with this:

 

[psXXXXX]$ ls /etc/php53/conf.d/
apc.ini memcache.ini php.ini uploadprogress.ini
[psXXXXX]$ ls /usr/local
APC-3.1.6 bin frontpage lib memcache-2.2.6.tgz package.xml php53 share uploadprogress-1.0.1
APC-3.1.6.tgz dh games memcache-2.2.6 miva php5 sbin src uploadprogress-1.0.1.tgz
[psXXXXX]$

 

With APC and uploadprogress installed there is a message on the Drupal status report page which will turn from red to green.

With APC installed your opcodes will be cached and as I said above, a performance improvement. I watched various things like memory load and CPU utilization carefully and they didn't change. However the sites feel snappier. If you really care to have demonstrable proof your work did some good, then perhaps benchmarking tools like httpperf will help.

Adding the Drupal memcache module

Project page: http://drupal.org/project/memcache

You just drop it in your sites/all/modules folder, unpack, and enable the module. There is a memcache_admin module that provides a user interface for settings.

Enabling it using Drush can give some error messages. First, drush will insist on the settings.php being set up to enable the memcache API. Second you might get this message:

 

[ps42313]$ drush en memcache_admin
You must enable the PECL memcached or memcache extension to use memcache.inc. [error]
You must enable the PECL memcached or memcache extension to use memcache.inc. [error]

 

This message will only appear if you didn't do something right in the previous steps.

For the required changes in settings.php, follow the documentation on the project page.

I should make a warning from the voice of experience. Do not, I repeat, do NOT, configure two sites to store their cache in the same memcached. The results are strange.

If you should happen to do that, disable the memcache modules on all sites but one, and clear the cache on all sites. I haven't yet gone to setting up multiple memcached's for the multiple sites.

"NOTE: As of 6.15, it is possible to efficiently run memcache in one daemon, shared by all bins. You can safely ignore advice found elsewhere on the internet that advises one memcached daemon per bin. In terms of reporting, there are still some advantages to having more daemons."

The default memcached configuration provided by Debian gave me this: /usr/bin/memcached -m 64 -p 11211 -u nobody -l 127.0.0.1

I haven't yet determined how to or whether to optimize this further. Obviously the important thing here is the cache utilization (how many of requests are satisfied by the cache) and whether the cache is too big or too small. A quick look over the memcached website (http://memcached.org/) doesn't reveal a simple way to learn vital statistics like cache utilization or cache hit rates.

Ensuring drush is run with PHP 5.3

The Dreamhost VPS is delivered with the "php" command being php5.2. But with the instructions on this page we're using PHP5.3. It may not be a problem for you, but for me because php5.2 didn't have the customizations in php5.3 drush commands failed. For example earlier we saw that "drush en memcache" would fail if the memcache PECL were not configured.

You can run drush with php5.3 this way:

 

$ export DRUSH_PHP=php-5.3
$ drush en memcache_admin
The following extensions will be enabled: memcache_admin
Do you really want to continue? (y/n): y
memcache_admin was enabled successfully.

 

You can also make this a permanent change this way:

 

[psXXXXX]$ cat .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
export PATH=$HOME/drush:$HOME/bin:${PATH}
export DRUSH_PHP=php-5.3
[psXXXXX]$ cat .bash_profile
# ~/.bash_profile: executed by bash(1) for login shells.
umask 002
PS1='[\h]$ '
. $HOME/.bashrc