Installing xhprof and XHGui on a Dreamhost Ubuntu 12.04 VPS to diagnose Drupal 6 performance
One of my websites has been running very slow for years - every so often I try to figure out why it's slow, and recently it had been suggested to install xhprof to gather some data. The website is a Drupal 6 site that gets 1000+ visits a day, and is an active forum website with lots of people chatting away about electric vehicles. The server is a VPS rented from Dreamhost. The latest iteration of Dreamhost VPS's uses SSD disks, and a customized version of Ubuntu 12.04. The database is Dreamhost's canned MySQL-VPS service that I've used for years.
I finally went through the process of installing xhprof and XHGui on the VPS - and found it to be far harder than it shoulda been. In part the difficulty has to do with the low level nature of xhprof's purpose - instrumenting PHP code to collect execution data to profile where the majority of CPU time and memory consumption goes. It was made worse by the abysmal documentation for XHGui that left out steps and generally made it very hard to understand.
I did make it through and successfully get XHGui running and showing data. Therefore it seems useful to document how I did this.
Again - installing xhprof and XHGui - Ubuntu 12.04 as offered by Dreamhost (meaning it is customized), with PHP 5.3 and Drupal 6.
The first step is to install xhprof, but since that's a PECL extension we have to get PECL installed first. That should simply require
# apt-get install php-pear
But that gave an error message about a config file controlled by both php-pear and ndn-php5-cgi. I wasn't able to find a workaround and ran this:
# apt-get remove ndn-php5-cgi
After which php-pear can be installed. Next:
# pecl search xhprof
# pecl install xhprof
# pecl install xhprof-beta
The last is necessary because xhprof is still in beta.
But an error message gets printed that, upon investigation, turns out to be a matter of /tmp being mounted with the noexec flag. Dreamhost obviously thinks this improves security. It might, I have no idea whether it does, but the fact is PECL wants to download scripts to a temporary directory and execute them. That's impossible with /tmp having the noexec flag.
The usual workaround is to run "mount -o remount,exec /tmp" then run the PECL command then run "mount -o remount,noexec /tmp". But that gives an error message as well.
The other idea which turned up is to configure PECL to use a different temporary directory, one which doesn't have noexec.
# mkdir -p /var/tmp/pear/temp
# pecl config-set temp_dir /var/tmp/pear/temp
That should have done the trick but there was another error message. I don't remember what that message was about - but the result of searching turned up the fact that PECL uses PEAR settings if available. So we simply change the command to this:
# pear config-set temp_dir /usr/local/tmp/pear/temp
Then the "pecl install" command works as it's supposed to.
The next step of installing xhprof is configuring the system to use it. That means telling the Apache server about the extension.
The usual recommendation is creating this file: "/etc/php5/mods-available/xhprof.ini" But on Dreamhost's Ubuntu we need to create this file instead:
# cat /etc/php53/conf.d/xhprof.ini
Notice the hard-coded path. That was the only xhprof.so I could find.
XHGui also requires mcrypt, so run this:
# apt-get install php5-mcrypt
The next major dependency is a MongoDB instance. With Dreamhost that's simply a matter of going to the VPS control panel and making MongoDB active. The control panel sets everything up then sends you an email with details including username and password.
However, there are a couple more things to set up.
# pecl install mongo
This gives us some MongoDB PHP classes. And then we need to install this .ini file:
# cat /etc/php53/conf.d/mongo.ini
# cp /etc/php53/conf.d/mongo.so /etc/php54/conf.d/
# cp /etc/php53/conf.d/mongo.so /etc/php55/conf.d/
# cp /etc/php53/conf.d/mongo.so /etc/php56/conf.d/
Finally, while Dreamhost installed MongoDB it didn't install the MongoDB client applications:
# apt-get install mongodb-clients
Because we've modified the webserver configuration the server must be rebooted for it to take effect.
You can validate that the extensions are installed two ways.
# php-5.3 -m
Lists the installed extensions - simply look for the MongoDB and xhprof to be listed.
Second you can put a phpinfo.php file (containing "<? phpinfo(); ?>") in the webroot of a virtual host to inspect the PHP settings.
XHGui purports itself to provide useful reports from data collected by xhprof. By itself xhprof gives a pile of data, but as always to make use of that data means making it understandable.
You get XHGui from their github repository. The README gives some installation documentation which is marginally above useless. This isn't your typical "unpack the PHP tarball in the root of a virtual host, then visit install.php in the browser".
You'll want to download their tarball and unpack it in the root of your home directory.
It's best to set up a virtual host - say, xh.example.com - for accessing XHGui rather than melding it into the middle of your website. Why? I think that's pretty obvious - it's like that "don't cross the streams" thing from Ghostbusters.
But the virtual host needs to point to the "webroot" directory inside the XHGui distribution. There's some sensitive files in the main directory of the XHGui distribution, such as config/config.php which will end up containing the MongoDB password. With the Dreamhost control panel I made sure the "Web Directory" refers to the webroot subdirectory.
Next ensure the cache directory is writable
$ chmod -R 0777 cache
Next, set up the config file. There is a sample config/config.sample.php, and it's easy enough to copy that to config/config.php and edit to suit. There's a number of things you can do with that file, but all I found necessary was to enter the MongoDB user/password
// Allows you to pass additional options like replicaSet to MongoClient.
// 'username', 'password' and 'db' (where the user is added)
'db.options' => array(
'username' => '... user name from Dreamhost ...',
'password' => '... password from Dreamhost ...'
The rest seems fine to leave as it is.
For Dreamhost and most hosting providers I'm familiar with, the mod_rewrite module is already installed and enabled in the Apache webserver. If not, you'll have to enable this for the virtual host you're using.
The install script is now run at the command line (rather than by visiting a web site)
$ cd path/to/xhgui
$ php install.php
If all has gone well no errors will be printed. Otherwise it'll print something - and I found their error messages to be helpful.
With this set up you can visit the virtual host you set up for XHGui (xh.example.com). But you'll be greeted by a screen saying no data is being collected. That's because we haven't enabled anything to collect data using xhprof.
The XHGui package comes with a script, path/to/xhgui/external/header.php, that's meant to be used along with the auto_prepend_file command to execute before any PHP code runs. That script enables xhprof hooks and ensures the data is gathered into the MongoDB database.
The way I did this on my VPS is with this file:
# cat /etc/php53/php.ini.local
auto_prepend_file = "/path/to/xhgui/external/header.php"
There are other methods but this works under the constraints of Dreamhost's VPS. For good measure I appended that text to /etc/php53/php.ini as well.
Another reboot is necessary to ensure the webserver picks up the config change. After which the XHGui virtual host should start picking up data. It's suggested you let it stew for awhile, collecting enough data so you have a statistically significant sample, before you look at the data.
Puzzling through the data is an exercise best left to the XHGui team to document. The application UI is fairly self-explanatory.
Drupal xhprof module
Before finishing this off I should mention the xhprof module for Drupal.
It's available for both D6 and D7. It's a lot easier to set up than XHGui - all you have to do is get the xhprof extension installed and enabled, then install the xhprof module and enable it inside Drupal.
Unfortunately the reporting it provides is -uh- extremely unuseful. It shows a bunch of hexcode strings, one for each "Run", each of which link to a page with details from that Run. Those details are interesting because it shows execution time and memory usage for each function. But there's no attempt to congeal the data from all "Run's" into one report. You're left browsing through hundreds of individual reports grasping at straws when what you want is for the whole haypile to be digested for you into a nice package.
XHGui purports to fulfill that function. But, I'm still puzzling over how to interpret the information XHGui tells me.