system administration

I noticed recently that on one of my Debian systems, my installation of varnish did not have any of its monitoring utilities installed. /usr/local/bin was missing varnishstat, varnishtop, varnishhist, and varnishsizes.

I re-ran configure and make a couple of times. I couldn't find any errors, yet none of these programs was ever compiled.

This explains how to use a UNIX-like command line (including Linux and OS X) and the find command to search through a subdirectory and find all of the files with a certain extension, and then combine those all into one file. Surprisingly, this isn't a difficult task. It can be accomplished with one command on the command line:

$ find ./src -name '*.txt' -exec cat '{}' \; > test.txt

The above looks through everything in the ./src directory (including all subdirectories) for any files with the .txt extension. Each file it finds, it adds to test.txt. So at the end of the command's run, all of the text files will be combined together into text.txt. You can use this strategy to easily combine lots of files into one.

Using find, it's easy to customize the command above to do all kinds of things with files. I gave a few examples in an earlier post about the UNIX find command.

Often times, I want to create a full directory structure, and I'd like to do it with just one call to mkdir. That is, I want to create a root directory and multiple subdirectories all at once. Here's how to do this.

mkdir -p myProject/{src,doc,tools,db}

The above creates the top-level directory myProject, along with all of the subdirectories (myProject/src, myProject/doc, etc.). How does it work? There are two things of note about the command above:

• The -p flag: This tells mkdir to create any leading directories that do not already exist. Effectively, it makes sure that myProject gets created before creating myProject/src.
• The {} lists: The technical name for these is "brace expansion lists". Basically, the shell interprets this as a list of items that should be appended individually to the preceding path. Thus, a/{b,c} is expanded into a/b a/c.

You can nest brace expansion lists. That means you can create more complex sets of subdirectories like this:

mkdir -p myProject/{src,doc/{api,system},tools,db}

Notice that this creates two directories inside of doc/.

I have recently been working on configuring a business-class Comcast SMC router to make use of a group of 5 static IPs. The documentation I found was sparse, and I spent a few days figuring out how to do this.

Turns out that it is very simple.
Configuring the Firewall

Our current environment uses AWStats to analyze our HTTP server log files and build reports. Because it has privileged access to our data, and because it is open source, we can glean more information out of it than we could from proprietary hosted analytics platforms.

It turns out that there is a PHP front-end to AWStats (called BetterAWStats) that comes complete with a Drupal module. Here, I explain how we've installed and configured this module to get our AWStats data imported into our Atrium server.

I noticed something unexpected in my nginx logs today: There were a bunch of 499 HTTP codes in the access log. Oddly, these didn't show up in Google Analytics, there were no corresponding errors in the error log, but they did show up in my AWStats. What's the deal?nginx

The answer is pretty simple: Nginx uses 499 as the status code when the client unexpectedly terminates a connection. (Thus the client may have already received a 200 in the header, AFAIK). This is consistent with the usage of 4xx errors as indicating a client error condition.

A quick calculation showed me that the 499s accounted for only 0.2% of our total traffic. Not bad. And in fact, I sorta like the ability to see how many times clients terminated connections to my server.

Every once in a while, I have some project that requires working with one of GoDaddy's servers. By far, the biggest frustration for me when dealing with GoDaddy is getting MySQL databases uploaded. I've tried all kinds of crazy tricks, from exporting MySQL databases in "bite sized chunks" to writing SQL processors that break large imports into smaller ones.The _db_backup directory

But today I think I found the Right Solution (TM) to the problem: Use GoDaddy's database restoration tool to load a large SQL file. Basically, instead of treating this as an initial import, we treat it as a restoration from a backup file.

Here's how you do it.

In Jauary we migrated a Drupal site that routinely has 40k+ hits per day. We moved the site from servers in the Pacific Northwest to a datacenter in Virginia. As if that wasn't enough, we moved the servers from Apache to Nginx, as well. But what makes this remarkable to me is that we managed to pull this off without so much as a minute of downtime. This blog explains how we did it (and it uses lots of pretty diagrams, too!).

Tuning Nginx ("engine-X") seems to be something of a black art. Today, I looked closely at the tcp_nopush, sendfile, and keepalive_requests settings for pages rendered from PHP as a FastCGI, and memcached content. We discovered that with a little careful tuning, we could shave off as much as 200-400 msec per request.

I have been working on several speed improvements on the Condition Centers at Spine-Health.com. Initially, these pages were taking upwards of 3.5 seconds just to render the HTML. Through a series of optimizations that I will document in another article, we have the conditions page rendering in around 100 msec now.

Before we get going, let me mention a few details of our system:

• We are running CentOS 5.3 (roughly equivalent to RHEL 5.3)
• We are running Nginx 0.6, which is behind the current stable, but is the latest in the Fedora EPEL repositories that we use.
• Since these settings make use of low-level kernel facilities (like TCP_UNCORK), other platforms may differ.

The typical enterprise-grade publishing model for web sites typically has two or three tiers. Content is created and edited on a staging server, where it is carefully reviewed before publishing it to the live server. Additional development may even happen on a third ("development") server.