January 2010

The Second in my three-part series on Drupal an OpenAmplify has been published on their community site. If you missed the first part, you may want to start there. Part three, coming soon, will cover the API, and will focus on development instead of configuration.
Part 2
In part two, I walk through the process of building a "mini portal" by taking semantic information returned from an OpenAmplify analysis of a node, and using that information in conjunction with other web services. For this demonstration, I released a new version of the module, and added support for Shopping.Com and Bloglines, both of which can return some impressively rich content.

It just came to my attention that a WebMonkey article (Parsing HTML? There's an App for That) from a few months ago suggested using QueryPath as an alternative to attempting to parse HTML by hand.

Webmonkey on QueryPath

Appropriately, last week I wrote a QueryPath script to analyze a site and extract all links so that I could feed them to Siege and simulate something like a real load against a server. It's nice to be able to easily extract data from HTML.

OS X does not come with wget, a command-line tool for retrieving websites. For a while, I grumbled about this. I knew that curl was installed, but I hadn't ever used curl from the command line. But once I tried it out, I realized that for my needs, curl is just as good as wget... and I don't have to install anything extra to get it.

Here's how to use curl to fetch a remote URL:

$ curl -OL h ttp://spine-health.com/index.php
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 34646    0 34646    0     0  15314      0 --:--:--  0:00:02 --:--:-- 17767

This will download the remote file and store it locally in index.php. If you leave off the -O, it will write the file to standard output (your terminal, usually).

curl: Remote file name has no length!

Fetching from a URL's root can behave differently. If you perform the same command as above, but pointing to the base URL, you will get an error:

$ curl -OL h ttp://spine-health.com/
curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information

What's going on here? The error is not terribly informative on this point. The problem is that curl doesn't know where to write the output file. A better command here is something like this:

$ curl -L h ttp://spine-health.com/ > out.html

This time, the retrieved data will be written to the out.html file.

Like wget, curl has many options. You can read the man page for details, or you can get a quick summary with the usual curl --help command.