curl

26 Oct

PHP and curl_multi_exec

Posted by matt in curl, performance, php, programming

This post explain how to get data off of a curl_multi handle. Some time back I posted this snippet of code inside of a larger sample of code:

<?php
  $active = NULL;
  do {
    $ret = curl_multi_exec($multi, $active);
  } while ($ret == CURLM_CALL_MULTI_PERFORM);
 
  while ($active && $ret == CURLM_OK) {
    if (curl_multi_select($multi) != -1) {
      do {
         $mrc = curl_multi_exec($multi, $active);
      } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
  }
?>

I didn't really document or explain it. And so it seems that this code snippet has caused some confusion. Let me explain what it does.

18 Jun

Connection Sharing with CURL in PHP: How to re-use HTTP connections to knock 70% off REST network time.

Posted by matt in curl, drupal, hpcloud, performance, php, programming

The PHP CURL library is the most robust library for working with HTTP (and other protocols) from within PHP. As a maintainer of the HPCloud-PHP library, which makes extensive use of REST services, I've been tinkering around with ways of speeding up REST interactions.

What I've found is a way to cut off nearly 70% of the processing time for a typical usage scenario. For example, our unit tests used to take four minutes to run, and we're now down to just over a minute, while our Drupal module's network time has been cut by over 75%.

This article explains how we accomplished this with a surprisingly simple (and counter-intuitive) modification.

15 Jan

OS X: Using curl instead of wget

Posted by matt in bash, curl, mac, os x, wget

OS X does not come with wget, a command-line tool for retrieving websites. For a while, I grumbled about this. I knew that curl was installed, but I hadn't ever used curl from the command line. But once I tried it out, I realized that for my needs, curl is just as good as wget... and I don't have to install anything extra to get it.

Here's how to use curl to fetch a remote URL:

$ curl -OL h ttp://spine-health.com/index.php
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 34646    0 34646    0     0  15314      0 --:--:--  0:00:02 --:--:-- 17767

This will download the remote file and store it locally in index.php. If you leave off the -O, it will write the file to standard output (your terminal, usually).

curl: Remote file name has no length!

Fetching from a URL's root can behave differently. If you perform the same command as above, but pointing to the base URL, you will get an error:

$ curl -OL h ttp://spine-health.com/
curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information

What's going on here? The error is not terribly informative on this point. The problem is that curl doesn't know where to write the output file. A better command here is something like this:

$ curl -L h ttp://spine-health.com/ > out.html

This time, the retrieved data will be written to the out.html file.

Like wget, curl has many options. You can read the man page for details, or you can get a quick summary with the usual curl --help command.