June 2009

TweetyPants.com is featured on Twitdom, a site dedicated to interesting Twitter applications.

A week ago, I was introduced to Visor, a utility for turning the Mac OS X console into a "Quake-style" console. What does that mean? In a nutshell, it means that the terminal runs not as a normal window, but as a boderless window attached to the top of the screen. Hit a button and the console slides down. Hit the button again (or unfocus from the window) and up it slides. It's a convenient way of keeping a terminal accessible at all times. (And, yes, it supports multiple tabs.)
Visor: "Official" screenshot

Visor is written in Ruby, and is easy to install (though the installation process is a little more, ahem, manual than your typical Mac OS application.

Tip: By default, Visor slides away when the window loses focus. Sometimes, though, it is desirable to keep the visor down for short (or perhaps long) periods while working in another window. To do this, expand visor and then click on the visor terminal icon on the right side of your menu bar. Select "pin" and the visor will stay open. To set it back to its regular behavior, select "unpin".

Thanks to sdboyer for turning me on to Visor.

DrupalEasy interviewed me about my latest JavaScript book, my work in Drupal, and so on.

Matt Butcher, author of Drupal 6 JavaScript and jQuery from Packt Publishing talks with Mike about the book and numerous topics regarding Drupal, JavaScript, and jQuery. [...] The interview covers Drupal behaviors and its JavaScript templating system as well as some of the other ways Drupal developers can leverage JavaScript.

http://drupaleasy.com/podcast/2009/06/drupaleasy-podcast-11-interview-au...

Earlier this month I posted a short example of a compressor I was working on for QueryPath. I received a couple of very helpful comments that pointed to a PHP built-in library that I didn't know about: tokenizer.

The tokenizer is just what I needed. It parses arrays of PHP code and returns arrays of token information. Since it does the hard work of breaking up a file into meaningful chunks, building a compressor on top of it is trivially easy.

As you may recall, the goals of the compressor are:

• Pack all of the PHP files in a package into one PHP file.
• Remove all of the comments
• Get rid of extraneous whitespace
• Keep the output in pure PHP (i.e. don't require any additional processing steps to interpret the file)

With the tokenizer, we can accomplish these goals in about sixty lines of code.

Note that compressed code is not likely to result in huge execution speed improvements. When presented with this fact, several people have asked me why anyone would want to compress their code. I have done this as a result of requests to make a tiny single-file QueryPath distribution. I suppose use-cases for compact libraries range from embedded systems to faster downloads to simply smaller overall codebases.

Here's the revised code that performs the same sort of compression as the previous article. Note that I have also generalized it and made it into a shell script.

<?php
/**
 * Compact PHP code.
 *
 * Strip comments, combine entire library into one file.
 */
 
if ($argc < 3) {
  print "Strip unecessary data from PHP source files.\n\n\tUsage: php compactor.php DESTINATION.php SOURCE.php";
  exit;
}
 
 
$source = $argv[2];
$target = $argv[1];
print "Compacting $source into $target.\n";
 
include $source;
 
$files = get_included_files();
 
$out = fopen($target, 'w');
fwrite($out, '<?php' . PHP_EOL);
fwrite($out, '// QueryPath. Copyright (c) 2009, Matt Butcher.' . PHP_EOL);
fwrite($out, '// This software is released under the LGPL, v. 2.1 or an MIT-style license.' . PHP_EOL);
fwrite($out ,'// <a href="http://opensource.org/licenses/lgpl-2.1.php'" title="http://opensource.org/licenses/lgpl-2.1.php'">http://opensource.org/licenses/lgpl-2.1.php'</a>);
fwrite($out, '// <a href="http://querypath.org.'" title="http://querypath.org.'">http://querypath.org.'</a> . PHP_EOL);
foreach ($files as $f) {
  if ($f !== __FILE__) {
    $contents = file_get_contents($f);
    foreach (token_get_all($contents) as $token) {
      if (is_string($token)) {
        fwrite($out, $token);
      }
      else {
        switch ($token[0]) {
          case T_REQUIRE:
          case T_REQUIRE_ONCE:
          case T_INCLUDE_ONCE:
          // We leave T_INCLUDE since it is rarely used to include
          // libraries and often used to include HTML/template files.
          case T_COMMENT:
          case T_DOC_COMMENT:
          case T_OPEN_TAG:
          case T_CLOSE_TAG:
            break;
          case T_WHITESPACE:
            fwrite($out, ' ');
            break;
          default:
            fwrite($out, $token[1]);
        }
 
      }
    }
  }
}
fclose($out);
?>

The main idea of this script is to read a main file and all of its included libraries and compress them all into one output file, which will function as executable PHP code.

Since most of this has been covered in the previous article, I just want to point out the section that makes use of the tokenizer.

The important part of the code above is the large foreach loop:

    $contents = file_get_contents($f);
    foreach (token_get_all($contents) as $token) {
      if (is_string($token)) {
        fwrite($out, $token);
      }
      else {
        switch ($token[0]) {
          case T_REQUIRE:
          case T_REQUIRE_ONCE:
          case T_INCLUDE_ONCE:
          // We leave T_INCLUDE since it is rarely used to include
          // libraries and often used to include HTML/template files.
          case T_COMMENT:
          case T_DOC_COMMENT:
          case T_OPEN_TAG:
          case T_CLOSE_TAG:
            break;
          case T_WHITESPACE:
            fwrite($out, ' ');
            break;
          default:
            fwrite($out, $token[1]);
        }
 
      }
    }

The main function for the PHP tokenizer is token_get_all(). This takes an array of strings and returns an array of tokens resulting from the parsing of that string. A token can be either a bare character or an array with a token identifier ($token[0]), the value of the token ($token[1]) and the line number on which the token appears ($token[2]).

What we do above is loop through the tokens in a file, and then apply this set of rules:

1. If the token is just a character, print it to the destination file.
2. Skip any tokens that are:
   • require, require_once, or include_once directives
   • comments of some sort
   • an open (<?php) or close (?>) PHP directive
3. Compress all whitespace sections into a single character. We could do some fancy additional compression, but there is really no need for present purposes.
4. All else is printed verbatim to the output file

At the end of this process, we are left with one compacted file. Here's how the above script is executed against the QueryPath library:

php ./compactor.php QueryPath.compact.php ../src/QueryPath/QueryPath.php 

This reads in four source files (QueryPath.php and the three files required) and processes them all into a single 65k output file named QueryPath.compact.php.

The source code is now hosted at GitHub. Feel free to fork: http://github.com/technosophos/PHPCompactor/tree/master

An issue in the QueryPath issue queu made me realize that parsing crufty old HTML documents is not exactly intuitive. Here's a quick tutorial for parsing HTML documents.

The most difficult part of handling documents is that it is not always clear what kind of content a document is. While some extensions (like .html and .xml) make this easy, others (like .qti, and XML format) are not so readily discernable. And files fetched from a URL may not have an extension at all!

So sometimes when QueryPath parses files, it basically guesses at the content type.

Here are the guessing rules that QueryPath follows:

1. If passed a string of markup (e.g, <foo><bar/></foo>), check whether it begins with an XML declaration. By definition, every XML file must begin with something like this:

<?xml version="1.0"?>
<sometag/>

If the declaration does not appear on the first line, the document is not to be considered XML. If this is the case, QueryPath throws it into the HTML parser.

1. If passed a file name and a context, inspect the contents. QueryPath uses the PHP stream system to handle files. One of the benefits of this is that you can pass in a context which tells QueryPath how to retrieve the document. Typically, this is used to modify HTTP parameters.

Whenever a QueryPath object is created with a context, QueryPath automatically inspects the contents of a file. Example:

<?php
require '../Code/QueryPath/bin/QueryPath.compact.php';
$url = 'http://example.com/old_crufty_html.foo';
 
$cxt = array('context' => stream_context_create());
 
// This will parse it as HTML:
print qp($url, 'title', $cxt)->text();
?>

Assuming that the URL above points to an old and crufty HTML document, this code will parse the document as HTML. And if the URL above pointed to an XML document (or an XHTML document), the XML parser would be used instead. This happens because when a context is passed into QueryPath, it checks the content to see what data type it is dealing with.

1. If passed a file name and no context, inspect the file extension. This is sort of a last-ditch attempt, and what it will do is assume that if the file ends with .html the code is HTML. Otherwise, it will assume that the file is XML.

Obviously this will work for simple cases. When retrieving URLs, though, it may have unexpected results. So why do things this way? One word: Performance. Large files work much better when we use this method. The underlying system can optimize reading of the file.

QueryPath 2.0 may change this behavior. The next version of QueryPath may use the method outlined above for all files. For QueryPath 1, though, this is how files are interpreted.

When parsing moderately sized old HTML files, you will do best to pass a context into qp(). This will give you the greatest chances of successfully parsing the document.

I've written a handful of tools that make use of the OpenAmplify web service. In all cases, I've used QueryPath to retrieve the XML from the remote server and then work with it locally. In this short article, I will explain how QueryPath can be used to retrieve content from OpenAmplify's web service. I will also provide brief examples of how QueryPath can work with the results.

To see these techniques in action, you can visit TweetyPants. To see some functional code, take a look at the Amplify and QP Services modules for Drupal.

Retrieving Data from OpenAmplify

Let's begin with some example code. The following code performs a simple POST-based query against the OpenAmplify web service.

<?php
require 'QueryPath/QueryPath.php';
 
$url = 'http://portaltnx.openamplify.com/AmplifyWeb/AmplifyThis?';
$key = 'OPENAMPLIFY_API_KEY_GOES_HERE';
$text = 'This is the text we are going to amplify.';
 
$params = array(
  'apiKey' => $key,
);
 
$url .= http_build_query($params);
 
$options = array(
  'http' => array(
    'method' => 'POST',
    'user_agent' => 'QueryPath/2.0',
    'header'  => 'Content-type: application/x-www-form-url-encoded',
    'content' => http_build_query(array('inputText' => $text)),
  ),
);
 
$context = stream_context_create($options);
try {
  $qp = qp($url, NULL, array('context' => $context));
}
catch (Exception $e) {
  print "Could not retrieve data from OpenAmplify." . $e->getMessage();
  exit;
}
?>

To begin, we define $url, $key, and $text. $url will just contain the URL of the OpenAmplify server. $key is the API key that OpenAmplify issues to you. $text is the text that we are going to amplify. Typically, this will come from some other source, like a document or user input.

The $params array is going to contain GET parameters. Posting documents to OpenAmplify will still require us to pass some information as part of the GET string. Namely, the API key must be passed in a URL like this:

http://portaltnx.openamplify.com/AmplifyWeb/AmplifyThis?apiKey=MY_KEY

The $options array is a little more complex. When working with QueryPath, we rely upon PHP stream contexts to configure a particular HTTP request. The $options array will be used to define the context.

For our purposes, we only need to configure the HTTP stream wrapper. Inside that array, we set the HTTP method to POST, define a user agent (which is optional), and add an HTTP header telling the remote server what type of encoding we are using. Finally, we add content, which holds the encoded name/value pairs that we need to pass to the server.

For our query, we only need to pass the inputText parameter, whose value will be the text we want to analyze.

Once we have the $options list created, we use this to build the stream context for PHP. And from there, we simply need to retrieve the data from OpenAmplify:

<?php  
$context = stream_context_create($options);
try {
  $qp = qp($url, NULL, array('context' => $context));
}
catch (Exception $e) {
  print "Could not retrieve data from OpenAmplify." . $e->getMessage();
  exit;
}
?>

There are a few things to notice here. First, the second argument to qp() is a null simply because we do not need to search the return document immediately. The third parameter holds the options for QueryPath. In our case, we need to set the stream context.

Finally, the try/catch block wraps this block so that we can detect an error right away.

This brief description should get you started when connecting to OpenAmplify. But what do you do with the resulting QueryPath object?

Working with the Results

From the point above, you can access the contents of the OpenAmplify XML document via the $qp variable. For example, this code prints out the top 20 proper nouns that OpenAmplify has identified:

<?php
$qp->find('ProperNouns>TopicResult>Topic>Name')->slice(0, 20);
 
// Set up the output:
$out = qp(QueryPath::HTML_STUB, 'body')->append('<ul/>')->find('ul');
 
// Add a list item for each noun:
foreach ($qp as $name) {
  $out->append('<li>' . $name->text() . '</li>');
}
// Write the contents to STDOUT
$out->writeHTML();
?>

The first line of the code snippet above searches for the name of every proper noun and then slices (keeps) the first 20. Note that we use the direct child combinator (>) because it is faster than using the any descendant combinator (represented by an empty space).

The $out QueryPath object will be the output document. The foreach loop simply goes through each proper noun and adds it to an unordered list in the output HTML document.

Finally, the results are printed out as an HTML document.

This should give you some ideas on how to work with OpenAmplify content from within QueryPath. To see some of the other techniques, take a look at the QP Services Drupal module, which makes frequent use of OpenAmplify data.

One of the most popular word processing document formats is the ODT (Open Document Text) format, supported natively by OpenOffice.org, and supported as an export format for other major word processors, including Microsoft Office.

An ODT document is actually a ZIP archive composed of several files, including metadata, the document text, and embedded items. Most of these files are XML documents. And that means we can easily access their contents using QueryPath.

In this short article, we'll see how to access the contents of an ODT file.

The code

The text content of an ODT file is stored in the content file inside of the ODT ZIP archive. To skip through it, you can do something like this:

$ unzip openoffice.odt
$ cat content.xml

The command above will display the text contents (in XML format) of the document named openoffice.odt. This is the file we are going to parse.

For our simple example, what we are going to do is print out a plain-text outline of the document. We will build the outline by reading the section headers from the ODT file.

Here's the code:

<?php
require_once 'QueryPath/QueryPath.php';
 
$file = 'zip://openoffice.odt#content.xml';
$doc = qp($file);
foreach ($doc->find('text|h') as $header) {
  $style = $header->attr('text:style-name');
  $attr_parts = explode('_', $style);
  $level = array_pop($attr_parts);
  $out = str_repeat('  ', $level) . '- ' . $header->text();
  print $out . PHP_EOL;
}
?>

After requiring the QueryPath library, we get to work parsing the file. The file is a ZIP archive. Rather than unzip it ourselves, though, we want to use PHP's ZIP stream handler to uncompress it (as needed) internally. To cause PHP to invoke the ZIP stream handler, we use a special URL to refer to the file:

$file = 'zip://openoffice.odt#content.xml';

The above tells PHP to unzip the openoffice.odt file and access the content.xml file in that archive.

We can pass that URL straight into QueryPath, which will then unzip and access the desired data.

The foreach loop contains the brunt of our application code. It iterates over all of the headers in the document, and then formats some output based on the header.

Notice the CSS selector passed into find(). It is a little out of the ordinary: text|h. The pipe operator is rarely used in CSS 3 when you are working with HTML documents. It provides XML namespace support for CSS. So the above will seek for elements that look like this:

<text:h>Header text</text:h>

Effectively, the pipe (|) in the selector replaces the colon in the tagname. In ODT, headers are stored as h tags inside of the namespace urn:oasis:names:tc:opendocument:xmlns:text:1.0, which in turn is usually aliased to text. Thus, text|h searches for all headers in the document.

So the iterator is now looping through all of the headers in the file. With each header, five things are done:

  $style = $header->attr('text:style-name');
  $attr_parts = explode('_', $style);
  $level = array_pop($attr_parts);
  $out = str_repeat('  ', $level) . '- ' . $header->text();
  print $out . PHP_EOL;

First, we extract the style of the header. This will enable us to determine what level of heading this is (e.g. level 1, 2, 3, and so on).

Style is stored in the namespaced attribute text:style-name. Since we are retrieving the attribute, we use the entire XML name: text:style-name. (We do not replace the ':' with a '|' because we are not executing a CSS 3 Selector.)

A style name will look like this: text:style-name="Heading_20_2". The last digit, 2 indicates the level of the heading, and that is the number in which we are interested.

We retrieve the heading number by exploding the attribute name and then retrieving the last item from the attribute array:

  $attr_parts = explode('_', $style);
  $level = array_pop($attr_parts);

Now $level contains a digit indicating the header number. From there, we simply want to display the header formatted to indicate its depth in the outline:

  $out = str_repeat('  ', $level) . '- ' . $header->text();
  print $out . PHP_EOL;

The str_repeat function pads the beginning of the string with one space for every heading level. A first level heading will be indented one space. A third level heading will be indented three spaces. After the spaces, we add a dash (for formatting) and then the title of the section.

Finally, each line is printed to standard output.

So what does the output of this command look like? Let's take a quick look at our sample document as rendered by OpenOffice.org:

OpenOffice doc

Notice the multiple levels of headings. Let's extract those now with the tool we just built:

  - Section One
    - Subsection A
    - Subsection B
  - Section 2
    - Subsection 2A
      - Item AA
      - Item BB
    - Subsection 2B
  - Conclusion

Our simple tool parsed OpenOffice.org's XML and displayed an outline based on the headings from the document.

The code presented here is based on documents output from OpenOffice.org 3.x. Because of the flexibility of XML, it is possible for other ODT documents to be generated which will not conform to the same namespace convention we have used. What does this mean? It means you may have to tweak this little example to make it work on certain files (YMMV). But the principles will remain the same.

Word DocX documents also contain an XML payload. In the future, perhaps we will examine parsing and reading such files.

DrupalCampWILast weekend, I joined a couple hundred other Drupal users at Drupal Camp Wisconsin at the University of Wisconsin, Madison. This well-organized two-day event was fantastic. I met many new people (and can now connect a face with an IRC handle for many more). And the crack team of conference organizers have already put together videos of many conference sessions.

For me, the conference highlights included a handful of sessions on Drupal in education, a pair of sessions on GIS and mapping, and a BOF that I attended on web services, portlets, and the future of distributed web applications. A perennial strong point for Drupal Camps is the coverage of Drupal basics. DrupalCampWI had around half a dozen sessions for beginners. If you are just learning Drupal, a camp like this can really help you find your footing.

The camp's commons area was fantastic, providing ample space for both small and large BOFs as well as impromptu brainstorming sessions. Many conferees stayed at the same hotel, making after-hours ad hoc get togethers easy. And Wisconsin food? I ate my first (and probably last) "bacon bratwurst pretzel burger with cheese."

I gave two sessions. The first was on JavaScript and jQuery in Drupal. The second was on Web Services, mashups and QueryPath in Drupal (a preview version of what I hope to show in Paris this September). Most of the conference sessions are now available in video form.

Update: Added link to QueryPath video

QueryPath 2.0 Alpha 1 has been released. You can grab a copy for testing from the download page.

This new version adds some new methods, adds a few of the straggling CSS 3 Selectors, provides a new object for global configuration, and employs new (faster) internal data structures. You should notice speed improvements with this version -- even if you are using an opcode cache.

QueryPath 2.0 Alpha 1 is a testing release. Further API changes will be made before the final release (though few will have impact on your development). It is not recommended for large production sites.

Existing QueryPath adapters, such as the Drupal QueryPath module should be able to use this new version seemlessly.

I am thrilled to hear that I am one of the finalists for the Author of the Year Award. Thanks to all who voted!

In the next stage of evaluation a panel of judges will select one of the six finalists to be named "Author of the Year".

http://packtauthor2009.posterous.com/announcing-packt-author-award-final...