Using a String as a Stream (Reader) in Node.js

Oct 19 2012

In its minimalism, Node.js does not have libraries to perform some common tasks. One such task is taking a string (or Buffer) of data and interacting with it as if it were a Stream. Here is a simple StringReader implementation that illustrates a no-nonsense way of exposing a string as if it were a stream.

Here's how it works at a high level:

  • A StringReader can take either a String or a Buffer.
  • For Buffer objects, it can also handle different encodings. You can use setEncoding() to set the encoding. When reading the stream, the given encoding will be used. (In other words, you can give it a Buffer and read it back as a String)
  • The resume() method is the workhorse. A StringReader is paused by default. Only when resume() is called will it begin emitting events. It will do the following:
    • Emit the entire String or Buffer in the first (and only) data event.
    • Emit the end event, indicating that there is no more data.
    • Emit the close event, indicating that there's nothing more this reader will do.

With this description in mind, let's look at the code. <!--break-->

// Core libraries we will use:
var util = require('util');
var Stream = require('stream');
var Buffer = require('buffer').Buffer;

/*
 * A Readable stream for a string or Buffer.
 *
 * This works for both strings and Buffers.
 */
function StringReader(str) {
  this.data = str;
}

// Make StringReader a Stream.
util.inherits(StringReader, Stream);

module.exports = StringReader;

/*
 * This is more important than it may look. We are going to 
 * create a "stream" that is "paused" by default. This gives
 * us plenty of opportunity to pass the reader to whatever
 * needs it, and then `resume()` it.
 *
 * This will do the following things:
 * - Emit the (entire) string or buffer in one chunk.
 * - Emit the `end` event.
 * - Emit the `close` event.
 */
StringReader.prototype.resume = function () {
  // If the data is a buffer and we have an encoding (from setEncoding)
  // then we convert the data to a String first.
  if (this.encoding && Buffer.isBuffer(this.data)) {
    this.emit('data', this.data.toString(this.encoding));
  }
  // Otherwise we just emit the data as it is.
  else {
    this.emit('data', this.data);
  }
  // We emitted the entire string, so we can finish up by
  // emitting end/close.
  this.emit('end');
  this.emit('close');
}

/*
 * Set the encoding.
 * 
 * This is used for Buffers.
 */
StringReader.prototype.setEncoding = function (encoding) {
  this.encoding = encoding;
}

/*
 * This is here for API completeness, but it does nothing.
 */
StringReader.prototype.pause = function () {
}

/*
 * This is here for API completeness.
 */
StringReader.prototype.destroy = function () {
  delete this.data;
}

Why does it emit the entire string (or Buffer) at once? The short answer is that there isn't a reason to do otherwise (though if we wanted, we could). The string is already stored entirely in memory, and our assumption is that the most efficient algorithm is to move it out to whatever is consuming the stream in the most efficient way.

Here's an example of this in action:

var Buffer = require('buffer').Buffer;
var StringReader = require('pronto/streams').StringReader;

var input = "This is the song that never ends...";

var sr = new StringReader(input);

sr.on('data', function (data) {
  console.log("Received %s", data);
});

sr.on('end', function () {
  console.log('All done.');
});

// "Unpause" the reader.
sr.resume();

The current version of this code can be found in Pronto.js