matt's blog

18 Mar

Convert DOS to UNIX (and UNIX to DOS) in VIM

in programming, vim

Once upon a time I knew a vi mantra for removing carriage returns from a DOS-formatted text file, thus transforming it into a UNIX file.It was something like :%s/^V^M//g. It has become such an ingrained habit that I can still type it without even thinking about it. Muscle memory.

But VIM knows more about a file than that old clunky version of vi did, and there's an easier and more consistent way of converting between DOS and UNIX in VIM.

To convert from DOS to UNIX:

:e ++ff=dos
:setlocal ff=unix
:w

This basically forces the document into DOS mode, and then tells VIM to switch it to a UNIX file. (That closing :w just writes the change to disk.)

ff is VIM's abbreviation for fileformat. If you want to type that all out, you can. And ++? It tells VIM to effectively reopen the current file. So make sure you save your work before running these commands.

To convert from UNIX to DOS is even easier:

:e ++ff=dos
:w

That forces the file into DOS mode and then saves the change to disk.

Since both of these hook into a deeper level of VIM, they're not as prone to bugs and inconsistencies as relying upon a regular expression.

But wait... I need to do something more complicated...

The above is a great way to do simple conversions, but if your needs are more complex, you might want to take a look at the VIM wiki page in this topic.

06 Mar

How to use Ubuntu's Upstart to Control Node.js Forever

in linux, node.js, system administration, ubuntu

Recently I needed to start a server written in Node.js. This server was deployed on Ubuntu 12.04, and I wanted it to be started using the system's init system. Yet I also wanted the safety of Forever, a script manager for Node.js.

Ubuntu still supports the classic SysV init style, but it also now supports the far more sophisticated Upstart system. It is appealing for a few reasons: One is that it feels simpler and cleaner. Another is that it is more powerful and easily more configurable. I decided to use it for my scripts.

This article explains how to use Upstart, Forever, and Node.js together to run a Node.js server as a daemon process that is automatically started at system startup, and automatically stops at shutdown or reboot.

27 Feb

Installing GlusterFS on HP Cloud

in cloud, devops, gluster, hpcloud, openstack, system administration

Gluster is a distributed filesystem that works well in the cloud. This post explains how to configure GlusterFS on an Ubuntu 12.04 image running in HP's cloud.

Using this setup, I gain all the benefits of a distributed and replicated (redundant) filesystem for my in-cloud services, and can back these servers to persistent block storage if I want. It's a great way to gain stable networked filesystems in the cloud, and reduce or eliminate single points of failure.

04 Jan

PHP SPLObjectStorage v. Arrays: Redux

in benchmarks, php, programming

A few years ago, I tested SPLObjectStorage and Arrays to see which performed better. My conclusion at the time was that SPLObjectStorage is the better performer.

One thing I did not examine there was the implications of size. I ran tests with a fixed size of 10,000 elements. The test was also run on PHP 5.2.

A recent email from Frederik Krautwald caused me to re-evaluate these benchmarks. Using PHP 5.4.10 on Windows 7, Frederik generated this set of benchmarks:

18 Dec

It's always fun to see your project get news coverage

in hpcloud, programming, stackato

Forbes covered HP Cloud's announcement of the new Platform as a Service (PaaS) project.

This is my favorite quote from the article:

With this announcement HP’s Cloud grabs a potentially big technological advantage. Expect other cloud providers to do the same by quickly following HP’s lead of an integrated PaaS technology as key a differentiator. It’s also interesting to see that HP has selected ActiveState to form the core of its new PaaS offering. ActiveState’s Stackato is built atop VMware’s Cloudfoundry project, and is described as an application platform for creating your own private, secure, and flexible enterprise Platform-as-a-Service (PaaS) using any language on any stack on any cloud.

The same story was picked up by other outlets. My favorite "negative" take on it was posted on Diversity, Ltd . The author uses some loaded language (a poisoned chalice? Really?) to describe HP's partnership with ActiveState (Why? No idea). And he makes much of the fact that Stackato is a fork of Cloud Foundry. But I think his conclusion is right:

It’s not all negative for HP however. PaaS is, I believe, the future of the cloud and, more importantly, provides an opportunity for large traditional IT vendors like HP to bridge to offerings that will be compelling in the future world. Of course the fact that they can sell some cloud servers to run the new private PaaS offering doesn’t hurt either and I’m looking forward to seeing how seriously the company articulates the value of this product.

(As usual, I speak for myself, not my employer.)

17 Dec

Run Node.js apps on low ports without running as root

in node.js, programming, system administration

On Linux (and UNIX), to open a port with a number less than or equal to 1024, traditionally a program must run as the root user. This impacts web applications, which use ort 80 (HTTP) and port 443 (HTTPS) to do business. Many programs (Apache being a great example) use this by running a master process as the root user, and farming work off to helper processes that run with lower privileges. But Node.js is designed to work on a single-process model. So how do you run Node.js apps on low ports without running the script as root?

I looked at a number of solutions, and one suggested by my company's security team seems to be the best solution: For operating systems that support capabilities setting (like Ubuntu 12.04 and later), you can configure the operating system to allow non-privileged apps to listen on low ports. Here I show how to do it, and then briefly explain what is going on.

10 Dec

Using the Go Syntax for Janus VIM

in go, golang, janus, ubuntu, vim

To add the Go language (golang) plugins to Janus-flavored VIM, it's not enough to install the VIM plugins into your system's default location. So using, for example, Ubuntu's vim-syntax-go package will not work.

The best way to install all of the Go VIM plugins is to do the following:

  • Create a directory in your .janus directory for the go VIM plugin: mkdir ~/.janus/go
  • Download a fresh copy of the Go source: hg clone https://code.google.com/p/go/ (you may need to sudo apt-get install mercurial first)
  • Inside of the go/misc directory, find the vim directory
  • Copy the contents of the vim directory to the .janus/go directory: cp -a vim/* ~/.janus/go.
  • Restart VIM

This will give VIM access to all of the Go plugins. Syntax highlighting should immediately work on all *.go files.

26 Oct

PHP and curl_multi_exec

in curl, performance, php, programming

This post explain how to get data off of a curl_multi handle. Some time back I posted this snippet of code inside of a larger sample of code:

<?php
  $active = NULL;
  do {
    $ret = curl_multi_exec($multi, $active);
  } while ($ret == CURLM_CALL_MULTI_PERFORM);
 
  while ($active && $ret == CURLM_OK) {
    if (curl_multi_select($multi) != -1) {
      do {
         $mrc = curl_multi_exec($multi, $active);
      } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
  }
?>

I didn't really document or explain it. And so it seems that this code snippet has caused some confusion. Let me explain what it does.

19 Oct

Using a String as a Stream (Reader) in Node.js

in javascript, node.js, programming

In its minimalism, Node.js does not have libraries to perform some common tasks. One such task is taking a string (or Buffer) of data and interacting with it as if it were a Stream. Here is a simple StringReader implementation that illustrates a no-nonsense way of exposing a string as if it were a stream.

Here's how it works at a high level:

  • A StringReader can take either a String or a Buffer.
  • For Buffer objects, it can also handle different encodings. You can use setEncoding() to set the encoding. When reading the stream, the given encoding will be used. (In other words, you can give it a Buffer and read it back as a String)
  • The resume() method is the workhorse. A StringReader is paused by default. Only when resume() is called will it begin emitting events. It will do the following:
    • Emit the entire String or Buffer in the first (and only) data event.
    • Emit the end event, indicating that there is no more data.
    • Emit the close event, indicating that there's nothing more this reader will do.

With this description in mind, let's look at the code.

05 Oct

Pronto 0.3.9 Features

in javascript, node.js, pronto

The new pre-1.0 version of Pronto.js has been released. This is the first release of the 0.3 branch.

Pronto is a high-performance asynchronous framework for Node.js.

Here are the big new changes:

  • Pronto.js now has an HTTPS server.
  • The Pronto logger is considerably more sophisticated. While cxt.log("foo", "debug") still works, so does cxt.log("Hi, my name is %s.", "Matt", "debug"). Loggers can also serialize data into Node's inspect format as well as into JSON.
  • The web server no longer parses all POST and PUT data. Now it only parses content that it understands, and that is less than 1M.
  • There is a new BufferedReader readable stream. This should be used any time you need to buffer a stream for more than a tick.

The 0.3 branch is now semi-stable, and so has been released to npm.