Computing file hashes with node.js

Since node.js has the shiny crypto module which binds some stuff to the openssl library, people might be tempted to compute file hashes with node.js. At least the crypto manual page shows how to do a SHA1 for a given file (mimics sha1sum). Should people do this? The answer is: NO. Some may say because it blocks the event loop. I say: because it is as slow as molasses in January. At least compared to dedicated tools.

Let’s have a look:

var filename = process.argv[2];
var crypto = require('crypto');
var fs = require('fs');
 
var shasum = crypto.createHash('sha256');
 
var s = fs.ReadStream(filename);
s.on('data', function(d) {
  shasum.update(d);
});
 
s.on('end', function() {
  var d = shasum.digest('hex');
  console.log(d + '  ' + filename);
});

time node hash.js ubuntu-10.04.3-desktop-i386.iso
208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac ubuntu-10.04.3-desktop-i386.iso
node hash.js ubuntu-10.04.3-desktop-i386.iso 28.92s user 0.80s system 100% cpu 29.661 total

time sha256sum ubuntu-10.04.3-desktop-i386.iso
208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac ubuntu-10.04.3-desktop-i386.iso
sha256sum ubuntu-10.04.3-desktop-i386.iso 4.86s user 0.21s system 99% cpu 5.093 total

time openssl dgst -sha256 ubuntu-10.04.3-desktop-i386.iso
SHA256(ubuntu-10.04.3-desktop-i386.iso)= 208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac
openssl dgst -sha256 ubuntu-10.04.3-desktop-i386.iso 4.40s user 0.17s system 100% cpu 4.567 total

Edit: to sum up for those with little patience:

node hash.js – 29.661s
sha256sum – 5.093s
openssl dgst -sha256 – 4.567s

/Edit

That’s a ~6.5X speed boost just by invoking openssl alone instead of binding to its library. node.js does something terribly wrong somewhere since the file I/O is not to blame for the slowness:

var filename = process.argv[2];
var fs = require('fs');
 
var s = fs.ReadStream(filename);
s.on('data', function(d) {
 
});
 
s.on('end', function() {
 
});

time node read.js ubuntu-10.04.3-desktop-i386.iso
node read.js ubuntu-10.04.3-desktop-i386.iso 0.62s user 0.60s system 106% cpu 1.148 total

This little example that I hacked together shows that using child_process.exec is pretty fine:

var exec = require('child_process').exec;
exec('/usr/bin/env openssl dgst -sha256 ' + process.argv[2], function (err, stdout, stderr) {
	if (err) {
		process.stderr.write(err.message);
	} else {
		console.log(stdout.substr(-65, 64));
	}
});

time node hash2.js ubuntu-10.04.3-desktop-i386.iso
208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac
node hash2.js ubuntu-10.04.3-desktop-i386.iso 4.44s user 0.19s system 100% cpu 4.630 total

So you can have your cake and eat it too. The guys with the philosophy got this one right.

3 thoughts on “Computing file hashes with node.js”

Sidney San Martín 1st November 2011 at 17:15

I would suggest using process.execFile instead of process.exec — exec runs a shell and passes the string to it as a command line. It leaves you open to command injection when you concatenate parts of the command together. execFile executes a program directly and lets you pass in its arguments as an array.

With process.execFile, that line would look like this:

require(‘child_process’).execFile(‘/usr/bin/env’, [‘openssl’, ‘dgst’, ‘-sha256’, process.argv[2]], function(…){ … });

SaltwaterC Post author1st November 2011 at 17:26

The script is actually just an example, but thanks for the suggestion. Didn’t knew about execFile() since it’s undocumented. It is a nice alternative to spawn().

In my case, this script is not an issue as I don’t deal with user input. But for some it may be.

Sidney San Martín 1st November 2011 at 18:03

execFile has been around for a while, but was only documented recently (and the docs could use some work): http://nodejs.org/docs/latest/api/child_processes.html#child_process.execFile

I realize that it’s just an example — but many programmers, including myself, have a nasty habit of taking inspiration from examples :).

Sidney San Martín 1st November 2011 at 17:15

I would suggest using process.execFile instead of process.exec — exec runs a shell and passes the string to it as a command line. It leaves you open to command injection when you concatenate parts of the command together. execFile executes a program directly and lets you pass in its arguments as an array.

With process.execFile, that line would look like this:

require(‘child_process’).execFile(‘/usr/bin/env’, [‘openssl’, ‘dgst’, ‘-sha256’, process.argv[2]], function(…){ … });
SaltwaterC Post author1st November 2011 at 17:26

The script is actually just an example, but thanks for the suggestion. Didn’t knew about execFile() since it’s undocumented. It is a nice alternative to spawn().

In my case, this script is not an issue as I don’t deal with user input. But for some it may be.
Sidney San Martín 1st November 2011 at 18:03

execFile has been around for a while, but was only documented recently (and the docs could use some work): http://nodejs.org/docs/latest/api/child_processes.html#child_process.execFile

I realize that it’s just an example — but many programmers, including myself, have a nasty habit of taking inspiration from examples :).

This site uses Akismet to reduce spam. Learn how your comment data is processed.

printf(" SaltwaterC ");

/dev/urandom

3 thoughts on “Computing file hashes with node.js”

Leave a Reply