Category Archives: Programming

Adding OpenWrt support for Netgear WNR1000v2

I’ve this router for the sole reason that it is a recommended piece of equipment for doing WiFu, but as as I previously mentioned, it fails to do its job.

In the mean time, I discovered that Netgear released some GPL firmwares which sit at the base of their firmwares. Basically you get a heavily modified OpenWrt Kamikaze (7.09) without a web interface and you need an ancient buildroot. You do have a command line utility for configuring it, but it is painful to do it so. I was unable to configure the WEP support in 1.0.0.5 GPL by using the same config as the 1.0.1.1 is using. Also, moving around the GPL firmware is difficult since there’s no vi support in busybox, but there’s a tftp client and cat.

The device itself is EOL, therefore the idea of using OpenWrt came up. I tried the generic AP81 build, as described here, but it corrupts the rootfs. The device boots in failsafe mode where you may flash a good firmware by using a TFTP client, therefore on the recovery side is good.

On the WikiDevi page for WNR1000v2 it is stated that it is using the same hardware as WNR612v2, but the WNR612v2 router has only two LAN ports, while WNR1000v2 has four. WNR612v2 itself uses hardware close to WNR2000v3 and both of these are supported by OpenWrt.

Flashing a WNR612v2 firmware, both factory and OpenWrt is impossible with the default images. The Netgear flashing support checks for the presence of a “magic number” in the firmware file. The procedure for finding that “magic number” is totally undocumented, therefore I’ve made a lot of wrong turns. I used the WNR612v2 “magic number” which is declared in OpenWrt, 0x32303631, and started to examine the firmware file. A quick grep confirmed that the byte sequence is present. Dumping the header of the file also confirmed it.

I repeated the procedure with the GPL firmware which I built for WNR1000v2 and with a factory firmware. It turns out that the magic number is: 0x31303031.

The rest of the work for adding WNR1000v2 support to OpenWrt was fairly straightforward after that as I used WNR2000v3 and WNR612v2 as template.

The rfkill and the WPS buttons don’t work at all. I don’t use WPS anyway, therefore for me is a non-issue. The lack of rfkill support may be a mild annoyance, but since this is my first router that actually has a button for toggling the wireless, doing it the usual way isn’t a big deal for me. I think the buttons may be controlled by GPIO, but I’m not sure and probably I’ll check this when my schedule allows it.

The patch and images are available into this Gist. The patch was made against the Barrier Breaker branch, r43617. The procedure for building your image if you don’t want to use my own image is described into the README.

I guess the next step is to add this support into the OpenWrt trunk, but I need to see if somebody is willing to merge the changes for supporting this device. I’ll need to test the patch and build more since the first merge with quilt wasn’t without issues. The purpose is to obtain a clean build in a single run.

Performance breakdown for libxml-to-js

Background

libxml-to-js was born to solve a specific problem: to support my early efforts with aws2js. At the time, the options were fairly limited. xml2js was a carry-over from aws-lib which aws2js initially forked. I was never happy with xml2js for a couple of reasons: performance and error reporting. Therefore I looked for a solution to have a drop-in replacement. Borrowed some code from Brian White, made it fit to the xml2js (v1) formal specifications, then pushed it to GitHub. At some point the project had five watchers and five contributors. I guess it hit a sweet spot. That’s why it’s got support for XPath and CDATA, most of it from external contributions. And only then I started using it for other XML related stuff.

The name was chosen to make a distinction from libxmljs which sits at the core of this library which actually binds to Gnome’s libxml2.

Due to the fact that aws2js gained some popularity and I’m doing a complete rewrite with 0.9, the output of libxml-to-js most probably won’t change beyond the “specs” of xml2js v1.

Performance

The actual reason for why I’m writing this article is the fact that people keep asking about the reason for choosing libxml-to-js over xml2js, therefore next time when this question arrives, I am going to simply link this article.

Even now, two and a half years later, with some crappy benchmark that I pushed together, it is somewhere around 25-30% faster than xml2js under usual circumstances. In only specific cases that don’t apply to the XML returned by AWS, xml2js closes in. The part where it really shines is still the error reporting where besides the fact that’s accurate, it is also screaming fast compared to xml2js. In my tests it came out to be around 27X faster.

The code:

var Benchmark = require('benchmark');
 
var suite = new Benchmark.Suite;
 
var parser1 = require('libxml-to-js');
var parser2 = new require('xml2js').Parser({
    mergeAttrs: true,
    explicitRoot: false,
    explicitArray: false
}).parseString;
 
require('fs').readFile(process.argv[2], function(err, res) {
    if (err) {
        console.error(err);
        return;
    }
    var xml = res.toString();
    // add tests
    suite.add('XML#libxml-to-js', function() {
        parser1(xml, function(err, res) {});
    })
        .add('XML#xml2js', function() {
            parser2(xml, function(err, res) {});
        })
    // add listeners
    .on('cycle', function(event) {
        console.log(String(event.target));
    })
        .on('complete', function() {
            console.log('Fastest is ' + this.filter('fastest').pluck('name'));
        })
    // run async
    .run({
        'async': true
    });
 
});

The results, based onto the XML files from the libxml-to-js unit tests and the package.json for the error speed test:

# package.json
XML#libxml-to-js x 18,533 ops/sec ±3.46% (75 runs sampled)
XML#xml2js x 673 ops/sec ±1.35% (68 runs sampled)
Fastest is XML#libxml-to-js
 
# ec2-describeimages.xml
XML#libxml-to-js x 1,122 ops/sec ±4.59% (74 runs sampled)
XML#xml2js x 818 ops/sec ±7.02% (83 runs sampled)
Fastest is XML#libxml-to-js
 
# ec2-describevolumes-large.xml
XML#libxml-to-js x 65.41 ops/sec ±3.13% (65 runs sampled)
XML#xml2js x 50.88 ops/sec ±2.14% (65 runs sampled)
Fastest is XML#libxml-to-js
 
# element-cdata.xml
XML#libxml-to-js x 14,689 ops/sec ±5.41% (72 runs sampled)
XML#xml2js x 11,551 ops/sec ±2.36% (88 runs sampled)
Fastest is XML#libxml-to-js
 
# namespace.xml
XML#libxml-to-js x 9,702 ops/sec ±5.75% (72 runs sampled)
XML#xml2js x 5,802 ops/sec ±2.41% (81 runs sampled)
Fastest is XML#libxml-to-js
 
# root-cdata.xml
XML#libxml-to-js x 22,983 ops/sec ±7.11% (69 runs sampled)
XML#xml2js x 14,849 ops/sec ±6.01% (87 runs sampled)
Fastest is XML#libxml-to-js
 
# text.xml
XML#libxml-to-js x 2,669 ops/sec ±3.68% (78 runs sampled)
XML#xml2js x 2,617 ops/sec ±2.41% (88 runs sampled)
Fastest is XML#libxml-to-js
 
# wordpress-rss2.xml
XML#libxml-to-js x 2,056 ops/sec ±4.08% (75 runs sampled)
XML#xml2js x 1,226 ops/sec ±2.79% (84 runs sampled)
Fastest is XML#libxml-to-js

The tests ran under node.js v0.10.22 / OS X 10.9 / Intel Core i5-4250U CPU @ 1.30GHz with the latest module versions for both libxml-to-js and xml2js.

git is distributed, stupid

Inlining the PEM encoded files in node.js

Leave a Reply

Multi line strings in JavaScript are a bitch. At least till ES6. The canonical example for a node.js HTTPS server is:

// curl -k https://localhost:8000/
var https = require('https');
var fs = require('fs');
 
var options = {
  key: fs.readFileSync('test/fixtures/keys/agent2-key.pem'),
  cert: fs.readFileSync('test/fixtures/keys/agent2-cert.pem')
};
 
https.createServer(options, function (req, res) {
  res.writeHead(200);
  res.end("hello world\n");
}).listen(8000);

All fine and dandy as the sync operation doesn’t penalize the event loop. It is associated with the server startup cost. However, jslint yells about using sync operations. As the code is part of the boilerplate for testing http-get, refactoring didn’t make enough sense. Making jslint to STFU is usually the last option. The content of the files never changes, therefore it doesn’t make any sense to read them from the disk either. Inlining is the obvious option.

Couldn’t find any online tool to play with. Therefore I fired a PHP REPL, then used my PCRE-fu to solve this one. The solution doesn’t look pretty, but it gets the job done:

php > var_dump(preg_replace('/\n/', '\n\\' . "\n", file_get_contents('server.key')));
string(932) "-----BEGIN RSA PRIVATE KEY-----\n\
MIICXAIBAAKBgQCvZg+myk7tW/BLin070Sy23xysNS/e9e5W+fYLmjYe1WW9BEWQ\n\
iDp2V7dpkGfNIuYFTLjwOdNQwEaiqbu5C1/4zk21BreIZY6SiyX8aB3kyDKlAA9w\n\
PvUYgoAD/HlEg9J3A2GHiL/z//xAwNmAs0vVr7k841SesMOlbZSe69DazwIDAQAB\n\
AoGAG+HLhyYN2emNj1Sah9G+m+tnsXBbBcRueOEPXdTL2abun1d4f3tIX9udymgs\n\
OA3eJuWFWJq4ntOR5vW4Y7gNL0p2k3oxdB+DWfwQAaUoV5tb9UQy6n7Q/+sJeTuM\n\
J8EGqkr4kEq+DAt2KzWry9V6MABpkedAOBW/9Yco3ilWLnECQQDlgbC5CM2hv8eG\n\
P0xJXb1tgEg//7hlIo9kx0sdkko1E4/1QEHe6VWMhfyDXsfb+b71aw0wL7bbiEEl\n\
RO994t/NAkEAw6Vjxk/4BpwWRo9c/HJ8Fr0os3nB7qwvFIvYckGSCl+sxv69pSlD\n\
P6g7M4b4swBfTR06vMYSGVjMcaIR9icxCwJAI6c7EfOpJjiJwXQx4K/cTpeAIdkT\n\
BzsQNaK0K5rfRlGMqpfZ48wxywvBh5MAz06D+NIxkUvIR2BqZmTII7FL/QJBAJ+w\n\
OwP++b7LYBMvqQIUn9wfgT0cwIIC4Fqw2nZHtt/ov6mc+0X3rAAlXEzuecgBIchb\n\
dznloZg2toh5dJep3YkCQAIY4EYUA1QRD8KWRJ2tz0LKb2BUriArTf1fglWBjv2z\n\
wdkSgf5QYY1Wz8M14rqgajU5fySN7nRDFz/wFRskcgY=\n\
-----END RSA PRIVATE KEY-----\n\
"
php > var_dump(preg_replace('/\n/', '\n\\' . "\n", file_get_contents('server.cert')));
string(892) "-----BEGIN CERTIFICATE-----\n\
MIICRTCCAa4CCQDTefadG9Mw0TANBgkqhkiG9w0BAQUFADBmMQswCQYDVQQGEwJS\n\
TzEOMAwGA1UECBMFU2liaXUxDjAMBgNVBAcTBVNpYml1MSEwHwYDVQQKExhJbnRl\n\
cm5ldCBXaWRnaXRzIFB0eSBMdGQxFDASBgNVBAMTC1N0ZWZhbiBSdXN1MCAXDTEx\n\
MDgwMTE0MjU0N1oYDzIxMTEwNzA4MTQyNTQ3WjBmMQswCQYDVQQGEwJSTzEOMAwG\n\
A1UECBMFU2liaXUxDjAMBgNVBAcTBVNpYml1MSEwHwYDVQQKExhJbnRlcm5ldCBX\n\
aWRnaXRzIFB0eSBMdGQxFDASBgNVBAMTC1N0ZWZhbiBSdXN1MIGfMA0GCSqGSIb3\n\
DQEBAQUAA4GNADCBiQKBgQCvZg+myk7tW/BLin070Sy23xysNS/e9e5W+fYLmjYe\n\
1WW9BEWQiDp2V7dpkGfNIuYFTLjwOdNQwEaiqbu5C1/4zk21BreIZY6SiyX8aB3k\n\
yDKlAA9wPvUYgoAD/HlEg9J3A2GHiL/z//xAwNmAs0vVr7k841SesMOlbZSe69Da\n\
zwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBACgdP59N5IvN3yCD7atszTBoeOoK5rEz\n\
5+X8hhcO+H1sEY2bTZK9SP8ctyuHD0Ft8X0vRO7tdt8Tmo6UFD6ysa/q3l0VVMVY\n\
abnKQzWbLt+MHkfPrEJmQfSe2XntEKgUJWrhRCwPomFkXb4LciLjjgYWQSI2G0ez\n\
BfxB907vgNqP\n\
-----END CERTIFICATE-----\n\
"
php >

This gave me usable multi line strings that don’t break the PEM encoding.

Update: shell one liner with Perl

cat certificate.pem | perl -p -e 's/\n/\\n\\\n/'

Async frameworks “Hello World” showdown

2 Replies

This is not intended to be a proper comparison between these frameworks. However, since the “Hello World” test is the lowest common denominator, it is a pretty clear indicator that an application can’t exceed in performance these numbers. Also, what Guillermo did not understand from my comment is the fact that 1000 requests at the concurrency of 10 is way to few for get a proper picture of a “Hello World” showdown.

Tested frameworks:

node.js – v0.6.17
vert.x – v1.0 final + OpenJDK 7 installed from the Ubuntu repository – using the JavaScript bindings
luanode – built from the master branch using the Ubuntu provided lua dependencies
luvit – built from the master branch
react – cloned the master branch

I also wanted to test node.native, but it kept crashing on me. You can see that it is a pretty old issue. I didn’t have the patience to make the v0.1.0 branch to work with the previously used code. But I’d like to give it a run for its money.

The system used for the testing is a modest Athlon II X2 240e (2.8GHz) with 4GB or DDR2 800MHz running the latest Kubuntu 12.04 LTS amd64. Since ab pretty much takes a CPU core for itself, the frameworks ran a single process that occupied a single CPU core. I tried running a node.js HTTP server wrapped with the cluster module. Or passing -instances 2 to the vertx framework. The results were pretty much the same, therefore using just a single CPU core is a fair comparison.

The ab command that I used to hammer the Hello World! output:

ab -r -k -n 1000000 -c 1000 http://127.0.0.1:{port_name}/

The command ran at least a couple of times before saving the results. Just to make sure that everything is properly warmed up.

The averages graph:

The test sources and full ab output is available on this gist. There’s interesting output in the results.txt file for the stats nerds.

PS: I have the impression (but did not test) that vert.x may be a little bit faster, but ab is the actual bottleneck.

Update: added React (node.php) to comparision. Too lazy to plot another graph. But at 1573.40 req/s, it is harly a match even for luanode. Used the PHP 5.3.10 from the Ubuntu repositories.

Update: added another React (node.php) to comparision, but with a custom build of PHP 5.4.3. This time, it managed to get 3727.49 req/s.

printf(" SaltwaterC ");

/dev/urandom