babuskov-blog

[babuskov-blog]

md5 in Node.js

If you come from PHP background, you're used to PHP's global top-level functions for everything. Some people say it's PHP's curse, others praise it. I'm doing some Node.js stuff lately and needed equivalent of PHP's md5sum() function. Turns out, it's really simple and included into base Node.js install. You need to use the "crypto" module and generate md5 hash using createHash. "createHash" might sound confusing as data is not really hashed by the function. You create hash and then add data to it. After all data is in, you read the digest:

var crypto = require('crypto'); crypto.createHash('md5').update(data).digest("hex");

That's all folks. Happy noding :)

[babuskov-blog]

Node.js and Express serving the same content for HTTP and HTTPS via SSL

In my previous post I explained how to set up SSL for Node.js/Express. Now, I want to serve the same content using the same logic for both http and https, and I don't want to duplicate my code.

The idea is to move everything involving Express app. into a function. Call the function for both http and https server. If you have global variables, make sure they are outside of this function:

var apps = express.createServer({key: pkey, cert: cert, ca: [dad1,dad2]}); var app = express.createServer(); apps.listen(443); app.listen(80); startServer(app, false); startServer(apps, true); function startServer(app, isSSL) {     app.configure(function () { // just some sample code         app.use(express.cookieParser());         app.use(express.bodyParser());         app.use(connect.static('public'));     });     app.get('/', function(req, res){         if (isSSL)             res.end('Hello HTTPS');         else             res.end('Hello HTTP');     }); }

This works, although I hope there is some nicer solution.

[babuskov-blog]

Setting up real SSL with Node.js and Express

I got my single-domain certificate from Godaddy. Suddenly, I got myself with .key file, .csr file, and two .crt files. Most examples you can google on the Internet use self-signed certificates (which is basically useless for Internet use) and .pem files. Wft is .pem, you might ask?

After wasting hours trying to get this to work, I finally did. I hope more posts like mine get written and reach google index, so that people trying to set up production systems don't have to waste time. Here's how I did everything, step by step:

1. create your private key and certificate-request file. I used the command suggested by Godaddy as it requires 2048 bit key. Suppose your domain is domain.com:

openssl req -new -newkey rsa:2048 -nodes -keyout domain.key -out domain.csr

Most of the questions that follow are straightforward. Godaddy suggest you use domain.com for "Common name" field.

2. log into Godaddy, go to SSL menu and select Manage option. You actually buy a credit for SSL cert. so you need to "use" it, and then request a certificate. After using the credit and pressing "Launch" button I was welcomed with a screen saying zero (0) in all categories (certificates, requests, credits, etc.). This was rather confusing. Googling around, I found the solution: go to credits or certificates even though it says zero. After the page loads, an option appears to "update" the list. Click this and your credit shows up finally. Now, you can "request" the real certificate. Paste the content of domain.csr file you created in step 1. and wait for GD to create the cert.

3. after the cert is created, download it (there's a download option on the certificate screen). You'll get a .zip file containing two .crt files: domain.com.crt and gd_bundle.crt. First file is your SSL cert. The second file contains CA certs. of Godaddy that were used to digitally sign you cert. gd_bundle.crt might contain multiple certs. of which most browsers only need the first one, but it's better to install both. I've read some reports that some clients (ex. Android) require both to be installed properly.

4. Time to add all this to our Node.js/Express setup. It's a little bit different if you don't use Express (you need to call http.setSecure() with credentials):

var express = require('express'); var privateKey = fs.readFileSync('domain.key').toString(); var certificate = fs.readFileSync('domain.com.crt').toString(); var dad = fs.readFileSync('gd_bundle.crt').toString(); var app = express.createServer({key: privateKey, cert: certificate, ca: dad}); app.listen(443); app.get('/', function(req, res){     res.end('Hello SSL'); });

Supplying "ca" field to createServer is crucial, and missing from most examples on the net since they use self-signed certs.

Now, open http://domain.com and you should see the welcome message.

5. This works fine for my Firefox test. However, if you inspect the certs, you'll see that only one CA cert. is sent. To send both, we need to split gd_bundle.crt into two files and tell express to read both:

var dad1 = fs.readFileSync('gd_bundle.crt').toString(); var dad2 = fs.readFileSync('gd_bundle.crt').toString(); var app = express.createServer({key: privateKey, cert: certificate, ca: [dad1, dad2] });

That's all. I hope this saved you some time. In case it did, please follow me on twitter @mbabuskov, as I will post more Node.js stuff as I develop my applications.

[babuskov-blog]

Why PHP is better than JavaScript

I started developing a small project using node.js with express and socket.io. Node is a nice server and socket.io is great. However, I'm having issues with javascript. Currently, two things really get on my nerves:

1. the plus operator

Most of the hard-to-debug bugs in my javascript code come from the + operator. It decides to concatenate strings instead of add numbers. Considering that all stuff that goes over the wire (i.e. socket.io) is treated as strings, it's really painful and ugly to have parseInt(..., 10) everywhere. PHP solves this issue with simple dot operator. Simple, no-brainer and always does what you expect. You don't have to think where does the data come from.

2. foreach

I miss PHP's foreach so bad. Consider:

for (ix in really.long[expression].toGet.theStuff) {     if (really.long[expression].toGet.theStuff[ix].value < 10 && really.long[expression].toGet.theStuff[ix] > 5) {         ...do something

versus PHP's:

foreach ($really.long[expression].toGet.theStuff as $ix=>$value) {     if ($value < 10 && $value > 5) {         ....do something

Of course, one could assign the array element to some local variable, and so I have local variables all around wasting code lines and making code error prone (if you need to change the collection you are iterating, you have to change in two places).

[babuskov-blog]

Google multi-account get screwd up again

Looks like guys at Google really have trouble with multi-accounts and sessions. Everything was working fine for months now, but they messed it up again. What does the problem look like:

I have 2 google accounts, one @gmail.com and other @mydomain.com

I cannot login into @gmail one directly. I have to log into @mydomain and then use the "switch account" feature

I cannot bookmark both gmails. Although bookmarks are different, both open @mydomain account

Most other google services I use (ex. Analytics) are tied to my @gmail account. I cannot access those at all, unless I log out of everything, clear all the cookies and then log just into @gmail.

My user experience with google is getting worse every day:

multi-account login problems

google docs become painfully slow when spreadsheets grow 300+ rows (only about 10 columns though)

search is polluted with g+ spam, translation offerings, etc.

on one of my computers it shows my location in the wrong country and I cannot find a way to tell it differently.

If someone build free replacements, I would surely give those a shot.

[babuskov-blog]

GMail ignores Reply-to header [SOLVED]

I have a conact form on my website. People can leave their e-mail so that I contact them back. I set it up so that e-mail is sent From my e-mail address (general rule: never put user's e-mail in From field), and Reply-to set to user's e-mail address.

However, when I click "Reply" in GMail, the reply gets sent to back to me. Looks like some glitch in GMail's design, and they did not bother to fix it for a long time.

The solution is rather simple, just change the From field to some other address you own (different from GMail account address). For example if your e-mail is office@example.com, you can use support@example.com in the header. After this little change, Reply-to started working properly.

I hope this helps someone.

[babuskov-blog]

How to extract mp3 from YouTube video using Linux?

It's rather easy. I used 3 components:

DownloadHelper extension for Firefox

MPlayer

lame

When DownloadHelper is installed and you open a YouTube video, it gives you the options to download the .flv or .mp4 file to your computer.

After the file is downloaded, we can use MPlayer to play it and also to dump the audio. It's nice because it plays both .flv and .mp4, so you just need one program. To dump the audio to .WAV format use:

mplayer -ao pcm:waveheader FILENAME.flv

This will create file called audiodump.wav. Now, use mp3lame to encode it to mp3 format. You can also use oggenc to convert it to OGG if you perfer open formats.

lame audiodump.wav song.mp3

That's all. I put these commands in a simple shell script (video2mp3.sh):

mplayer -ao pcm:waveheader $1 lame audiodump.wav $1.mp3

Run it from command line like this:

. video2mp3.sh FILENAME.flv

[babuskov-blog]

Lean Startup vs Rework

Some time ago I finished reading the Lean Startup book by Eric Ries. Although I have been using some techniques from it before I have learned a lot. Yesterday, I found a mention of Rework on some website. Rework is a book by founders of 37 Signals, which I also read before. I started to compare the content of Lean Startup and Rework and I got some interesting conclusions.

Rework and 37 signals business model is really only a sub-set of Lean Startup philosophy. Basically, Rework stops somewhere during lean startup process and says "we're content with this". You build MVP, test it with customers, tweak a little bit and whoa, if you get good product market fit, what's next? Depends who you ask: 37signals guys would tell you: "Well done, now enjoy your success". They refuse to grow business, add features and go for a larger market. I'm not saying this is bad, sometimes you need to know what is your field of competence and stay there.

Which one would you follow? It depends on your personality. If you're going to become a serial entrepreneur, rework is not enough. If you have an urge to move forward, discover new horizons, you might need to use Lean Startup in each new project again. After all, one can hardly call 37signals a startup anymore. They behave like established business, not a startup. As DHH said on Twist, "if you're not doing your best idea now, you're doing it wrong". But, how do you know what is your best idea going to be if you do not explore? Maybe something looks like my best idea now, and I should be working on it. But, by the time that project becomes mature and stable, I might get a dozen of better ideas. And once I can turn the reigns of the current project to some good manager, I can go back to "startup" mode and explore new boundaries.

[babuskov-blog]

Twitter Timestamp out of bounds [solved]

Some time ago, automatic sending of status updates from one of my applications stopped working. I haven't changed anything on the server, so this was strange. Looking into error message from Twitter:

[request] => /1/statuses/update_with_media.json [error] => Timestamp out of bounds

well, that's strange. I recall time was moved to DST (daylight savings) Summer time in US recently, and apparently so did Twitter's servers. The rest of the world - bah, they don't seem to be interested, they can barely run the servers for US users apparently.

So, until DST change comes to the rest of the world, we need a hack. At first I thought that Twitter does not like timestamps to be in the future, so I thought about decreasing the timestamp of tweets. That did not work. So, I tried to increase the timestamp and everything is working now. The change is quite simple, just increase OAuth oauth_timestamp by a couple of hours and you're done. In PHP OAuth client it looks like this (I added 5 hours):

  private static function generate_timestamp() {     return time()+5*3600;   }

[babuskov-blog]

What's the conversion on e-mail newsletter with a call to action?

I have measured that in the past week. One of my websites has more than 500.000 user accounts. I picked the users who were not on the site in the last week, because they have already seen the news on the site, and got some 384.000+ distinct e-mail addresses. I needed to contact them regarding an important issue about the website. The e-mail was composed like this:

Hello,

you are reading this mail because you are a member of [mysite link].

Issue explained and link with [call to action]

Regards,

Your webmaster [mysite link]

Links where not plain text, but special URLs I used to track the clicks.

I sent the e-mail slowely over a 7 day period. I wanted to track weekends and working days as well.

And here are the stats:

384408 different e-mail addresses

13966 bounced back (3.6% bounce rate)

9345 clicks (yielding 2.5% conversion rate)

Hot spots in the e-mail message:

23% clicks on link in the first sentence

65% clicks on call-to-action

12% clicks on website link in signature

Here are the weekday stats:

Monday 13%

Tuesday 16%

Wednesday 18%

Thursday 18%

Friday 15%

Saturday 9%

Sunday 11%

[babuskov-blog]

Using JavaScript to split text string into word tokens, taking account of punctuation and whitespace and UTF-8 charset

I got an interesting problem today. I was supposed to check some HTML form before submitting to see if the text entered by the user in textarea has some specific words in it. Googling around I found a lot of stuff like "how to split text separated by commas" and such, but I simply wanted to extract words from a paragraph like this one.

My instinct was to use String.split() function, but it splits on a single character and I would have to write a recursive or iterative function to split on all non-word characters. Not being able to predict all the crap users can enter, this did not look like the right choice.

Luckily, I discovered String.match() which uses regex and is able to split text into an array of words, using something like this:

var arr = inputString.match(/\w+/g);

Cool, eh? Now, this all went fine for ASCII English text. But I need to work with UTF-8, or more specifically, Serbian language. Serbian Latin script used by my users has only 5 characters that are not from ASCII set, so I wrote a small replace function to replace those 5 with their closest matches. The final code looks like this:

var s = srb2lat(inputString.toUpperCase()); var a = s.match(/\w+/g); for (var i = 0; a && i < a.length; i++) {     if (a[i] == 'SPECIAL')         alert('Special word found!'); } function srb2lat(str) {     var len = str.length;     var res = '';     var rules = { 'Đ':'DJ', 'Ž':'Z', 'Ć':'C', 'Č':'C', 'Š':'S'};     for (var i = 0; i < len; i++)     {         var ch = str.substring(i, i+1);         if (rules[ch])             res += rules[ch];         else             res += ch;     }     return res; } ";

If you use some other language, just replace the rules array with different transliteration rules.

[babuskov-blog]

How to use IDB files of Quicken Home Inventory on 64 bit Windows?

Directly load your IDB file from Quicken Home Inventory on any 64 bit Windows system. It works on 32 bit as well, of course. Today, a new version of Attic Manager is released, version 3.00. This version is able to load data directly from IDB files, there is no need to install any additional software. You don't even have to have Quicken installed. This also means that you can run this option on 64 bit Windows 7 for example, or even on Linux.

Attic Manager can also load the inventory data from QHI and MDF files. QHI files are also loaded without any additional software.

For MDF files you need to have Microsoft SQL Server Express Edition installed. This is a freeware from Microsoft that comes with QHIM, so if you already have Quicken installed on the same computer, you don't need to install anything.

In any case, Attic Manager is now unique software on the market, being able to load all Quicken Home Inventory formats and allowing you to keep track of your items on any PC.

There are even hints of Mac version coming soon.

[babuskov-blog]

Why aren't you using InnoDB instead of MyISAM?

For this particular application, I simply cannot afford it. MyISAM is able to cache index and data separately, using it I can keep the whole index in RAM and website works great. I tried to convert to InnoDB, the result was 4x larger database and 20 times worse performance, mostly due to the fast that index and data gets the same priority for caching so it was killed by disk I/O. If this website was earning enough money to buy at least 4 x more RAM it might not be a bad idea. However, it this case I'd rather use Firebird - it has a similar memory footprint as InnoDB, but has much more features (*real* stored procedures and triggers that work without problems, ability to use table aliases in delete statements, database events functionality, better resistance to system crashes, etc.)

[babuskov-blog]

Why is MySQL still a toy database

I have been using MySQL for a very intensive read-write web application (averaging 102 queries per second) for more than two years. I had ups and downs with it, like crazy MyISAM behavior that readers can block writers AND OTHER READERS. Basically, a table level lock is issued for read. I have 100+ million records in a table, so it takes a while to find anything that is not indexed. In the meantime, users are pondering (102qps, remember) and load goes up so much because of web server processes queuing like crazy. Ok, I learned not to do that anymore. I now use binary logging, restore to a different server and query there. Maybe a switch to InnoDB would be a good idea, but in this case I'd rather use a serious MVCC database like Firebird. Why, you might ask... well, here's one of many reasons, the one that prompted my to write this:

In Firebird, I can happily do this:

delete from atable a1 where exists ( select 1 from atable a2 where a1.data = a2.data and a1.id <> a2.id );

It just does it, and fast, because index on primary key field ID is used. In MySQL, to quote the manual:

"Currently, you cannot delete from a table and select from the same table in a subquery."

Come on, this is one of the most basic database operation. So, what am I now to do? Waste my time dumping the list of IDs to delete to some temporary location, and then iterating that list to delete. :(

[babuskov-blog]

Scrolling back in screen

A few years ago I discovered screen, a nice Linux tool that enables you to detached from terminal with commands running and all in the background. You can even connect later from a different computer and continue where you left off. I initially used it for rtorrent, but now I also use it to administer remote computers, for example when I start to do something that might take more than a day, I can log back in tomorrow. Also loggin in from home/work to complete some task, etc. Another use is administering remote computers on dial-up (yes, there are some) or slow and unstable 3G connections. Even if connection breaks down, I can log in later and pick up where it stopped.

One of the annoying "problems" with screen is that shift+page up/down does not scroll the buffer. This is due to the fact that screen has its own buffers. To work with them you need to enter the "copy mode" using Ctrl+a followed by [. Since I use non-English keyboard that's Ctrl+a, AltGr+f. Hard to remember when you don't use it often.

I use Konsole, and I found a way to make it work by adding the following lines to .screenrc (in my home directory):

termcapinfo xterm|xterms|xs|rxvt ti@:te@

[babuskov-blog]

Why is Firebird better DBMS than Oracle?

Beside being free (both as beer and also open source), you don't need 24x7 DBA and there are generally less headaches. Here's a nice example explained by Norman Dumbar in a mailing-list post. Norman administers over 600 Oracle databases and about 40 Firebird ones:

Oracle uses log files for REDO and has ROLLBACK_SEGMENTS or UNDO Segments (depending on Oracle version) for UNDO. It never uses log files for UNDO - and UNDO is what provides Read Consistency/MVCC in an Oracle database. Changes are written to the LOG_BUFFER (n memory) and periodically - on commit, every 3 seconds max, or when the buffer is 33% full - flushed to the REDO logs. These REDO logs might be archived to disc when they fill up. That Depends on the database archive log mode though. These logs are used when a database is restored and rolled forward (using the RECOVER DATABASE command, for example). In order to roll back changes and to ensure read consistency, UNDO is used. These do live on disc - as tablespace files - but remain in memory in the buffer cache alongside data blocks etc. When a SELECT is started, the data returned are the data from the data blocks. Each row in a block has an indicator that tells when it was last updated. If a pending update is taking place (currently uncommitted) or if a commit has taken place since this SELECT started then the data read from that data block has changed - and is not consistent with the start time of this SELECT transaction. When this is detected, Oracle "rolls back" the changes to the start time of the SELECT taking place by looking for the UNDO block(s) associated with the transaction that made the changes. If that results in the correct (consistent) data, that's what you get. If it turns out that there were other transactions that also changed the data, they too will be detected and undone. In this way you only ever see data that was consistent at the start of your own transaction. As long as the DBA correctly sizes the UNDO tablespace and correctly sets the UNDO_RETENTION parameter to a decent enough value, data changes are able to be rolled back happily all the time. If the DBA failed miserably in his/her duties, the ORA-01555 Snapshot too old" errors are the result. And are most irritating. Long running SELECTS - batch reports for example - tend to show up this error mostly.

Of course, you would never see such problems with Firebird, because the old record versions are stored in database and not the log files. You don't have to care if system crashes - after reboot it simply works.

You might think that engineers who build Firebird are smarter than Oracle's but sometimes I think Oracle is deliberately made so complicated to require DBA and also offer them job security. And also makes sure nobody can complain it's too easy to use.

[babuskov-blog]

Using Quicken Home Inventory Manager on 64bit systems [FINALLY]

No, Quicken does not support 64bit Windows 7 yet. And there are no plans to do so. A few months back, GuacoSoft has released a new version of Attic Manager that is able to load data from Quicken directly. You can then export it into csv, excel, whatever OR simply use Attic Manager to manage the inventory.

Initial version of Attic Manager with this support (2.03) was only able to load data from .MDF files. However, a new version (2.50) is out now that supports .QHI files as well. It can load all data from .MDF. For files with .QHI extension, it loads all the data except image thumbnails. However, if you still keep your original images on the disk in same location where they were when you loaded them into QHIM, the Attic Manager will pick them up while importing and create thumbnails automatically. Not only that, but it will store a copy of each image into it's database, so that you never lose it in the future.

So far, this is the only way to extract data from Quicken, and it's really the only Home Inventory program on the market that enables you to transfer all your data before migrating to a new program.