TURN OFF MACHINIST CACHING!

Add this to your environments/test.rb file:

Machinist.configure do |config|
  config.cache_objects = false
end

Machinist tries to do some weird caching to make your tests run faster. But, it doesn’t quite work the way you’d expect. If you are running into strange problems where your objects are persisting through many tests, even though you are using DatabaseCleaner after each test, you might try this. If you run into problems where running one test at a time works, but running “rake spec” results in errors, this is also worth a shot. Don’t let Machinist caching drive you nuts!

Sidenote: In my experience, the best way to debug these errors that appear when running the entire test suite, but do not appear when running individual tests is to use rspec to run all but one test. Remove one at a time, and see if removing that single test helps eliminate errors.
Example:

# If this gives errors:
$ bundle exec rspec ./spec/models/user_spec.rb ./spec/models/account_spec.rb ./spec/models/favorite_spec.rb
# Try removing the first
 $ bundle exec rspec ./spec/models/account_spec.rb ./spec/models/favorite_spec.rb
# Try removing the 2nd
 $ bundle exec rspec ./spec/models/user_spec.rb ./spec/models/favorite_spec.rb
# Repeat...

Doing things like data mining with huge dataset pre-processing and manipulation, matrix computation, and other machine learning tasks can be computationally expensive and take forever! This has forced me to pay close attention to things such as code efficiency (not using Ruby for certain things – gasp!), database tuning, refactoring queries, optimal database schemas (and database choices), caching at all levels (application, db, etc), parallel processing (along with all its problems, concurrency issues, deadlocks, etc), system memory management, map-reduce, and on, and on..

One thing I especially love about working with massive data sets is that it really forces me to think “big”. No longer am I constrained to solving the problem with a single machine! In a couple clicks, I can create additional servers or double or triple the size of my server, quickly do computation or processing, and then bring the server back down to a more affordable size.

In this day and age, with cloud computing services like Amazon EC2 / Rackspace at your fingertips, you only pay for what you use. This is an important concept to consider, and definitely requires a different way of thinking about solving a “big” problem.

Of course the first step should still be to optimize your code, and use the right tools for the right problem. But, these are additional tools in your toolbox, that really pay off when you need them.

Here’s an example I recently encountered.

Prototype:
1. I started with a small test dataset matrix (3 MB, 3000 rows)
2. I wrote a Ruby/mysql script that would prepare this matrix from our DB. It took about 5 minutes, and that was fine. 5 minutes I can spare.
3. Next, I wanted to try this on my 2nd dataset matrix (20MB, 20000 rows). Hmm.. took over half an hour. That’s not THAT bad. But, imagine running this on a 200MB dataset, 20 GB dataset, or 2 TB dataset ?!! Gotta do better.

Step 1: optimize code / queries:
I spent a good amount of time tuning Mysql’s memory/caching/threads settings, making smarter queries, doing joins and de-normalizing tables to reduce queries. Knowing that Ruby is not the fastest language out there – I tried to be smarter about not doing extra iterations, using extra memory, etc.
- Result: I was able to cut down the time of the 2nd dataset from 30 minutes to about 7 minutes! (not bad)

Step 2: Parallelize code
I knew that the code was basically iterating over a huge array, so I wanted to try parallizing some of the actions it was performing. This was my first try at using Ruby’s Parallel gem, which allows you to split up any code into parallel threads or processes. Pretty cool stuff. I experimented with a bunch of different #’s of processes / threads to find out what the best settings were. Turns out the optimal setting was to use 4 concurrent processes. For what I was doing, my code was accessing the database (very costly I/O operations). So, the database was actually a bottleneck. Using more than 4 concurrent processes would result in too much DB traffic, and would hurt the performance.
-Result: I was able to cut down the time of the 2nd dataset from 7 minutes to about 1.5 minutes! (Getting there!)

Step 3: Try my sexy parallelized code on a bigger data-set (12GB).
- Of course I hit all kinds of problems – memory limit problems, database collisions, what’s the optimum number of parallel processes? etc, etc. Not so fast!
-Result: I was forced to run the code on a single process, that took something like 5-6 hours (boo )

Step 4: Figure out how to run the parallelized code on the large dataset
This required examining the logs and figuring out why I was hitting bottlenecks. It turns out that when I ran many concurrent processes, they were all fighting for DB access. Mysql didn’t like this and in the end, all processes became stalled. I found that by putting an increasing delay in each subsequent child process, I could avoid collisions. Since each process performed a huge DB query up front, and then spent the rest of its life using memory / CPU (but not competing for DB), they would all run together, nicely.
-Result:I was able to cut down the time of the huge dataset from 5-6 hours to about 1.5 hour! (yes!)

Step 5: Not really completely satisfied, I wanted to try enlarging my cloud instance and running the same exact command. I knew that on my 4GB server, the CPU and memory were all maxed out by my processing script. So, I tried increasing it to 16GB. The results were not so stellar.
-Result: I was able to cut down the time of the huge dataset from 1.5 hours on the 4GB machine to about 1 hour on the 16GB machine. This is with no additional db tuning (to make use of the additional system resources), nor adjusting the number of processes (a powerful server could have made use of more concurrent processes).

What I should have done next: Tune the db settings, tune the number of processes, and try again.

What I actually did instead: Called it a day, and wrote this article

My personal takeaways:
- Big data is exciting
- Use the right tools for the right job
- Optimize my code
- Tune my DB settings
- Use the cloud (Thanks Rackspace!)
- Think big!

var https = require('https');
var fs = require("fs");

var https_options = {
  key: fs.readFileSync("/path/to/server.key"),
  cert: fs.readFileSync("/path/to/mydomain.crt")
};
var https_server = https.createServer(https_options);

However, with the PositiveSSL certificate, Comodo will actually send you 3 files:
1) PositiveSSLCA2.crt
2) AddTrustExternalCARoot.crt
3) mydomain.crt

This is quite confusing for someone who doesn’t really understand (nor want to understand) all the details of how an SSL certificate works. Which one do I use for the cert: option??

Naturally, I started with the mydomain.crt file. This led to a cryptic web browser error message:
“this certificate was signed by an unknown authority”

So, a bit of googling found that when installing the PositiveSSL cert on Apache servers, you must use a chain file (mod_ssl option: SSLCertificateChainFile). If you check the Apache mod_ssl documentation you will see that this file is a concatenation of certificate files:

“Such a file is simply the concatenation of the various PEM-encoded CA Certificate files, usually in certificate chain order.”

So, what you have to do is the following:
1) Create a “bundle” file by concatenating the PositiveSSLCA2 and AddTrustExternalCARoot certificates

cat PositiveSSLCA2.crt AddTrustExternalCARoot.crt > mydomain.ca-bundle

2) Add this certificate as the “ca” option when creating your Node.js sever:

var https = require('https');
var fs = require("fs");

var https_options = {
  ca: fs.readFileSync("/path/to/mydomain.ca-bundle"),
  key: fs.readFileSync("/path/to/server.key"),
  cert: fs.readFileSync("/path/to/mydomain.crt")
};
var https_server = https.createServer(https_options);

This should properly set up the CA chain so that browsers can verify the SSL certificate.

We will create the certificate in the “/etc/ssl/self-signed” directory

# Become root first !
$ mkdir /etc/ssl/self-signed && cd /etc/ssl/self-signed
$ openssl genrsa -des3 -out server.key 1024
$ openssl rsa -in server.key -out server.key.insecure
$ mv server.key server.key.secure && mv server.key.insecure server.key
$ openssl req -new -key server.key -out server.csr
$ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Now, you can use the files server.key and server.crt in your web server setups. See my post on Setting up Node.js with to accept HTTP and HTTPS connections for a quick example of using this SSL certificate with Node.js!

We want to re-use as much code as possible because the two servers (HTTP and HTTPS) are identical besides the protocol.

In the example, we created a self-signed SSL certificate. If you need help on that, check out this quick guide on how to set up a self-signed SSL certificate on Ubuntu.

var restify = require('restify');
var fs = require('fs');

// Setup some https server options
var https_options = {
  key: fs.readFileSync('/etc/ssl/self-signed/server.key'),
  certificate: fs.readFileSync('/etc/ssl/self-signed/server.crt')
};

// Instantiate our two servers
var server = restify.createServer();
var https_server = restify.createServer(https_options);

// Put any routing, response, etc. logic here. This allows us to define these functions
// only once, and it will be re-used on both the HTTP and HTTPs servers
var setup_server = function(app) {
        function respond(req, res, next) {
           res.send('I see you ' + req.params.name);
        }

        // Routes
        app.get('/test/:name', respond);
}

// Now, setup both servers in one step
setup_server(server);
setup_server(https_server);

// Start our servers to listen on the appropriate ports
server.listen(80, function() {
  console.log('%s listening at %s', server.name, server.url);
});

https_server.listen(443, function() {
  console.log('%s listening at %s', https_server.name, https_server.url);
});

# Update the system and install some necessary packages

$ apt-get update
$ apt-get install g++ curl libssl-dev apache2-utils
$ apt-get install git-core

# Find the latest package to install at nodejs.org (example uses 0.6.11)

$ wget http://nodejs.org/dist/v0.6.11/node-v0.6.11.tar.gz
$ tar xzf node-v0.6.11.tar.gz
$ cd node-v0.6.11
$ ./configure
$ make
$ make install

# Install npm (Node.js package manager)

$ cd /usr/local/src
$ git clone http://github.com/isaacs/npm.git
$ cd npm/
$ make install

# Install express framework, -g flag installs the express executable globally

$ npm install -g express

# Create an app with the express framework

$ express /var/www/foo
$ cd /var/www/foo/

# Install dependencies

$ npm install -d

# Start the node app!

$ node app.js

Voila! I told you it was fast

When you try to export from MySQL in CSV format, the MySQL reference manual itself suggests using

FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'

Now, I thought I had a nicely formatted CSV file where fields that have double quotes would be properly escaped. Looking through this .csv file, I can see that sure enough, all the double-quotes are turned from ” into \”.

This is fine, except that when I tried to use Excel to open the CSV file or FasterCSV (a nice Ruby library for working with CSV files), I find that the double quotes were not escaped “properly”. They are expecting to escape double-quotes with TWO double-quotes (”"). According to CSVreader.com, this is a known discrepancy between the way different programs escape double-quotes in CSV files. Great.

Well, don’t worry – here’s a sed 1-liner that you can use on the command line to convert those \” (backslash double-quote) into “” (two double-quotes):

$cat file.csv | sed -E 's/\\"/""/g' > file2.csv

Hope that helps someone out there!

My calculations work in IRB, so I was really confused when I ran into this weird situation where (what I thought was) a simple arithmetic calculation led to strange results in my unit tests (I cannot stress the importance of good unit testing!).

My backend calculation was basically this (simplified):

# arbitrary amounts for these two variables
percentage = 12
total_in_cents = 400

discount = percentage.to_f / 100.0
total_in_float = total_in_cents.to_f * 100.0
new_price = (total_in_float * discount ).round / 100

Now, it’s pretty obvious that 12% of (400 cents) $4.00 should just be $0.48 (48 cents)
However, my barrage of unit tests kept producing strange results where a simple calculation was returning incorrect results. Doing some research, I discovered a series of articles worth reading including:

• How to properly round and calculate totals
• BigDecimal arithmetic in Ruby

Also, check out the Money gem – I’ve never used it personally, but people have said good things about it.

Heeding the advice I found online, I re-wrote all my money-related calculations using BigDecimals instead of Floats.

percentage = 12
total_in_cents = 400
discount = BigDecimal(percentage.to_s) / 100
total_in_float = BigDecimal(total_in_cents.to_s) * 100
new_price = (total_in_float * discount ).to_i / 100

After switching over from Floats to BigDecimals, my unit tests all passed!
Lesson learned and hope this heads-up helps you guys too.

Summary:
Use BigDecimals for money calculations and remember to write good UNIT TESTS!!

It’s easy:

$> git push -u <remote-name> <branch-name>

Or, usually:

$> git push -u origin branch-name

# Pass it into the GET request!
get '/programs/100', nil, {'HTTP_HOST' => 'sub.domain.com'}

The 3rd parameter to the get method is a hash of HTTP headers.
See the Rails API documentation for details.

Note:
Depending on the type of test you are working with (support / controller / request / integration etc.) you pass in the sub domain differently.

Here’s some good reference posts on Stack Overflow for setting subdomains in controller specs:
1. Rails RSpec Set Subdomain

# Set the @request.host in a before block
before(:each) do
  @request.host = "#{mock_subdomain}.example.com"
end

2. Subdomains in RSpec Controller Tests

 # I haven't tried this, and not sure you would need to mock out the current_subdomain method.
  @subdomain = 'sub.domain.com'
  controller.expects(:current_subdomain).returns(@subdomain)
  @request.host = "#{@subdomain}.test.host"