For one of our clients we’re deploying an application to multiple application instances on Amazon EC2. All the applications need access to three things: credentials, certificates and services.

To connect to the database server and Amazon S3 the application needs access to usernames and passwords. Server certificates are used in Apache and client certificates are used to communicate with external API’s. Finally we have a custom daemon that generates TLS client certificates called Certificate Depot.

We want to keep these three things as safe as possible without making frequent and automated deployment difficult. Our solution is to keep all the secret stuff in one location and only allow one point of entry.

We use a small EC2 instance as a little vault. Let’s call it the secure instance. The secure instance only runs an SSH daemon and Certificate Depot. A firewall blocks all connections, except on the SSH port. All communication with the secure instance goes over SSH. All data on the instance is stored on encrypted EBS volumes.

When a brand new application instance is booted it needs access to credentials and certificates so it can configure itself. This is done by temporarily uploading an SSH key to the application instance. The key allows a non-privileged user to connect to the secure instance. Once the app instance has everything it needs from the secure instances the key is deleted.

We wanted to keep the Certificate Depot code base small so we didn’t implement any authentication or authorization. We solve this at the network level. Certificate Depot only listens to localhost. Application instances access the Depot through an SSH tunnel to the secure instance.

By keeping the architecture simple we archieve what I like to call Security Through Clarity. The secure instance key is only needed when booting up new application or secure instances so only a few people need access to it. The single point of entry makes the secure instance less vulnerable to attacks. Credentials and certificates only need to be kept on the secure instance and a secure backup so they’re less likely to be leaked by sysadmins or developers.

When the secure instance key, credentials, or certificates are lost replacing them is straightforward because their owners and location are well defined.

If you were looking for a good reason to visit Amsterdam, now is your chance. On the 21st and 22nd of October the Ruby and Rails conference is taking place in Amsterdam.

Speakers are slowly lining up and Eloy and I were the first to enter the Rails Rumble.

For more information and registration see the website. I hope to meet you there!

I just pushed version 0.4 of the Nap gem. For those who don’t know: Nap is a really simple ReST API, basically a small wrapper around Ruby’s net/http.

New in this version is support for TLS client certificates, this means you can use certificates to authenticate with a server.

response = REST.get('https://example.com/pigeons/1',
  { 'Accept' => 'application/json' },
  { :tls_key_and_certificate_file => '/Users/Manfred/example.pem' }
)
response.body #=> {"pigeon":{"id":1}}

In an application we wrote back in 2004 I found MD5 hashed passwords. We decided this was too weak for modern standards so we wanted to switch to bcrypt. During the move we wanted the user to be affected as little as possible.

In order to compute the crypted password we need the cleartext version. We only have a hashed version so the user has to type her password. Luckily they do this every time they authenticate, so that is a nice opportunity to upgrade their password.

First I added a crypted_password column to the accounts table. We now have two columns for storing the password: the old hashed_password and the new crypted_password.

add_column :accounts, :crypted_password, :string

After that we updated the password accessor methods; assignment and verification.

class Account
  def password=(password)
    if new_record? or !password.blank?
      self.crypted_password = BCrypt::Password.create(password)
    end
  end
  
  def has_password?(input)
    BCrypt::Password.new(crypted_password) == input
  rescue BCrypt::Errors::InvalidHash
    false
  end
end

Now need to make sure we can authenticate with both the hashed as well as the crypted password stored for an account.

class Account
  def self.authenticate_with_crypt(params={})
    if account = find_by_username(params[:username]) and
       account.has_password?(params[:password])
      account
    end
  end
  
  def self.hash_password(password)
    ::Digest::MD5.hexdigest(password)
  end
  
  def self.authenticate_with_md5(params={})
    find_by_username_and_hashed_password(
      params[:username],
      hash_password(params[:password])
    )
  end  
end

Finally we need to make sure the password automatically updates. We try to authenticate using bcrypt. BCrypt raises an exception when the crypted_password is blank. This makes authentication fail and we fall back to trying the hashed password. When authentication with a hashed password succeeds we know the cleartext password and we can update it.

class Account
  def self.authenticate(params={})
    if account = authenticate_with_crypt(params)
      account
    elsif account = authenticate_with_md5(params)
      account.password = params[:password]
      account.hashed_password = nil
      account.save!
      account
    else
      account = Account.new(params.slice(:username, :password))
      account.errors.add_to_base("The credentials you entered are"
        "invalid. Please try again.")
      account
    end
  end
end

This solution will leave a group of users with a hashed password indefinitely. After a few months we could decide to throw away the hashed passwords. This means that infrequent users will have to reset their password if they do decide to log in again. It could cause some support requests, but I think we can handle them.

Note that this change doesn’t make the application safer. In case we leak information or when the database is somehow stolen it will make it harder to recover passwords. A lot of people use the same password for multiple accounts so this will give them time to reset their other accounts in case it is compromised. The change only took 15 minutes in this application so it’s totally worth the time.

As you probably already know the String class became encoding aware in Ruby 1.9. This makes it possible to manipulate strings on the character level instead of on byte level. However, it’s still a general purpose API which means writing a few lines of code to get stuff done.

It’s common to choose one internal representation for character data in an application and convert all incoming strings to this representation. For example, in modern applications strings are often encoded in UTF-8 or UTF-16. I took some time to figure out how to do this in Ruby 1.9.

The biggest problem with receiving data from external sources is trust. Sources can lie about their encoding or provide broken data. Sometimes it gets mangled accidentally and sometimes someone is attacking your application with a carefully crafted payload.

Problems can arise on a lot of levels. Just think about receiving an HTTP response from a webserver. Things can go wrong in the proxy, in the client library, in the string implementation of your language. Meta-data about the encoding is stored in HTTP headers, the HTML, and now in String. The same problems exist with data coming from databases, filesystems, and caches.

You can trust some of these sources more than others. For example, you can control the data going into a database so you can trust the data coming out. In contrast, anything coming from the internet should be considered potentially dangerous.

My solution for these problems is a new method on String called ensure_encoding. It makes sure the data in the string is at least compatible with your internal strings. Depending on the options you pass it will respond differently to broken data.

As an example, let’s take an HTTP POST to a web API. Assume we’ve explained in the API documentation we only accept UTF-8 character data and will be very strict about this. Our code might look something like this:

require "ensure/encoding"
begin
  params.each do |key, value|
    params[key].ensure_encoding!(Encoding::UTF_8,
      :external_encoding  => Encoding::UTF_8,
      :invalid_characters => :raise)
  end
rescue Encoding::InvalidByteSequenceError => e
  send_response_document :unprocessable_entity,
    "Sorry, your request contains invalid encoding" +
    "and can't be processed (#{e.message})"
end

You can find more examples on the GitHub project page and in the source. Normally I try to extract code from a running project, but we don’t run any production code on 1.9 yet. It would be great if you can help out with testing the code. I’ve released the code as a gem, so it should be really easy to install.

$ gem install ensure-encoding

Please leave any bugs, problems, or suggestions in the GitHub issue tracker.

It’s time for another ‘Morning coffee’ meeting. You’re invited to come chat with your fellow designers and developers over a nice cup of coffee.

Please leave a comment if you’re coming, any questions you might have are welcome too.

Ordering and comparing text is a lot trickier than a lot of people expect, computer scientists even came up with a complicated name for it: collation. There are two groups of problems associated with collation: cultural and technical. Today we’re not going to focus on technical problems, but rather how the cultural problems influence the technical solution.

An example of a cultural difference is letter ordering in a language, in Swedish the Ä is ordered after the z and in German it follows the letter a. You can also discuss whether the ä is an alternative form of the a or if it’s a completely different character, this is relevant when implementing search. When searching for ‘nächste’ you might also be interested in text containing ‘nachste’.

MySQL implements a number of collations solutions so you can use the one relevant for your application. For instance utf8_icelandic_ci when you want to order UTF-8 encoded text based on the cultural norm in Iceland. You can get a full list of supported collations from the mysql client.

mysql> SHOW COLLATION;

When you have an international site that might contain multiple languages you can use the default Unicode collation algorithm, which is called utf8_unicode_ci in MySQL. The UCA is pretty sensible so a lot of frameworks, including Rails, use it as a default. Unfortunately this collation also changes character equality. Lets look at an example how this might go wrong.

mysql> CREATE DATABASE books_example CHARACTER SET utf8
  COLLATE utf8_unicode_ci;
mysql> USE books_example;
mysql> CREATE TABLE books ( title VARCHAR(255) );
mysql> SHOW FIELDS FROM books;
+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| title | varchar(255) | YES  |     | NULL    |       | 
+-------+--------------+------+-----+---------+-------+
1 row in set (0.00 sec)
mysql> INSERT INTO books SET title = 'Pokemon';
mysql> INSERT INTO books SET title = 'pokemon';

Now we have a database with books, currently with two entries.

mysql> SELECT * FROM books;
+---------+
| title   |
+---------+
| Pokemon | 
| pokemon | 
+---------+
2 rows in set (0.00 sec)

A sensible action might be to select a book by name.

mysql> SELECT * FROM books WHERE title = 'pokemon';
+---------+
| title   |
+---------+
| Pokemon | 
| pokemon | 
+---------+
2 rows in set (0.00 sec)

But unfortunately this returns both books because in utf8_unicode_ci P equals p, in the same way a equals ä equals A. This is useful for ordering and searching but not for selecting.

We can fix it by specifying a binary collation algorithm for the select so it will not use these fuzzy equality rules.

mysql> SELECT * FROM books WHERE title = 'pokemon' COLLATE utf8_bin;
+---------+
| title   |
+---------+
| pokemon | 
+---------+
1 row in set (0.00 sec)

Note that this problem can introduce a lot of bugs and maybe even security problems. For instance, imagine two accounts: ‘Manfred’ and ‘manfred’. With the following query it is undetermined which of these two will be returned.

SELECT * FROM accounts WHERE username = 'manfred' LIMIT 1

My advise is to set the default collation for your database to utf8_bin and include the collation in queries where you want to order the entries nicely for the user interface or when you need fuzzy equality for searching. In Rails you can specify the collation in config/database.yml.

development:
  database: books_development
  adapter: mysql
  encoding: utf8
  collation: utf8_bin

Today we released a lightweight implementation of the web API for 37signal’s Campfire called Broach. It’s really handy to use in your notification scripts.

Broach.settings = { 'account' => 'example', 'token' => 'xxxxxx' }
Broach.speak('Office chat', 'The filesystem on Server 1 is almost full!')

If you want to try it out, you can simple install the gem.

$ gem install broach

You can find more details on the GitHub project page. The source is documented so you probably want to start reading there.

From version 0.9.8 onwards Mocha wants to be loaded after the test framework. I suspect this has something to do with detecting how to hook into the framework so it can run its expectations.

If you don’t do this, Mocha will not define its methods on the test case or context and you will get a load error.

NoMethodError: undefined method `mock' for #<#<Class>

Note that config.gem immediately loads the gem you’re configuring so you will also have to require those in the correct order.

config.gem 'test-spec'
config.gem 'mocha'

Last friday Eloy and I participated in the Ruby en Rails Rails Rumble 2009, we had 8 hours to write an application which helped developers find out more information about the dependencies in their Rails application.

Our submission is called Apprise. Apprise is a tool to check gem dependencies and external repositories for newer versions of the currently installed ones in your Rails application.

To be more specific, it looks for three sources of dependencies. Git and Subversion externals in vendor/plugins and bundled Gems in the Gemfile. Checking for Gem dependencies is delegated to Bundler. If you don’t use Bundler to manage Gem dependencies you will still see outdated externals and submodules.

Outdated dependencies are listed and you can decide to update the externals, submodules or gems.

$ apprise                 
Outdated dependencies
 * forestwatcher (Subversion external)
 * risosu-san (Git submodule)
 * miso (Gem)

If you want to try it out you can easily install the gem and run the command in the root of your Rails application.

$ gem install apprise --source http://gemcutter.org

Unfortunately we did not win, but our patch for Bundler to list outdated gems was pulled by Yehuda Katz right after the competition.