In an application we wrote back in 2004 I found MD5 hashed passwords. We decided this was too weak for modern standards so we wanted to switch to bcrypt. During the move we wanted the user to be affected as little as possible.

In order to compute the crypted password we need the cleartext version. We only have a hashed version so the user has to type her password. Luckily they do this every time they authenticate, so that is a nice opportunity to upgrade their password.

First I added a crypted_password column to the accounts table. We now have two columns for storing the password: the old hashed_password and the new crypted_password.

add_column :accounts, :crypted_password, :string

After that we updated the password accessor methods; assignment and verification.

class Account
  def password=(password)
    if new_record? or !password.blank?
      self.crypted_password = BCrypt::Password.create(password)
    end
  end
  
  def has_password?(input)
    BCrypt::Password.new(crypted_password) == input
  rescue BCrypt::Errors::InvalidHash
    false
  end
end

Now need to make sure we can authenticate with both the hashed as well as the crypted password stored for an account.

class Account
  def self.authenticate_with_crypt(params={})
    if account = find_by_username(params[:username]) and
       account.has_password?(params[:password])
      account
    end
  end
  
  def self.hash_password(password)
    ::Digest::MD5.hexdigest(password)
  end
  
  def self.authenticate_with_md5(params={})
    find_by_username_and_hashed_password(
      params[:username],
      hash_password(params[:password])
    )
  end  
end

Finally we need to make sure the password automatically updates. We try to authenticate using bcrypt. BCrypt raises an exception when the crypted_password is blank. This makes authentication fail and we fall back to trying the hashed password. When authentication with a hashed password succeeds we know the cleartext password and we can update it.

class Account
  def self.authenticate(params={})
    if account = authenticate_with_crypt(params)
      account
    elsif account = authenticate_with_md5(params)
      account.password = params[:password]
      account.hashed_password = nil
      account.save!
      account
    else
      account = Account.new(params.slice(:username, :password))
      account.errors.add_to_base("The credentials you entered are"
        "invalid. Please try again.")
      account
    end
  end
end

This solution will leave a group of users with a hashed password indefinitely. After a few months we could decide to throw away the hashed passwords. This means that infrequent users will have to reset their password if they do decide to log in again. It could cause some support requests, but I think we can handle them.

Note that this change doesn’t make the application safer. In case we leak information or when the database is somehow stolen it will make it harder to recover passwords. A lot of people use the same password for multiple accounts so this will give them time to reset their other accounts in case it is compromised. The change only took 15 minutes in this application so it’s totally worth the time.

As you probably already know the String class became encoding aware in Ruby 1.9. This makes it possible to manipulate strings on the character level instead of on byte level. However, it’s still a general purpose API which means writing a few lines of code to get stuff done.

It’s common to choose one internal representation for character data in an application and convert all incoming strings to this representation. For example, in modern applications strings are often encoded in UTF-8 or UTF-16. I took some time to figure out how to do this in Ruby 1.9.

The biggest problem with receiving data from external sources is trust. Sources can lie about their encoding or provide broken data. Sometimes it gets mangled accidentally and sometimes someone is attacking your application with a carefully crafted payload.

Problems can arise on a lot of levels. Just think about receiving an HTTP response from a webserver. Things can go wrong in the proxy, in the client library, in the string implementation of your language. Meta-data about the encoding is stored in HTTP headers, the HTML, and now in String. The same problems exist with data coming from databases, filesystems, and caches.

You can trust some of these sources more than others. For example, you can control the data going into a database so you can trust the data coming out. In contrast, anything coming from the internet should be considered potentially dangerous.

My solution for these problems is a new method on String called ensure_encoding. It makes sure the data in the string is at least compatible with your internal strings. Depending on the options you pass it will respond differently to broken data.

As an example, let’s take an HTTP POST to a web API. Assume we’ve explained in the API documentation we only accept UTF-8 character data and will be very strict about this. Our code might look something like this:

require "ensure/encoding"
begin
  params.each do |key, value|
    params[key].ensure_encoding!(Encoding::UTF_8,
      :external_encoding  => Encoding::UTF_8,
      :invalid_characters => :raise)
  end
rescue Encoding::InvalidByteSequenceError => e
  send_response_document :unprocessable_entity,
    "Sorry, your request contains invalid encoding" +
    "and can't be processed (#{e.message})"
end

You can find more examples on the GitHub project page and in the source. Normally I try to extract code from a running project, but we don’t run any production code on 1.9 yet. It would be great if you can help out with testing the code. I’ve released the code as a gem, so it should be really easy to install.

$ gem install ensure-encoding

Please leave any bugs, problems, or suggestions in the GitHub issue tracker.

Today we released a lightweight implementation of the web API for 37signal’s Campfire called Broach. It’s really handy to use in your notification scripts.

Broach.settings = { 'account' => 'example', 'token' => 'xxxxxx' }
Broach.speak('Office chat', 'The filesystem on Server 1 is almost full!')

If you want to try it out, you can simple install the gem.

$ gem install broach

You can find more details on the GitHub project page. The source is documented so you probably want to start reading there.