AirBlade Software

Give Me A Better # Key

2012-02-07T00:00:00+00:00

This is in reply to Give me back my # key! I was already doing my Vim thing but I didn’t realise I could map keys in readline until I read that article. Thanks Graeme!

If you use a Mac in the UK and often type #, you may find the Option-3 key combo annoying. I do.

Fortunately there’s another key nearby which I never use: the §/± key underneath Escape. We can map that to # and forget about the Option key.

Command line

To sort out your command line, add this to your ~/.inputrc:

"§": '#'

You’ll need to reload your readline configuration, most easily by restarting Bash.

Vim

Vim is slightly trickier. We want to map § to # in insert mode and on the command line (for : commands), but not in normal mode where # means the same as * but backwards. So far, so good. But we also want to be able to replace a character with # using r§, and move forward to the next # with f§.

Fortunately Ben Schmidt explained to me how to do this. Add this to your ~/.vimrc:

set iminsert=1
set imsearch=-1
noremap  § #
noremap! § #
lnoremap § #

IRB

Unfortunately IRB doesn’t pay attention to your ~/.inputrc, as far as I can tell, even though it is supposed to integrate with readline.

If anyone knows how to fix this, I’d love to hear about it!

Fixing Truncated, Accented Text In MySQL

2012-01-27T00:00:00+00:00

So when I loaded my data into MySQL the other day, MySQL silently truncated every piece of text at the first accented character. This affected about 13,000 records. The truncation was bad enough, though I suppose MySQL just couldn’t figure out what the byte stream meant at that point. But to do it silently?

Not cool.

Why did MySQL truncate my text? I have no idea.

The correct solution

Move to PostgreSQL.

The solution in the meantime

Find the affected records
Extract their original, intact values from the old database
Update the truncated records in the new database with the original values

I couldn’t find the affected records in the new database because the truncated data was truly lost. How do you query what’s not there?

So I loaded my data dump into a fresh development database. Strangely this worked despite having gone pear-shaped in production.

I then looked for all texty fields in the database containing a French or German accent (based on my customers' languages) and generated a SQL script with their values. The accent query below just wouldn’t work until I used the binary keyword.

Here’s the Rake task I wrote:

task :fix_accents => :environment do
  accents = %w[ ß ä ë ö ü é à è â ô ]
  ActiveRecord::Base.connection.tables.
    reject { |name| %w[delayed_jobs schema_migrations].include? name }.
    each do |table|
      klass = table.classify.constantize
      texty_columns = klass.columns_hash.
        select { |name,col| col.type == :string || col.type == :text }.
        map    { |name,col| name }.
        each do |name|
          accents.each do |accent|
            klass.select("id, `#{name}`").where("binary `#{name}` like '%#{accent}%'").each do |row|
              puts "UPDATE #{table} SET `#{name}` = #{row.send(name).inspect} WHERE id = #{row.id};"
            end
          end
        end
    end
end

I piped the output into a new file, tested a few key lines on the production database, then piped the whole thing into the production database. Problem solved.

I’m still moving to PostreSQL though.

Fixing MySQL's Illegal Mix of Collations

2012-01-24T00:00:00+00:00

I recently started to encounter MySQL’s dreaded illegal-mix-of-collations error. The occasional query would result in Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='.

MySQL’s default character set is latin1 with an adventurous Swedish collation, which presumably seemed like a good idea at the time. However Rails uses UTF-8 everywhere and from time to time the two collations collide.

Fortunately this problem is not new and clever people have already solved it: see Getting out of MySQL Character Set Hell and Fixing a MySQL Character Encoding Mismatch.

Those two articles advise more or less the same process. Here I’m just jotting down some notes where I found something confusing or ambiguous in case I ever have to do this again.

Order of events

After exporting the data and hacking the dump file, I wasn’t sure whether to configure MySQL for UTF-8 and then import the data, or the other way round.

I chose to import the data and then configure MySQL to avoid double-encoding the data. ~~It seems to have worked.~~ It silently truncated all my accented text but I was able to fix that.

Take stock

First figure out where you stand. In MySQL do this:

mysql> show variables like 'char%';

The result you want is this:

+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+

At this stage several of these values will be latin1.

Similarly for collation:

mysql> show variables like 'collation%';
+----------------------+-----------------+
| Variable_name        | Value           |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database   | utf8_general_ci |
| collation_server     | utf8_general_ci |
+----------------------+-----------------+

The utf8_general_ci collation will do but ideally we want utf8_unicode_ci. The former basically ignores all accents: it treats ‘ü’ the same as ‘u’. The latter pays attention to accents; it’s a tiny bit slower but more accurate.

To see what your tables are using:

mysql> show table status \G

…and look at the Collation field in each table.

Dumping your data as latin1

Here’s the command I used:

mysqldump -uUSER -pPASSWORD --quick --single-transaction --create-options --skip-set-charset --default-character-set=latin1 DATABASE > dump.sql

…where USER, PASSWORD, and DATABASE should be replaced appropriately.

My SQL dump was only 40MB so I was able to edit the tables' character sets in Vim (set :syn off for speed). In Vim:

:%s/DEFAULT CHARSET=latin1;/DEFAULT CHARSET=utf8;/

Configuring MySQL to use UTF8

I added this configuration to /etc/mysql/my.cnf (on Ubuntu 10.04):

[client]:
default-character-set=utf8

[mysqld]
default-character-set=utf8
default-collation=utf8_general_ci
character-set-server=utf8

Although the documentation claims there is a collation-server setting in the [mysqld] section, using it prevented my mysql (v5.1.41) from starting up. I ran up 30min of downtime trying to figure out why mysql wouldn’t start.

After I had done all this and restarted mysql, I found all the character_set_* variables were correct except character_set_database. To fix this:

mysql> alter database DATABASE default character set utf8;
mysql> alter database DATABASE default collate utf8_general_ci;

Verify with show create database DATABASE. (Documentation.)

Verifying everything worked

In the MySQL console I ran these:

mysql> show variables like 'char%';
mysql> show variables like 'collation%';
mysql> show create database DATABASE;
mysql> show table status \G

And then in the Rails console:

>> ActiveRecord::Base.connection.collation
"utf_general_ci"

Cleaning Up Code

2012-01-16T00:00:00+00:00

I've found a good trick for fixing crufty code is to pretend I'm going to open-source it. It's like inviting guests over to your house.

Maciej Cegłowski

I've used this trick for a while and it always works.

Logging Subdomains in Rails 3

2011-11-24T00:00:00+00:00

Rails 3.2 will have tagged logging so you can stamp your logs with subdomains (among other things). Until then you can use the Notifications API as follows.

In app/controllers/application_controller.rb:

# Put this before your other filters
before_filter :instrument_subdomain

private

def instrument_subdomain
  ActiveSupport::Notifications.instrument 'subdomain.action_controller', :subdomain => request.subdomain
end

In config/initializers/notifications.rb:

ActiveSupport::Notifications.subscribe 'subdomain.action_controller' do |name, start, finish, id, payload|
  Rails.logger.info "Subdomain: #{payload[:subdomain].presence || '[none]'}"
end

Now your logs will show each request’s subdomain. For example:

Started GET "/dashboard" for 123.456.42.153 at 2011-11-24 10:43:52 +0000
  Processing by DashboardsController#show as HTML
Subdomain: bigclient
Rendered dashboards/show.html.haml within layouts/account (15.4ms)
Completed 200 OK in 97ms (Views: 16.1ms | ActiveRecord: 32.3ms | Sphinx: 0.0ms)

This lets you see who’s doing what when you browse the logs.

In terms of bang for the buck, or utility per line of code, this was one of my better commits.

Managing Monit with Upstart

2011-11-21T00:00:00+00:00

I just moved my production server from a Centos 4.6 VPS to Ubuntu 10.04. I had been quite happy with the Centos — but I’ve discovered Ubuntu is far nicer. The packages are more up to date and it feels like considerably more thought has gone into it.

Ubuntu uses Upstart to manage its services. While Upstart is excellent at starting services and keeping them up, it has limited ability to restart services when they grow too big for their boots. Furthermore, not every service has been moved over to Upstart’s management; some — in particular Apache — are still managed the legacy init.d way.

So we need another process manager.

Monit

I like monit. It solves the Apache problem and the proactive-restart problem, but now we need to manage Monit itself. Upstart is the future so we’ll use that.

First we disengage init.d:

$ sudo /etc/init.d/monit stop
$ sudo update-rc.d -f monit remove

Now we configure an Upstart job in /etc/init/monit.conf:

start on runlevel [2345]
stop on runlevel [06]
respawn
exec /usr/locall/bin/monit -Ic /etc/monit/monitc

N.B. I’m not sure whether the stop runlevels should include 1.

Next we edit /etc/default/monit and change startup=0 to startup=1. (I’m not sure whether this is necessary but it doesn’t hurt.)

Finally we activate Upstart:

$ sudo start monit

Conclusion

We can use Monit to manage services which Upstart doesn’t yet manage, and Upstart to manage Monit. And Monit can cut tall poppies down to size whenever needed.

Hopefully in due course Upstart will encompass all of the above. In the meantime this works for me.

Export to Excel in Rails 3

2011-11-09T12:00:00+00:00

Recently I added the ability to export data to Excel from a Rails 3 webapp. There were pleasingly few lines of code.

The Code

In config/initializers/mime_types.rb:

Mime::Type.register 'application/vnd.ms-excel', :xls

In config/initializers/xls_renderer.rb:

ActionController::Renderers.add :xls do |xls, options|
  send_data(xls.respond_to?(:to_xls) ? xls.to_xls(options) : xls, :type => :xls)
end

In Gemfile:

gem 'to_xls', '~> 1.0.0'

In the view:

link_to 'Export to Excel', url_for(:format => 'xls')

In the controller:

def index
  @widgets = Widget.all
  respond_to do |format|
    format.html
    format.xls do
      render :xls => @widgets,
                     :columns => [ :name, :ref ],
                     :headers => %w[ Name Reference ]
    end
  end
end

The Tests

In test/unit/xls_renderer_test.rb:

require 'test_helper'
class XlsRendererTest < ActiveSupport::TestCase
  test 'xls mime type' do
    assert_equal :xls, Mime::XLS.to_sym
    assert_equal 'application/vnd.ms-excel', Mime::XLS.to_s
  end
end

In features/step_definitions/excel_steps.rb:

Then /^I should download an Excel document$/ do
  assert_equal 'binary',                   page.response_headers['Content-Transfer-Encoding']
  assert_equal 'application/vnd.ms-excel', page.response_headers['Content-Type']
  assert_equal 'attachment',               page.response_headers['Content-Disposition']
  assert page.source.length > 0  # not ideal but better than nothing
end

In features/whatever.feature:

Scenario: export to Excel
  When I go to the whatever page
  And I follow "Export to Excel"
  Then I should download an Excel document

References

The best source of information I found was José Valim’s Crafting Rails Applications. I like this book because it’s non-trivial and it explains everything test first.

The Social Graph

2011-11-09T00:00:00+00:00

Imagine the U.S. Census as conducted by direct marketers - that's the social graph.

Social networks exist to sell you crap. The icky feeling you get when your friend starts to talk to you about Amway, or when you spot someone passing out business cards at a birthday party, is the entire driving force behind a site like Facebook.

Because their collection methods are kind of primitive, these sites have to coax you into doing as much of your social interaction as possible while logged in, so they can see it. It's as if an ad agency built a nationwide chain of pubs and night clubs in the hopes that people would spend all their time there, rigging the place with microphones and cameras to keep abreast of the latest trends (and staffing it, of course, with that Mormon bartender).

We're used to talking about how disturbing this in the context of privacy, but it's worth pointing out how weirdly unsocial it is, too. How are you supposed to feel at home when you know a place is full of one-way mirrors?

We have a name for the kind of person who collects a detailed, permanent dossier on everyone they interact with, with the intent of using it to manipulate others for personal advantage - we call that person a sociopath. And both Google and Facebook have gone deep into stalker territory with their attempts to track our every action. Even if you have faith in their good intentions, you feel misgivings about stepping into the elaborate shrine they've built to document your entire online life.

Open data advocates tell us the answer is to reclaim this obsessive dossier for ourselves, so we can decide where to store it. But this misses the point of how stifling it is to have such a permanent record in the first place. Who does that kind of thing and calls it social?

Maciej Cegłowski, The Social Graph is Neither

Social Networks I’m Not In

I’m not on Facebook, Google+, or Flickr. I see they’re a convenient way of sharing photos and snippets of news, but I know they would decrease the quality of my friendships. I already struggle to find time regularly to meet, phone, or write to friends; pfaffing on a social network would use up that precious time and make me feel I’d done enough when, in reality, it’s not enough.

Also I don’t see why these networks should have all the intellectual property rights to my stuff.

Social Networks I Am In

I am on GitHub, Twitter, and LinkedIn. The difference is these are all for work: to try to get more business. Even this blog exists only to try to get people to give me money; it’s not a diary to show the world I am a unique and beautiful snowflake. (Obviously I am but I don’t feel the need to tell anyone that.)

DNS Downtime

2011-11-08T00:00:00+00:00

I use Dyn’s Standard DNS for my company’s and my customers' DNS. The service is excellent and the web interface first rate.

Until yesterday it was the only internet service I knew of with a 100% uptime — a remarkable feat. It had to end sometime…

So today, for the first time since 2001, we experienced a full 22 minute outage of our Dyn Standard DNS nameservers, which means that we reset our Dyn Standard DNS uptime counters back to zero.

Post Mortem: Attack To Dyn Standard DNS Nameservers

Dyn’s pricier Managed DNS platform kept on trucking without incident but unfortunately I’m not in a position to upgrade just yet!

When you think about all the things that have to operate when you ask your browser to pull up a URL, it’s amazing that it works at all.

Using rbenv in Production

2011-10-20T00:00:00+00:00

I just switched to using rbenv in production so I could easily upgrade to Ruby 1.9.2-p290. 37signals uses rbenv in production like this:

Each app is isolated through disciplined use of RubyGems/Bundler-style binstubs. We start with the stubs Bundler generates (bundle install --binstubs) and change the shebang from #!/usr/bin/env ruby to usr/bin/env ruby-local-exec. Update your Capistrano recipes, Chef cookbooks, Bluepill config, etc to use these binstubs throughout.

This draws a solid line between your app and your ops, meeting at just a handful of binstubs.

Now executing /u/apps/basecamp/current/bin/unicorn uses the Ruby in /u/apps/basecamp/current/.rbenv-version and the Unicorn bundled in /u/apps/basecamp/current/Gemfile.lock. Your app is self-contained. Your ops tools no longer know or care whether you use Bundler or whether you're on 1.8.7 or 1.9.3. Achievement unlocked.

Jeremy Kemper

Shebangs and binstubs

As far as I can tell, binstubs have two advantages:

Nothing else need know you use Bundler.
You don’t need to change into the app directory before running a command.

However I couldn’t get the Rails runner to work via a binstub, and I don’t mind if my “ops tools” know about Bundler. So I decided to forgo the binstubs and stick with bundle exec. Since I’m using the same Ruby everywhere on the server, via rbenv global 1.9.2-p290, it’s fine.

But I did need to update my app’s scripts' shebangs to ruby-local-exec; for example script/rails:

#!/usr/bin/env ruby-local-exec
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.

APP_PATH = File.expand_path('../../config/application',  __FILE__)
require File.expand_path('../../config/boot',  __FILE__)
require 'rails/commands'

Passenger

Don’t forget to re-install Passenger in your rbenv-managed Ruby, and rbenv rehash afterwards.

And on Apache, you’ll also need to update the Passenger configuration to use the new Ruby.

Non-app gems

You’ll also need to install (and rbenv rehash) non-Gemfile gems, such as those your crontab commands use. For example I use a gem via cron to backup the database to S3.

Conclusion

With rbenv managing Ruby, and Bundler managing my app’s gems, everything is nicely isolated. As Jeremey Kemper went on to say:

Now, say Ruby 1.9.3-rc1 just came out and you want to upgrade your app. Push the new package to your servers (using Chef or whatever), change .rbenv-version from 1.9.3-preview1 to 1.9.3-rc1, and cap deploy. That's it.

Didn't work out? cap deploy:rollback and you're back at 1.9.3-preview1. Ahh.

Which is lucky, because my Rails 3.0.10 app takes a glacial 3m47s to start up on Ruby 1.9.2-p290. As my friend Kamyar says, that’s not a computer number: ~4 minutes on modern CPUs corresponds to gazillions of instructions and we are not modelling the climate here.

So I’m going to take advantage of my easily-upgradable environment and upgrade to 1.9.3-rc1 as soon as possible.