Sorry this page looks weird. It was automatically migrated from my old blog, which had a different layout and different CSS.

Resizing and Uploading Images Asynchronously to S3

Here’s a pattern for processing user-uploaded images in the background and storing them on S3. It uses Paperclip and Delayed Job:

### Models ###

class Person < ActiveRecord::Base
  has_attached_file :local_image,
                    path: ":rails_root/public/system/:attachment/:id/:style/:basename.:extension",
                    url:  "/system/:attachment/:id/:style/:basename.:extension"

  has_attached_file :image,
                    styles: {large: '500x500#', medium: '200x200#', small: '70x70#'},
                    convert_options: {all: '-strip'},
                    storage:         :s3,
                    s3_credentials:  "#{Rails.root}/config/s3.yml",
                    s3_permissions:  :private,
                    s3_host_name:    's3-eu-west-1.amazonaws.com',
                    s3_headers:      {'Expires'             => 1.year.from_now.httpdate,
                                      'Content-Disposition' => 'attachment'},
                    path:            "images/:id/:style/:filename"

  after_save :queue_upload_to_s3

  def queue_upload_to_s3
    Delayed::Job.enqueue ImageJob.new(id) if local_image? && local_image_updated_at_changed?
  end

  def upload_to_s3
    self.image = local_image.to_file
    save!
  end
end

class ImageJob < Struct.new(:image_id)
  def perform
    image = Image.find image_id
    image.upload_to_s3
    image.local_image.destroy
  end
end

### Views ###

# app/views/people/edit.html.haml
# ...
= f.file_field :local_image

# app/views/people/show.html.haml
- if @person.image?
  = image_tag @person.image.expiring_url(20, :small)
- else
  = image_tag @person.local_image.url, size: '70x70'

What’s going on?

Although we ultimately store images on S3, we use the server’s file system as a temporary cache. So when a user uploads an image it’s held as a local_image on the file system, which makes the upload as fast as possible. We don’t resize or process the image at this stage because that would slow things down.

When a local_image is saved, an after_save callback pushes a job onto the queue for background processing if appropriate. The job simply assigns the local image on the file system to the :image attribute, whereupon the image is processed, resized, and uploaded to S3. Once all that is done, the local image is removed.

In the meantime, we let the browser resize the local image if someone needs to see it.

How does this differ from other solutions?

Surprisingly, although allowing users to upload images is a standard webapp feature, and thus resizing and uploading images to S3 asynchronously is a standard problem, there are few standard solutions.

Paperclip and CarrierWave don’t support asynchronous operation out of the box.

Delayed Paperclip solves this problem but, because it’s designed to work on Heroku and therefore not touch the local file system, it blocks the user while uploading the initial file to S3 instead of the file system. Thus it’s slower than the pattern presented above.

Mark Dodwell showed a nice solution in Paperclip, S3 & Delayed Job in Rails, but although it defers image resizing to the background it still uploads to S3 in the foreground.

Roberto Soares' Paperclip Recipes was the basis for much of the pattern above, but he defines his image styles not only on the (S3-stored) image but also on the local image. This means the user’s image upload is held up unnecessarily by resizing, which could be done by the browser.

The Paperclip Recipes post shows a couple of tips to hide the fact you’re working with two attachments, but I prefer to handle them explicitly.

Does it work for me?

I’ve been using this pattern in production for two months and it’s going swimmingly.

Will it work for you?

The pattern above will suit you if:

The pattern uses Paperclip and Delayed Job but I expect you could adapt it for alternatives such as CarrierWave and Resque.

Andrew Stewart • 26 June 2012 • Rails
You can reach me by email or on Twitter.