RecCenter Blog: Automated Asset Sync to Amazon S3 With Engineyard Deploy

I recently moved all static assets for a client project off the main application server and onto Amazon S3. I needed an easy way for these assets to be synced to S3 on every deploy so that all of my team members could continue to work and deploy as the normally do, and that required hooking into the Engineyard CLI deploy process using deploy hooks. Additionally i wanted the sync to happen on the server since moving files to S3 from an EC2 instance is much faster than syncing from local machines. This required a custom chef recipe to install my S3 syncing library of choice, s3cmd *note that there are several tools with the same name - we only want the python one from http://s3tools.org/s3cmd.

I keep all required cookbooks with the application in the /cookbooks directory similar to my setup for the seamless_database_pool plugin on engineyard with n-slaves, so i created the appropriate directories and files:

/cookbooks
  /main
    /recipes
      default.rb # enable the cookbook here by including "require 's3cmd'"
  /s3cmd
    /recipes
      default.rb # utilize the following code

And added the chef code to install and link the s3cmd binary:

# /cookbooks/s3cmd/recipes/default.rb
# Cookbook Name:: s3cmd
# Recipe:: default

s3cmd_tar_gz = "/engineyard/portage/distfiles/s3cmd-1.0.0.tar.gz"

remote_file s3cmd_tar_gz do
  checksum "e82f0246479015ce50a09d8d4ada8429"
  source "http://downloads.sourceforge.net/project/s3tools/s3cmd/1.0.0/s3cmd-1.0.0.tar.gz"
end

bash "install s3cmd 1.0.0" do
  cwd "/engineyard/portage/distfiles/"
  code <<-EOH
    tar -zxf #{s3cmd_tar_gz}
    cd /engineyard/portage/distfiles/s3cmd-1.0.0 && sudo ln -nfs $PWD/s3cmd /usr/local/bin/s3cmd
  EOH
  not_if { ::FileTest.exists?("/usr/local/bin/s3cmd") }
end

Once this was setup, I applied the recipe to my server using the engineyard cli:

ey recipes upload
ey recipes apply
# if you have issues check your custom log on the ey cloud panel

I then set about creating a rake task that could be run on every deploy that would be FAST and not duplicate efforts using s3cmd. I created the config using s3cmd --configure to reference in the rake task. This file must also exist on the server so make sure your rake task can find it both places.

# /lib/tasks/asset_deploy.rake
require 'rubygems'
require 'aws/s3'

## You should probably pull these from elsewhere with YAML or ENV['CONSTANT]. I am not here to judge you.
AWS_BUCKET = "YOUR_S3_BUCKET_"

## I like to namespeace my rake tasks. 
namespace :assets do 
  desc "Deploy all assets in public/**/* to S3/Cloudfront"
  task :deploy do
    STDOUT.sync = true
    installed = system("which s3cmd > /dev/null 2>&1")
    unless installed
      print "run `brew install s3cmd` and try again - you do not have the s3cmd binary installed."
    else
      print "== Uploading assets to S3"
      # you can specify as many directories as you like - make sure your local and remote match up properly. i use gsub because i am lazy
      ["public/images/*","public/stylesheets/*", "public/assets/*", "themes/*"].each do |local_path|
        remote_path = local_path.gsub("*", "").gsub("public/", "")
        print "Uploading #{local_path} to #{remote_path} \n"
        # my theme directory also has templates, but i dont want them on S3. read the docs for the other options explanations
        print `s3cmd -c .s3cfg --skip-existing  --rexclude=".*\.erb|.*\.haml" --acl-public -rvMH sync #{local_path} s3://#{AWS_BUCKET}/#{remote_path}`
      end
      
      print "== Done uploading assets to S3"
    end
    STDOUT.sync = false
  end

end

And the final piece of the puzzle that makes this all work like magic - the deploy hook. I chose “/deploy/before_migrate.rb” and included the following:

# i only want this to run once, and from the master is fine. easier to target than some random non-master.
if ['app_master'].include?(node[:instance_role])
  run "cd #{release_path} && bundle exec rake assets:deploy"
end

Fin. Any Questions? How do you keep your assets in order?

Comments