Fog and S3 multipart uploads

I was hitting Excon::Errors::SocketError: Broken pipe (Errno::EPIPE) when trying to upload relatively small file (30MB) to S3. The most common suggestion is to set region when creating Fog’s storage object.

Fog::Storage.new(
  provider:              'AWS',
  aws_access_key_id:     aws_access_key_id,
  aws_secret_access_key: aws_secret_access_key,
  region:                'us-east-1',
)

Unfortunately this didn’t help.

To allieviate that problem I decided to go route of S3 multipart uploads, which Fog supports. The problem it’s not exposed in any friendly way. To get multipart uploads you need to do two things: make sure body is a file like object (with at least #read method) and to set multipart_chunk_size on the file which you want to save.

file = bucket.files.new(
  key: filename,
  body: StringIO.new(content)
)
file.multipart_chunk_size = 5242880  # 5MB
file.save

When those two conditions are met fog will happily upload file using S3 multipart upload, which in my case eliminated those broken pipe errors.