Fog and S3 multipart uploads
Digging through issues and source code
I was hitting
Excon::Errors::SocketError: Broken pipe (Errno::EPIPE) when trying to upload relatively small file (30MB) to S3. The most common suggestion is to set region when creating Fog’s storage object.
Fog::Storage.new( provider: 'AWS', aws_access_key_id: aws_access_key_id, aws_secret_access_key: aws_secret_access_key, region: 'us-east-1', )
Unfortunately this didn’t help.
To allieviate that problem I decided to go route of S3 multipart uploads, which Fog supports. The problem it’s not exposed in any friendly way. To get multipart uploads you need to do two things: make sure body is a file like object (with at least
#read method) and to set
multipart_chunk_size on the file which you want to save.
file = bucket.files.new( key: filename, body: StringIO.new(content) ) file.multipart_chunk_size = 5242880 # 5MB file.save
When those two conditions are met fog will happily upload file using S3 multipart upload, which in my case eliminated those broken pipe errors.