Sharing Files using S3 Pre-signed URLs

Amazon’s S3 is a reliable, cheap way to store data. We use it to store user-uploaded images and documents as s3 objects. By default, objects stored in S3 are only accessible to the owner. Let’s say you are building a photo storage application and need to display the stored photo in the app, how do you go about it? You have two options:

Option 1: Your service reads from S3 and sends it to client

You can implement http endpoints like https://myawesomeapp.com/images/3434323 and your service, in turn, talks to S3 using the right keys to get access to the data. You can then stream the results from S3 back to the user using code like this.

This approach has two problems:

  1. If you wish to embed images in your site using <img> tags, this results in a GET request from the browser. You need to be able to attach auth information so your service knows if you can actually access the image. The way to do that is cookies. The browser automatically attaches cookies for the domain. This gets a little tricky if you are accessing these files using AJAX / CORS (see this SO post for a discussion) — but it is solvable.
  2. The second problem is that your server now becomes the bottleneck for serving images and files. Every file request now goes from client -> your server -> s3 and back. It’s two hops — so it’s slower. It also keeps your connection around for longer and that may become problematic as your service becomes more popular.

Option 2: S3 Pre-signed URLs

S3 pre-signed URLs allow you to create a short-lived URL for an S3 file that is accessible by anyone you share the URL with. In your server, you can generate this URL using the AWS SDK. Here’s a little Java snippet:

This generates an S3 url like:

You can then pass this URL back to the client app who can then make the GET request which is served by AWS directly. The highlights of this approach are:

  1. You can have a REST end-point that returns this URL. Use your favorite auth mechanisms — cookies, JSON web tokens whatever to make sure only legitimate requests are allowed. JSON web tokens are pretty cool and I’ll write a post about them in the future.
  2. This URL is technically publicly accessible till the expiry time. Typically, you want to give a short timeframe, like 3 mins, by which the client should be making the request. To allow the browser to make AJAX calls, you’ll need to make one bucket-level configuration to enable CORS. See this article on how to do it. Note that this particular access (GET request to S3) is not authenticated. However, step 1 + combined with short expiry time gives you a good level of security.

Using signed S3 urls, your heavy-duty images and files aren’t flowing through your server anymore. This approach can vastly improve loading time and reduce the burden on your service.

As usual, comments and suggestions are welcome. Ping me on twitter at @k2_181

Siva Narayanan

I am known to be "the CTO of one, the father of two, and the roasting baba of many."

More of our stories from

Engineering
Demystifying Class Variables In Python

Understanding class variables in python

Read more...
Interview Experience: Backend Engineering Internship at Fyle

Wanna know the secret to crack backend engineering interviews? Learn them here and intern at Fyle!

Read more...
The curse of being a Senior Engineer, how to deal with timelines, frustrations, etc

Being a good developer is 50% skill and 50% emotional support; here's my secret to balancing both at the right amount!

Read more...
How did I build cropping of receipts in the mobile app?

Follow Yash's journey of what it takes to reduce manual work for our customers when receipts come in all shapes and sizes!

Read more...
How did we increase Data Extraction accuracy by a whopping ~50%?

Wanna know the secret of data extraction, the complex machine learning models we use, the experiments we did? Read on...

Read more...
The not so secret sauce of my work

From chaos to clarity, follow Chethan's not so secret sauce to excelling at work!

Read more...
From Zero to Hero: The Policy Tests Journey!

The story of policy tests at Fyle

Read more...
How Fyle changed my life from a naive intern to a confident Engineering Lead

A blogpost that documents Shwetabh's journey at Fyle.

Read more...
Vikas Prasad @ Fyle

This document is a user guide to Vikas at work.

Read more...
Gokul K's README

This document is a user guide to Gokul at work.

Read more...

All Topics