Sharing Files using S3 Pre-signed URLs

Amazon’s S3 is a reliable, cheap way to store data. We use it to store user-uploaded images and documents as s3 objects. By default, objects stored in S3 are only accessible to the owner. Let’s say you are building a photo storage application and need to display the stored photo in the app, how do you go about it? You have two options:

Option 1: Your service reads from S3 and sends it to client

You can implement http endpoints like https://myawesomeapp.com/images/3434323 and your service, in turn, talks to S3 using the right keys to get access to the data. You can then stream the results from S3 back to the user using code like this.

This approach has two problems:

  1. If you wish to embed images in your site using <img> tags, this results in a GET request from the browser. You need to be able to attach auth information so your service knows if you can actually access the image. The way to do that is cookies. The browser automatically attaches cookies for the domain. This gets a little tricky if you are accessing these files using AJAX / CORS (see this SO post for a discussion) — but it is solvable.
  2. The second problem is that your server now becomes the bottleneck for serving images and files. Every file request now goes from client -> your server -> s3 and back. It’s two hops — so it’s slower. It also keeps your connection around for longer and that may become problematic as your service becomes more popular.

Option 2: S3 Pre-signed URLs

S3 pre-signed URLs allow you to create a short-lived URL for an S3 file that is accessible by anyone you share the URL with. In your server, you can generate this URL using the AWS SDK. Here’s a little Java snippet:

This generates an S3 url like:

You can then pass this URL back to the client app who can then make the GET request which is served by AWS directly. The highlights of this approach are:

  1. You can have a REST end-point that returns this URL. Use your favorite auth mechanisms — cookies, JSON web tokens whatever to make sure only legitimate requests are allowed. JSON web tokens are pretty cool and I’ll write a post about them in the future.
  2. This URL is technically publicly accessible till the expiry time. Typically, you want to give a short timeframe, like 3 mins, by which the client should be making the request. To allow the browser to make AJAX calls, you’ll need to make one bucket-level configuration to enable CORS. See this article on how to do it. Note that this particular access (GET request to S3) is not authenticated. However, step 1 + combined with short expiry time gives you a good level of security.

Using signed S3 urls, your heavy-duty images and files aren’t flowing through your server anymore. This approach can vastly improve loading time and reduce the burden on your service.

As usual, comments and suggestions are welcome. Ping me on twitter at @k2_181

Siva Narayanan

I am known to be "the CTO of one, the father of two, and the roasting baba of many."

More of our stories from

Engineering
How we created a Medium-like blurry background effect

Here's how we improved user experience, decreased load time and made Fyle accessible for users without a fast internet.

Read more...
Bye bye WordPress, welcome Webflow.

This blogpost documents our journey as we bid goodbye to WordPress and migrated to Webflow.

Read more...
How we reduced our website build time by 59%

I came up with five 3 second changes to reduce the build time by over 59%. Here's more about my experience.

Read more...
Hello, Web Technologies!

I’m a first-time entrepreneur and I’ll be recording my learnings and experiments over time. I am always eager to interac

Read more...
The Non-Boring Guide to OAuth 2.0

If you’re developing an application that needs access to a user’s Google / Facebook / LinkedIn information, you’ll need

Read more...
Dealing with Nested Objects in your Web Application

A couple of weeks ago, I ran into a peculiar problem that I think might be useful to talk about. It took me a bunch of

Read more...
Eliminate Boilerplate Java code with Lombok

I’ve been writing a lot of boilerplate Java code, lately — getters, setters, hashCode, equals and toString. Actually, I’

Read more...
Hello, Web Technologies! — Part II

This is a follow-up to my first post about technology choices I made while building out our product. I wanted to pen my

Read more...
Sharing Files using S3 Pre-signed URLs

Amazon’s S3 is a reliable, cheap way to store data. We use it to store user-uploaded images and documents as s3 objects

Read more...
JSON Web Token Concepts

There are many technical articles about JSON web tokens (JWT) on the interwebs, but I haven’t found one that explains...

Read more...

All Topics