Sharing Files using S3 Pre-signed URLs

Amazon’s S3 is a reliable, cheap way to store data. We use it to store user-uploaded images and documents as s3 objects. By default, objects stored in S3 are only accessible to the owner. Let’s say you are building a photo storage application and need to display the stored photo in the app, how do you go about it? You have two options:

Option 1: Your service reads from S3 and sends it to client

You can implement http endpoints like https://myawesomeapp.com/images/3434323 and your service, in turn, talks to S3 using the right keys to get access to the data. You can then stream the results from S3 back to the user using code like this.

public Response img(@PathParam("file_id") String fileId) throws Exception {
    StreamingOutput stream = new StreamingOutput() {
      public void write(OutputStream output) {
        InputStream is = s2helper.fileRead(fileId);
        IOUtils.copy(is, output);
      }
    };
    return Response.ok(stream).header("content-disposition", "attachment; filename = report.pdf").build();
}

This approach has two problems:

1. If you wish to embed images in your site using <img> tags, this results in a GET request from the browser. You need to be able to attach auth information so your service knows if you can actually access the image. The way to do that is cookies. The browser automatically attaches cookies for the domain. This gets a little tricky if you are accessing these files using AJAX / CORS (see this SO post for a discussion) — but it is solvable.

2. The second problem is that your server now becomes the bottleneck for serving images and files. Every file request now goes from client -> your server -> s3 and back. It’s two hops — so it’s slower. It also keeps your connection around for longer and that may become problematic as your service becomes more popular.

Option 2: S3 Pre-signed URLs

S3 pre-signed URLs allow you to create a short-lived URL for an S3 file that is accessible by anyone you share the URL with. In your server, you can generate this URL using the AWS SDK. Here’s a little Java snippet:

public URL fileDownloadUrl(String s3Url) {
    java.util.Date expiration = new java.util.Date();
    long msec = expiration.getTime();
    msec += 1000 * 60 * 2; // 2 minutes
    expiration.setTime(msec);

    GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(s3bucket, s3Url);
    generatePresignedUrlRequest.setMethod(HttpMethod.GET); // Default.
    generatePresignedUrlRequest.setExpiration(expiration);

    URL s = s3.generatePresignedUrl(generatePresignedUrlRequest);
    return s;
  }

This generates an S3 url like:

https://<bucket-name>.s3.amazonaws.com/bae728c7-a7a3-4942-b9b5-3ca0…-b91126bb3d8f.image.jpg?AWSAccessKeyId=<AWS_ACCESS_KEY>&Expires=145671

You can then pass this URL back to the client app who can then make the GET request which is served by AWS directly. The highlights of this approach are:

1. You can have a REST end-point that returns this URL. Use your favorite auth mechanisms — cookies, JSON web tokens whatever to make sure only legitimate requests are allowed. JSON web tokens are pretty cool and I’ll write a post about them in the future.

2. This URL is technically publicly accessible till the expiry time. Typically, you want to give a short timeframe, like 3 mins, by which the client should be making the request. To allow the browser to make AJAX calls, you’ll need to make one bucket-level configuration to enable CORS. See this article on how to do it. Note that this particular access (GET request to S3) is not authenticated. However, step 1 + combined with short expiry time gives you a good level of security.

Using signed S3 urls, your heavy-duty images and files aren’t flowing through your server anymore. This approach can vastly improve loading time and reduce the burden on your service.

As usual, comments and suggestions are welcome. Ping me on twitter at @k2_181