Goodbye, Heroku! Hello, Docker, Kubernetes on AWS!

It has been a couple of months since my last post. We’ve been lucky to get some rockstar developers to join us in our journey. One major infra change that we did was to move from Heroku to Kubernetes. We will also write a series of posts on how we’ve setup our kubes, but this post is about why we moved off Heroku.

Heroku was fantastic for us in the early stages. It allowed us to write services in different languages. It allowed us to setup custom domains and SSL easily enough. It allowed us to push updates with just a git push. Most importantly, deploying to Heroku helped maintain all dependencies cleanly — we didn’t fall into the trap of installing custom libraries on the production machine by hand and then forget about it.

So why did we move off Heroku? The main reasons were:

  • We were running our services in us-east region with Heroku whereas our customers were in India. This caused some noticeable latency issues when we started getting serious users. Heroku only supports us-east and eu regions currently.
  • We wanted to setup a staging environment (we’re growing up!). We had deployed our services as separate apps in Heroku (we had 5 IIRC). It seemed error prone to maintain two environments by hand. Interestingly, Heroku came up with Pipelines around the time we were migrating.
  • Usage was getting to a point when we couldn’t afford downtime. We needed to have rolling deploys with no downtime. While this is theoretically possible with Heroku, it would tie us too tightly to Heroku.
  • We were getting feedback from prospects that bigger companies would want a private deployment of our product within their infrastructure (or within the geographical region). This would be impossible to do with Heroku.

Given all of these considerations, we started looking at alternatives. We figured we needed three pieces: a container mechanism, a container orchestration system and a public cloud provider. After doing a little bit of research, we decided to go with Docker, Kubernetes and AWS.

Why AWS and not Google? We had a bunch of AWS credits (thanks, folks!). I was very familiar with AWS. At my previous workplace at Qubole, we used to hammer EC2 and S3 and I was super-impressed by how robust AWS was. Finally, we knew AWS was launching in India soon (and they did). As an aside, AWS seems to be routing credits to early stage startups through events, shared spaces and investors and getting them hooked early — Google really needs to up its sales game.

Some of you may be wondering what the hell is a Docker and a Kubernetes and why should I care? This is a little beyond the scope of this article — but I will use a food-based analogy to summarize.

www.abrowntable.com

The service you write is the food you prepare and eat. In the above picture, it happens to be noodles. It could very well be a hot curry. Or some breakfast cereal. Often times, you end up preparing and eating a couple of different dishes. Similarly, your services could be written in Python, Go, Java or whatever.

The bowl that holds on the food while you eat is the container. The same bowl can hold a variety of dishes. Bowls do come in different sizes. Usually, people don’t create a new bowl when they cook something new. Similarly, you shouldn’t write a bunch of new bash scripts from scratch to package a new service. Docker is the biggest player here and we went with it, but there are a couple of alternatives like Rocket. Whenever we want to deploy a new version of an app, we would create a new docker image.

Finally, the table on which the bowls sit is the orchestration system. The system must support placement of multiple bowls. The system must allow for adding an extra bowl of rice before removing the old one so that hungry people aren’t left waiting. This is termed a rolling upgrade. If the soup is really popular, it must support adding another bowl of soup. This is replica management and load balancing. Finally, containers need to be able to discover and talk to each other — there really isn’t an analogy for this :)

We compared Kubernetes and Docker Swarm and it was a difficult choice — both were attacking the same problem and both were in relatively early stages. Kubernetes seemed more mature and it seemed like more organizations were running Kubernetes in production than Docker Swarm. So we went with Kubernetes. It would be interesting to see how this ecosystem develops.

It has been almost 2 months since we migrated to Docker / Kubernetes running in AWS Singapore and we’ve been super happy. Kubernetes has been pretty solid and our services seems much snappier — I suspect Heroku had additional overheads. We managed to setup a staging environment within a day after getting production environment running. We also run kubernetes on our laptops for development — this was probably the one hard part and there wasn’t much on the web on how to do this. These are all topics for future posts — so stay tuned.

In conclusion, I’d say Heroku was a great choice for us to get started. It allowed us to not worry about deployment and upgrades and I highly recommend it for early stage startups. It was time for us to move on, though and so it’s goodbye, Heroku!

Siva Narayanan

I am known to be "the CTO of one, the father of two, and the roasting baba of many."

More of our stories from

Engineering
How we created a Medium-like blurry background effect

Here's how we improved user experience, decreased load time and made Fyle accessible for users without a fast internet.

Read more...
Bye bye WordPress, welcome Webflow.

This blogpost documents our journey as we bid goodbye to WordPress and migrated to Webflow.

Read more...
How we reduced our website build time by 59%

I came up with five 3 second changes to reduce the build time by over 59%. Here's more about my experience.

Read more...
Hello, Web Technologies!

I’m a first-time entrepreneur and I’ll be recording my learnings and experiments over time. I am always eager to interac

Read more...
The Non-Boring Guide to OAuth 2.0

If you’re developing an application that needs access to a user’s Google / Facebook / LinkedIn information, you’ll need

Read more...
Dealing with Nested Objects in your Web Application

A couple of weeks ago, I ran into a peculiar problem that I think might be useful to talk about. It took me a bunch of

Read more...
Eliminate Boilerplate Java code with Lombok

I’ve been writing a lot of boilerplate Java code, lately — getters, setters, hashCode, equals and toString. Actually, I’

Read more...
Hello, Web Technologies! — Part II

This is a follow-up to my first post about technology choices I made while building out our product. I wanted to pen my

Read more...
Sharing Files using S3 Pre-signed URLs

Amazon’s S3 is a reliable, cheap way to store data. We use it to store user-uploaded images and documents as s3 objects

Read more...
JSON Web Token Concepts

There are many technical articles about JSON web tokens (JWT) on the interwebs, but I haven’t found one that explains...

Read more...

All Topics