How we used AMP Emails to get feedback on AMP Emails!
How we used AMP Emails to get feedback on AMP Emails!
AMP is considerably a nascent technology. When you are dealing with such an adolescent piece of tech, it is not the easiest thing to build a top-of-the-line production-ready feature for your product that ticks all your requirements.
Storytime!
In one of the quarters, we were trying to adopt the OKR way of developing features. Now features were just not features; they were initiatives. The process became more than just coding or solving a problem. It became about genuinely solving real customer problems to ensure they have a delightful user experience.
With this in mind, we built a dynamic reminder using AMP Emails that allowed the user to submit their expense report right from their emails.
During this process, we learned feedback from users and having other metrics like regular stats on the feature usage were essential steps to make the whole initiative successful. This required us to log and monitor feedback from users constantly.
The roadblocks we faced
The feature was outside our product, actually inside Gmail.
This led us to the questions:
How would we know what the user might have seen on the screen? And, how would we know the actions he performed as all of this was happening inside Gmail?
Now the problem to solve was to build an infra around:
How to log and monitor actions that are happening inside the email
How to get feedback from users considering there are so many limitations in AMP Emails
First, the monitoring part!
As mentioned earlier, AMP Emails being a nascent technology don’t have the required logging and monitoring support.
Our dynamic reminder is like a mini web app in itself. This resulted in us recording a lot of steps that were needed. We also realised, logging each of the events in all the screens was necessary to understand user behavior.
What did we do?
Our dynamic reminder is smart, and so are we 😎 . We handled most of the things in our backend notification service to keep up with all the good practices:
Exposed our APIs using signed URLs
Handled most errors in the backend
Queued infra to handle the load on emails
Had a common wrapper for all the fallback emails
Made all triggers agnostic of the AMP events
Most of the inter-service communication happened within the cluster, i.e., through Internal Calls. This made the API responses faster. (You don't want the loader to spin forever!)
Fun fact: While building all of this we used telepresence to create a two-way network proxy for our stag environment's Kubernetes pod. This helped build and test real-life scenarios to emulate the whole email sending cycle end to end.
The Monitoring & Logging Part!
We logged everything into Mixpanel from our backend service and later created a report out of it. This helped us know at what step the user would drop. This enabled us to accurately predict what the user could've seen on his email before dropping off.
Since we log other events from the web app too, we could combine events to map a conclusion to know when a particular user came from the email to the web app or if he saw something on the email and then took some action.
Problem 1 solved!
The feedback?
How do we take feedback?
Well, we used AMP to get feedback on AMP!
We built a small infra for triggering an AMP email whenever our dynamic reminders were triggered using a few Google Sheets and webhooks. The responses were directly sent on a slack channel.
How was all this hard work helpful?
We got regulars usage stats which helped us with our OKRs.
- Over conversion rate was 27%
- On average, it took a user just 10-13 seconds to select and submit his expenses belonging to different categories from the moment he opened the email
- 72% of the total feedback was a 5/5 star rating!
Users could directly send their feedback right away. And we could act upon it quickly.
We could observe user behavior and take the necessary steps to improve the feature.
Overall, we got some great feedback; here are a few!
All in all, this was fun!