The way we migrate APIs - Projects API Migration

Navigating API Migrations: Insights and Lessons from the Journey.

Introduction

Hi, I'm Sahil K., currently working as MTS1 at Fyle. When I joined Fyle as an intern, I was very new to the world of Angular. During my internship, I worked on fixing many UI and functionality-related bugs in the Fyle App and also on two product roadmap initiatives.

The latest initiative I have just wrapped up and am writing about is “Projects API Migration.” To date, I have finished migrating the mobile app Projects APIs and web app Projects APIs across all personas. This blog is intended to share my experiences and insights, especially for new interns who may initially find API migrations challenging.

Why This Blog?

When I was assigned this initiative as an intern, it felt overwhelming at first. Migrating all the Projects APIs from settings, admin, spender, especially upsert-expense, and mobile app seemed challenging, but I was ready for it.

Before starting with the coding part, I had many questions: Why do API migration? How exactly to do it? How to write the Engineering Doc? What kind of impact will API migration have? I took many calls with other developers who had done API migrations to get started. Special thanks to Dimple KH, Devendra Singh Rana, and Aniruddha Shriwant for their support.

This blog is intended to be a go-to resource for new interns working on API migrations. For the senior engineers, while you already know the why, what, and how of API migrations, I’ve included some interesting stories and insights as well. So, without further ado, let's dive into Projects API migration.

What exactly is API migration? Why do we do it in the first place?

API migration is a little complex but rewarding process that can significantly enhance an application’s performance and capabilities. It involves transitioning from an old API to a new one. At Fyle, we currently have three versions of APIs: v1 (public), v2, and Platform. Each version has its own SQL Database table. Since Fyle deals with transactions and finance-related data, maintaining consistency is crucial, and we cannot rely on eventual consistency.

Here are the key reasons for API migration:

• Consistency and Accuracy: Financial transactions require absolute accuracy and consistency. We need to ensure that all data remains consistent across different API versions and their respective database tables.

• Reduced Latency: To maintain consistency across all tables, we face increased latency due to the master-slave architecture. When a new write request is made on the platform APIs database, the related tables for v1 and v2 also need to be updated. This additional processing can result in extra latency per request. While this might seem minor for a single user, it scales significantly for thousands of users, introducing much more latency. By migrating everything to the Platform API, we can eliminate the need to update multiple tables, thus reducing latency.

• Performance Improvement: New platform APIs are optimized for better performance, handling more requests per second and providing faster response times, significantly enhancing the user experience.

This process is crucial for maintaining the high standards expected in financial transactions and data consistency.

How exactly do we migrate `Public / V2` APIs to `Platform` APIs?

Milestone 1: The Engineering Doc.

Ok, Now that we know the whys and whats of API migration, let’s start with how to migrate APIs.

Before heading directly to your code editor, we start with an Engineering Document (ED). Any API migration begins with writing an ED to determine the scope and timeline of the migration. We aim to migrate all the methods that call public or v2 Projects API to the platform. To do this, we first need to figure out which methods are calling public or v2 API and document it for future reference.

Here are two approaches to identifying these methods:

1. Global Search in Code Editor:

   • Go to your code editor (VS Code or another) and globally search for places where APIV2Service (for v2 API) and APIService (for public APIs) are called.

2. UI Inspection:

   • Identify UI actions that trigger public or v2 API calls. This involves opening the developer console and network tab, performing various actions, and checking which API calls are made (for projects in this case). This method helps ensure there are no hardcoded calls to APIV2Service or APIService.

As I was migrating Projects API, I identified the personas where Projects are used: SETTINGS, UPSERT, ADMIN, and SPENDER. Then I inspected each page’s network calls to verify the API usage.

For reference, here is the Engineering Document I wrote during the Project API migration.

Milestone 2: Migrate the Projects API to the platform.

With the Engineering Doc in place, I proceeded with the actual migration. Here are the steps I followed:

Step 1: Update API Calls:

1. Replace all instances of APIV2Service and APIService with the new PlatformAPI service in the codebase.

2. Ensure all methods that were previously called public or v2 APIs are now called Platform APIs.

Step 2: Transform the Response:

With the API migration, the key names of the responses have changed. For example, v2 API is used to give a response like:

{
  "project_name": "",
  "project_id": "",
  ...
}

But the new Platform API gives a response like:

{
  "name": "",
  "id": "",
  ...
}

This introduces an issue because the entire web app uses the old key names in the templates. Manually changing all the key names across the entire web app increases the chances of bugs. Therefore, we must transform the API responses from the new Platform API to match the old v2 or public API format.

Before transforming, it’s crucial to check and compare the new response with the old response. You can refer to the Projects API response diff document for detailed comparisons. Example transformation method:

self.transformToV2Response = (platformProjectArray, activeCategoryList) => {
  const allCategoryIDs = activeCategoryList?.map((category) => category.id);

  const transformedProjects = platformProjectArray.map(
    (platformProject) => {
      const updatedProject = {
        project_active: platformProject.is_enabled,
        project_id: platformProject.id,
        project_name: platformProject.display_name,
	      ...
      };
      PlatformV1DateService.fixDates(updatedProject);
      return updatedProject;
    },
  );

  return transformedProjects;
};

Step 3: Testing your changes:

• Testing is a crucial phase in the API migration process. Given the sensitivity and importance of financial data, we must ensure our application is error-free and performs optimally.

• For a comprehensive testing strategy, I document all the test cases, personas, and actions in a detailed manner. You can find this exhaustive list of test cases in this Web App Test Cases document.

• Testing Environments:

  • Local Environment: Initial testing starts in the local environment to catch and resolve any obvious issues.

  • Test Private Branches: I then push the changes to private branches for further testing.

  • Master Branch Testing: Once the changes pass the private branch tests, they are merged into the master branch for more rigorous testing.

  • Pre-release Testing on Release Branch: Before the final release, I did one more round of pre-release testing on the release branch.

  • Post-release Testing on Production: Finally, post-release testing ensures that everything works perfectly in the production environment.

• Why So Much Testing?

  • At Fyle, our users depend on our software service for accurate, consistent, and timely financial transaction management. Any bugs, performance issues, or data inconsistencies can disrupt their operations, leading to financial losses or user dissatisfaction. Therefore, we emphasize rigorous testing across multiple environments and org-based scenarios to maintain the trust our users place in us.

Anecdotes of Pitfalls and Lessons Learned:

Throughout the migration journey, I encountered several challenges that provided valuable lessons.

Issue 1: Changed API Behavior

When projects are bulk uploaded, the category IDs assigned to them are null. Previously, we used V2 and public API to send the correct mapped response from the backend. However, with the new Platform API, this behavior has changed without updating the API documentation. As a result, I had to undo and redo all the changes, which was time-consuming. Projects were widely used, requiring logic changes and handling category dependency. Additionally, implementing the new solution, unit tests, and increasing code coverage added to the time required.

Learning:

• Thorough Documentation of API Changes: Ensure all changes in API behavior are meticulously documented. This helps prevent unexpected issues during implementation and testing.

Issue 2: Edge Case P0 - Project Field Not Visible for Mileage and Per Diem Expenses

This issue was there only in a specific case where all projects in the organization were added in bulk (with no category IDs) and the organization did not have any subcategories for mileage or per diem.

Learning:

• Comprehensive Testing Scope: The scope of testing should be increased, including edge cases, checking the places where changes might affect, and testing all the configurations that might not be immediately apparent.

Issue 3: P0 - Category Field Not Visible

This problem occurred for organizations with disabled projects or no projects at all. The implemented changes related to the project-category dependency had bugs. In the web app, the category field is handled in app-v2. I had to adjust the logic in Quick Fyle and the category component to ensure we first fetched all categories before making the project's API call. The error stemmed from a logical bug where the categories API promise was mistakenly placed inside the ifProjectsIsEnabled condition.

Learnings:

• Integrated Impact Analysis: Before implementing changes, study the detailed impact analysis to understand how your modifications will affect different parts of the application. This includes identifying dependencies and potential breakpoints.

• Small PRs, Incremental Changes, and Merging: Instead of merging large sets of changes all at once, use an incremental approach. Merge and test changes one by one to isolate issues more effectively and reduce the risk of widespread problems.

By applying these lessons, we can improve our process and minimize the risk of encountering similar issues in future API migrations. The experience from these challenges has been invaluable in refining our approach and ensuring more robust and reliable implementations moving forward.

Conclusion

By following a structured approach, documenting each step, and leveraging the support of experienced team members, you can navigate the complexities of software development successfully.

For new interns and developers who will be working on similar projects, I hope this blog provides a clear roadmap and answers to your questions. For senior engineers, I hope this offers some useful insights and stories from the field.

Feel free to reach out if you have any questions or need further details. Happy coding!