Enhancing the MongoDB Atlas Go SDK with Automated Code Generation

The MongoDB Atlas API Experience team is committed to offering a seamless developer experience to customers who build and automate against the MongoDB Atlas developer data platform. We offer various programmatic tools, such as the Atlas CLI and the Terraform Atlas Provider, and maintain the Atlas Admin API with key features, including versioning and Open API specification.

Our latest offering, the MongoDB Atlas Go SDK, empowers developers to manage Atlas using Go. It eliminates the need for low-level Admin API calls by utilizing higher-level abstractions. New SDK versions are automatically generated on each Atlas update, ensuring access to the latest Atlas administrative capabilities.

In this post, we’ll discuss the advantages of using an SDK over direct API calls, explore our decision for code generation over manual development, share insights from engineering challenges, review our adoption experience, and discuss some next steps for the Atlas SDKs.

Benefits of using an SDK over direct API calls

Direct API calls offer flexibility and fine-grained control but become cumbersome for complex interactions. Developers manually handle tasks like authentication, error handling, and response parsing, which can be time-consuming and error-prone. Additionally, keeping up with API updates can require regular low-level code updates.

The Atlas Go SDK simplifies development with higher-level abstractions. Pre-built functions and structs encapsulate API interactions, reducing boilerplate code. This frees developers to focus on core application logic rather than API integration intricacies.

The SDK automates authentication elements, error handling, and response parsing, reducing boilerplate code and errors. Plus, staying up-to-date becomes low-effort, as new SDK versions are auto-generated with each Atlas release. While leveraging new functionalities might require some code updates, the SDK significantly simplifies integration compared to direct API calls.

Moving from a manual client to an auto-generated SDK

Previously, MongoDB’s bespoke Go Client for MongoDB Atlas powered internal tools and customer applications. However, maintaining a manually written client increased toil and impacted the team’s productivity. Keeping code quality high required constant effort, leading to a backlog of unsupported Atlas features and a struggle to keep pace with new functionalities.

As Go-based Atlas applications proliferated, the limitations of manual maintenance became clear. We needed a more scalable solution.

We had previously built a mechanism to automatically generate the Open API specification for the Atlas Admin API. This machine-readable document details how to interact with the Atlas functionality through its REST API, accessible through the “Download” option in the MongoDB Atlas Administration API portal. This led us to explore leveraging that specification for client code generation as well.

Building on this foundation, the Atlas Go SDK leverages the Open API spec and a robust delivery pipeline to streamline client generation. The pipeline scans for changes in the spec and triggers the generation of a new SDK version upon detection, producing a pull request as an intermediate output. Following our review and merge, the pipeline requires no other manual steps to continue working on the release process, ultimately publishing a new SDK version.

Code generation has improved output consistency and speed over manual development, freeing up resources for other impactful projects. To showcase the time savings, adding support for a new resource in the old manual Go Client could take roughly two engineer days. The new SDK reduces that time to a less-than-an-hour code review, as all other steps are automated.

Major challenges and solutions

Transitioning to an auto-generated SDK wasn’t without its hurdles. Selecting the right tooling was crucial, and we explored both commercial and open-source solutions for Open API-based code generation. We opted for openapi-generator due to its open-source nature. The choice prioritized flexibility and control, ensuring long-term project autonomy. It also allowed us to open-source the Go SDK generation codebase, fostering community contributions and improvements.

Leveraging the Atlas Open API specification, originally intended for documentation, presented unique challenges. While this approach offered a single source of truth, we encountered discrepancies between the spec and actual API behavior. Notably, minor, non-breaking API changes sometimes resulted in significant, breaking changes in the generated client. For instance, marking a field as optional in the API (non-breaking) turns the equivalent Go model property into a pointer (breaking).

This discovery necessitated a two-fold solution. We addressed the root cause by improving the Open API specification, ensuring better alignment with code generation requirements. Then we developed an automated transformation process to optimize the spec for generating code. This transformed version remains optimized for code generation while the original spec continues to serve its original purpose of live API documentation.

Having a pull request review as part of the release process has helped us ensure quality. In some early cases, it allowed us to catch issues, apply SDK-wide improvements, and add linting rules to prevent reoccurrence.

Maintaining compatibility with the existing Go client was crucial for an easy migration experience. However, achieving perfect compatibility wasn’t always feasible due to inconsistencies in the handwritten client. We meticulously evaluated each change, balancing compatibility with the benefits of code generation. We also created a migration guide and best practices to aid with migrating existing applications.

Finally, to minimize confusion and client bloat, we decided that each Go SDK release should target a single Admin API version. That simplified integration but created a versioning challenge: Go’s semantic versioning uses major bumps for breaking changes. We needed to communicate the targeted Atlas API version within the SDK version while simultaneously denoting breaking SDK changes unrelated to API updates. Our solution was to develop a unique major versioning scheme for the Go SDK. It incorporates the Admin API’s date as a prefix, with additional digits signaling breaking SDK changes for that specific API version. While unconventional, it adheres to semantic versioning and keeps developers informed.

Adoption experience

Our Atlas CLI tool was the proving ground for the new Go SDK. In 2023, it became the first application to migrate from the manual Go client by leveraging pre-release versions of the Go SDK. That staged adoption approach yielded significant benefits. By migrating early, we uncovered valuable improvement areas for the SDK, directly influencing its architecture. This early feedback loop also provided crucial insights into effective SDK generation.

The migration wasn’t without its complexities. It addressed cross-cutting issues across three distinct layers: the Open API spec, the SDK generation, and the CLI integration, each presenting unique development challenges. However, this comprehensive approach ensured that all layers benefitted from the process, and we raised quality across the board.

Following the successful CLI integration, we confidently expanded the SDK’s reach across our product portfolio. Now, it serves as a prominent and trusted middleware solution, not only for the Atlas CLI, but also for several Atlas DevOps tools. Recent additions include the Atlas integrations for AWS CloudFormation and CDK. The Terraform Atlas Provider and the Atlas Kubernetes Operator are also on the immediate roadmap, further solidifying the SDK as a core component within our DevOps ecosystem.

External adoption of the Atlas Go SDK is also gaining momentum. Over 30% of customer-developed Go-based interactions with the Atlas Admin API now leverage the new SDK, a trend we see increasing monthly.

Looking ahead

As Atlas continues to scale, the pace of features will only continue to accelerate. Our primary focus for the Go SDK is to continuously expose new Atlas features as they are released in the Admin API while introducing quality-of-life improvements for developers at every opportunity. We strive to reduce boilerplate code, improve Error Handling, simplify Authentication / Authorization, and to enrich documentation with high-quality examples.

The success of code generation has us exploring what else can be automated. Our team is looking at which other offerings could benefit from automation to free up development time for impactful projects that might be less readily automatable.

Finally, we understand that our developer community thrives on various languages and technologies. Not using Go? Let us know what other languages you’d like to see supported (e.g. Python, Java, TypeScript) by sharing your feedback.

Conclusion

The Atlas Go SDK empowers Go developers to streamline interactions with the Atlas cloud platform. By leveraging code generation and a focus on developer experience, the SDK offers several advantages over manual API calls.

• Reduced complexity: pre-built functions and higher-level abstractions simplify development by tackling authentication, error handling, and response parsing.

• Improved maintainability: auto-generation of updated SDK versions with each Atlas release ensures access to the latest functionalities, minimizing manual code changes.

• Enhanced reliability: Tailored API models promote code reliability and catch potential errors at compile time.

Our in-house adoption experience was a valuable proving ground, influencing the SDK’s development and uncovering key optimization points. Today, the SDK is a cornerstone within our DevOps ecosystem, accelerating the development of downstream DevOps tools.

This successful transition proved the value of code generation for developer productivity and code quality. It also opened up the possibility of generating SDKs for more programming languages in the future.