Engineering

OpenAPI Tutorial: How to Automatically Generate Tests for OpenAPI Generator SDKs

Lee Wang

January 19, 2022

Intro

At Merge, our customers build their products in a variety of different programming languages. We take our role as a Unified API provider seriously and meet their needs by offering SDKs in each of those languages. We found early on we needed to build custom templates in order to ensure we support and test every SDK in a scalable way. 

If you’re interested in building your own testing framework for your SDKs - continue reading! We’ll use Merge’s use-case and API as examples, but our process and methodology can apply to any schema looking to utilize OpenAPI.

What is OpenAPI?

The OpenAPI initiative offers an industry-standard interface definition language, or IDL. That’s a fancy way of saying it describes an API, like Merge’s API, in a patterned, uniform way. You can see our API specifications here:

Why Do We Need OpenAPI?

To lead by example, here is some sample json that describes our ATS’ get-candidate-by-id endpoint, generated through OpenAPI. It lists url, input properties and response type:

The whole point of this json is so we can then use a well-formatted interface definition to generate SDK/client code for us. In order for the SDKs and Clients we provide to properly call our API we need our packages to be:

  • In a programming language our customers already use
  • Well-tested and safe

SDKs provided through templating and code generation are a more scalable approach, given the diverse array of API clients our customers need to call Merge.

Custom Templating With OpenAPI

While the OpenAPI specification is very well-defined, there are a few options for generating SDK code with a given specification json:

For the purposes of this guide, we’ll specifically focus on the OpenAPI Generator tool.

The Basics of OpenAPI

The OpenAPI Generator project comes with a lot of default templates for generating SDKs from API specifications. Here are some examples, and note how they’re written in  the syntax:

When generating an SDK, the tool uses these Mustache template files to define what the final code output looks like. The README of the OpenAPI Generator github repository has good instructions on how to run the tool, and for what it’s worth Merge uses the docker method.

Here’s an example command for generating our SDK:

Customizing Your Own Templates Files

Knowing where the templates come from is the biggest hurdle when working with OpenAPI. However, once you’re armed with this information, you can override these template files for your own custom configuration simply by creating a new file locally that has the same name! Since we are focusing on unit tests in this article, let’s first consider this Python template, which uses the .mustache syntax:

Next we’ll:

  1. Learn the basics of .mustache syntax
  2. Configure the override of this file
  3. Talk about limitations and apply a slightly different approach to Golang

You can download that file to your local directory, and use it in the docker SDK generation command from the previous section by adding this flag:

-t ./local/REPLACE-WITH-YOUR-TEMPLATES-FOLDER$

Mustache Basics

Mustache started out as an HTML templating language. Personally, I think this is a rather awkward choice for code templating since it lacks things like if x == y { output this } (though we’ll see how to get around that later). Because HTML is full of tags organized in a tree-like manner, Mustache does make it easy to traverse the OpenAPI specification json (I guess that’s why they chose it). 

Let’s take a super simplified example of a json specification:

And a file called test.mustache, like so:

This is very reminiscent of HTML in that it has both a start and an end “tag”. In this case, it uses {{}} instead of <> in HTML, and the start tag has a #, but it is structurally the same. The result of the above would be a file called test that contains:

item1

item2

While there is no if statement, you can test for the absence of a tag by doing the following:

Which would print the “There was no input” comment because the sample json above had "existingName" not the notExistName tag here. Lastly, you may occasionally encounter a bug where you have a Mustache like:

And you see output like &quot;item1&quot. Remember how HTML was the original target language of Mustache? Well, Mustache will HTML-escape your values by default unless you use three braces like {{{.}}} instead of {{.}} when iterating through values.

Test Strategy

Before we get to templating out a test, let’s first decide what we are going to test. The SDK is going to contain:

  1. Client code which wraps http connections and auth configuration
  2. API methods that correspond to endpoints we have in our OpenAPI specification
  3. Model classes that correspond to request and response payloads

We’ll start with the last point, since the other two are better suited to integration tests rather than unit tests. We’ll work through a good basic test: deserializing a model class from an example json payload. First, we’ll need an example json payload. Remember that the input to the *.mustache template files is the OpenAPI json specification of your API. We, therefore, need to add our example payloads in there. At Merge, we use OpenAPI custom vendor extensions for this, which is just a fancy way of saying we add a property that begins with x-. You can see this in practice by going to one of our specifications and looking for x-merge-sample-json.

Modifying an Existing Python Template

Now let’s modify that model_test.mustache template file to output our deserialization test. If at any point your generated output does not match expectations, use this Mustache line to see what raw data you are dealing with:

{{{.}}}

There’s not really an easy way to set breakpoints and debug, so the only option is to guess and check based on what the input is and where you are in the Mustache template. For example, modifying the file like so:

def test{{classname}}(self):
        """Test {{classname}}"""
        # FIXME: construct object with mandatory attributes with example values
        # model = {{classname}}()  # noqa: E501
        # Add this
{{{.}}}

Will show you the json payload of the model from your API specification as it is seen by the OpenAPI Generator tool.

There are a great many properties which we don’t care about, but with some work, we find the sample json custom property we added to the API spec earlier can be output by modifying the Mustache file:

Unfortunately, this will produce some weird stuff if we don’t plan on adding a test to every model class. This is where we run into difficulties with a lack of if statements, so instead we have to think like a browser and do something more verbose:

Remember how the Mustache tag like {{^name}} gets applied when name is missing from your input? This is how we simulate an if statement, and you’ll see this pattern all over the default template files in the OpenAPI Generator repository.

From here, we can write standard json into our template file:

And there you have it, a generated file like so:

Generating Your Own Output Files

OpenAPI Generator also provides the ability to generate files not in the default templates folder.

Specifically, the go template directory has no model_test.mustache file at all so we can use a config file like this:

to create a file as opposed to overriding one. The example above, along with Merge’s go_test.mustache file, is what generated the Go test class linked above.

Conclusion

Perhaps you find yourself in a similar situation to Merge, where you need to generate SDKs for a variety of languages and you want the model classes to have test coverage without writing each one by hand. With the approach outlined here, we can now test our SDKs for every language to provide a baseline guarantee that the model changes we make to our API specification will result in functional request and response classes. Additionally, we’ve added these tests to our continuous integration and delivery processes so that any time we change our API we re-validate against our SDK client code. This is all to ensure that the SDKs our customers rely on are tested thoroughly, as our customers are relying on these classes to communicate with Merge API in order to support the features they are shipping that need HR, Payroll, Accounting, and Recruiting data.

Curious about learning about what Merge does, or interested in joining? Check out our careers page, or poke around in our docs for more info!

Email Updates

Subscribe to the Merge Blog

Get stories from Merge straight to your inbox

Subscribe to Blog