Programmatically validating an informal API

May 9, 2018 3:32 PM

I need to create an automated solution for ensuring several client and server software modules of various versions can successfully interoperate. They don't have a formally specified API, so what are the best approaches?

Assume the existence of a client C with versions C₁…Cₓ and a server S with versions S₁…Sₓ. We'd like to validate which version combinations of client and server can successfully communicate. If we were starting from scratch, we'd likely use something like RAML or gRPC to generate the client/server interface stubs and enforce the API in code, but that's not an option here as previous versions' code can't be modified. Also the actual API is fairly ad hoc and can't be easily described by these structured specifications.

The best approach I can think of so far is to create a simulator that can act as either client or server, but shares the same code to ensure that it can't get out of sync and then write some automation to exercise the various versions of the real client and server code against it.

Does this seem like a reasonable approach? Can you point me towards any examples of applying something like this?

What other approaches might work?

posted by Cogito to Technology (3 answers total)

I think that step 0 is going to be creating a formal specification for each version of your API, no matter what approach you take. You'll need to do that on some level to do the API mocking that your simulator idea requires, and why not make that specification in a machine-readable format (e.g. JSON or XML or something) that you can then build your own tools around? Since you're defining the schema, you don't have to worry about whether the schema supports a given concept or structure or not. Once your specification format defines the various protocols, verbs and data types that your API exposes and uses, then I feel like the tools you'd need to write to verify them would become more self-evident. You could probably even do some basic scenario validation that doesn't involve running automation/simulators against your clients or servers, if you're able to define the protocols and data types in the various versions of your API strictly enough.
posted by Aleyn at 4:20 PM on May 9, 2018

Perhaps you have your use cases already well in hand, but since you describe the APIs as fairly ad hoc, seems like something could easily be missed. I'd want to be sure I have the functionality we care about, and I'd gain confidence in that by writing the important use cases and automating those.
posted by cyclicker at 9:41 PM on May 9, 2018

This is very dependent on what data structures they're using, but OpenAPI (formerly Swagger) works with JSON and is a lot looser than some other definitions if you want to go the semi-formal route.

Honestly, I'd probably just set up test requests in a tool like SoapUI/Postman or the command-line version of the same as individual projects, and then run each project against each "version" of the API and tweak accordingly. Then you'd have an ad hoc regression test suite.
posted by mikeh at 7:17 AM on May 10, 2018