Developing RESTful APIs becomes tedious and time consuming when even for the slightest changes applied you need to modify not only the backend code responsible for processing API calls, but also test scripts and test data used to verify that the API works as needed.

As part of my software engineering work with RESTful APIs, I kept seeing the same problem: avoiding drift – i.e. keeping RESTful API and RESTful API tests in sync – each time requires additional manual and time consuming work, which is mostly repetitive.

I thought, since RESTful APIs are so widespread, being utilized for many years already, there must be tools which automate testing those. Was I mistaken! Or, perhaps my definition of automated solution goes beyond what these tools I found can accomplish. However, my requirements were quite modest, i.e.:

  • ability to generate REST API test scripts based on RESTful API, so these are kept in sync, solving the drift problem, and if there needs to be some intermediate step for the REST API test scripts to be generated – e.g. describing RESTful API via OpenAPI (Swagger) specification, then generating REST API test scripts off that spec – then such solution must be compatible with OAS 3 (the latest OpenAPI spec)
  • support for OAuth2.0 authentication – so authenticated REST API calls can be made
  • support for test data
  • support for variety of test types – smoke tests, load tests, scenario tests
  • support for gathering metrics – such as response time, number of successful requests, as well as other SLA/SLO metrics
  • preferably open source

Out of the tools evaluated (“OpenAPI Test Templates” / “Swagger Test Templates”, “Dredd”, “SoapUI”, etc.), “K6” (https://k6.io/) came closest to meeting the above mentioned requirements. The commercial version of ”SoapUI” also came quite close, but cost of license was prohibitive (~ 8000 EUR).

When I started putting together what I envisioned, utilizing what K6 already made available, I encountered an additional set of problems:

  • Generated tests being only a skeleton, which does not solve the drift problem, also makes it impossible or very difficult to run REST API tests as part of CI/CD pipeline
  • No ordering guarantee – requests must be ordered for anything but most trivial tests
  • No support for example data – testing with randomly generated data is very often not an option, specific inputs are needed to get predictable results
  • Requires additional tech stack knowledge, which for K6 meant knowledge of JavaScript – introducing bugs is one of the most serious problems lack of tech stack knowledge causes, when attempting to keep both the REST API and the REST API tests in sync

Thankfully, with K6’s REST API test script generation solution being based on the open source OpenAPI Generator (https://openapi-generator.tech/) project, it was not a problem to go ahead and enhance it up to the level I needed:

  • Taking advantage of OpenAPI’s support for adding examples (see https://swagger.io/docs/specification/adding-examples/) to enable parameter and request body example data extraction – additional logic was added which extracts these examples and feeds as test data for the generated test script;

  • Introducing custom OpenAPI extensions (see https://swagger.io/docs/specification/openapi-extensions/) to enable:

    • Request grouping and ordering (via x-k6-openapi-operation-grouping extension) – which allows to group operations and define their ordering (i.e. order in which operations must be executed, so entity is created first, only then queried for, then deleted, etc.), facilitating automatic generation of more complex test scripts, e.g. scenario tests;

    • Response visibility (via x-k6-openapi-operation-response extension) – i.e. whether given API response should be “hidden” (in other words ignored when generating test script), so that in case of multiple code 2xx responses (e.g. 200 and 204), generated script checks only against one, e.g. code 200 response;

    • Response data extraction for chaining requests (via x-k6-openapi-operation-dataextract extension) – which allows to specify path to value in body of response which should be extracted and assigned to variable for later use by other operations, which further enhances automatic generation of more complex test scripts like scenario tests, as well as load tests, since in load tests, which normally execute calls simultaneously (as K6 calls these virtual users, “VU”), same payload cannot be used simultaneously if it contains static ID (i.e. ID as part of example data) – ID generation must be handled by REST API backend code (e.g. create entity), then that generated ID referenced in subsequent API calls (e.g. retrieve, update, delete entity).

With OpenAPI (Swagger) specification utilized as the single source of truth for automated REST API test generation, including extracting of test data from examples embedded in the spec itself, the enhanced K6 OpenAPI Generator allows to generate REST API test scripts which are ready to run, out-of-the-box. This enables full round-trip development flow, ensures REST API tests are always up to date, and facilitates running REST API tests continuously as part of CI/CD pipeline, as an additional, post-automated-deployment step.

The ready-to-run generated scripts can be simple smoke tests, as well as more complex scenario and load tests, depending on your use case.

In addition, since tests are generated automatically, no additional tech stack knowledge is required – developer working on REST API can focus only on REST API code, and simply add relevant OpenAPI annotations and example data to enable generation of smoke tests, scenario tests and load tests.

For automated REST API testing as part of CI/CD pipeline, continuous deployment must be in place – i.e. latest version of application exposing such REST API must be first deployed, then automated REST API testing added as an additional, post-continuous deployment stage. In such pipeline, generating the OpenAPI (Swagger) spec, then the REST API tests off that spec, are simply run in separate, earlier, steps of the pipeline.

As for generating the OpenAPI (Swagger) specification itself, it can be generated dynamically, both at runtime and at build time – generating it can be easily integrated, e.g. via existing Gradle/Maven plugins (see https://swagger.io/tools/open-source/open-source-integrations/).

All of the above mentioned enhancements were contributed to the OpenAPI Generator project and starting with release v5.3.1 (https://github.com/OpenAPITools/openapi-generator/releases/tag/v5.3.1) are now part of the OpenAPI Generator project.

PLEASE NOTE: Unfortunately, one of the OpenAPI Generator contributors (https://github.com/agilob), while fighting extra whitespace (details in https://github.com/OpenAPITools/openapi-generator/pull/10614) introduced a bug, which was pointed out (https://github.com/OpenAPITools/openapi-generator/pull/10614#discussion_r747119103), however apparently not fixed. This problem was reported again and until it’s fixed, please use the mirror repository, which contains working code https://gitlab.com/oss-contrib/EclipseCon2021/openapi-generator/ (branch: k6-openapi-extract-examples-enhancements).

Example project demonstrating these is available at: https://gitlab.com/oss-contrib/EclipseCon2021/example-project, created as part of my recent EclipseCon 2021 talk “Automated testing of OpenAPI-described RESTful microservices utilizing open source tools” https://www.eclipsecon.org/2021/sessions/automated-testing-openapi-described-restful-microservices-utilizing-open-source-tools, which can be viewed via EclipseCon2021 YouTube channel https://www.youtube.com/playlist?list=PLy7t4z5SYNaRXTpMGCTjbUtSwBnTtFa0U.

As with any open source solution, you are free to enhance it further. Possible next steps could include e.g. support for PUT and PATCH methods.

More information

OpenAPI Generator

Swagger.io

K6