Recently, while looking for ways to improve Schemathesis' test suite I found an old issue titled "Test with generated schemas". The idea is to generate random Open API specs and verify that Schemathesis does not fail with internal errors, to ensure it will work with a wider range of real-life API schemas.
Initially, I considered using hypothesis-jsonschema with Open API's official meta schemas. However, this approach was hindered by a limitation in the current hypothesis-jsonschema implementation: its lack of support for recursive references, which are common in Open API specs.
This article explores an alternative approach to generating valid Open API specs by leveraging Hypothesis' from_type and abusing Python's "typing" module. In the following sections, we will go through the implementation details, challenges faced, and potential future improvements.
If you develop a tool that works with Open API specs, you may find useful ideas for enhancing your tool's test suite and improving its resilience.
Schemathesis is a powerful tool for testing web applications built with Open API and GraphQL specifications. The prototype described in the article immediately found a bug in it.