API Quality is more than meets the eye

As Saxo is working to create the vision for our next version of OpenAPI, we are looking for many sources of inspiration as to what needs to get improved. One initiative is a collaboration with the Department of Digitalization at Copenhagen Business School with the purpose to assess API quality and how different ways of working, different design methodologies, and different forms of governance may affect delivery speed and API quality.While the project is just getting started, it has already provided me with a first "aha" moment, or insight as you may say.

One of the first activities was to gauge the current quality of our OpenAPI. Instead of just relying on our own gut feeling about what may constitute a “good” API, the CBS team wanted to derive a more objective measure. Being quite thorough and academic, they scoured a long list of 59 resources, including developer blogs, academic articles, and vendor material to identify attributes which were mentioned as important to API quality. They subsequently grouped these into categories, themes, and indicators, and presented these to a number of OpenAPI front-runner companies, such as Saxo for feedback and refinement.

Their findings will later be published and detailed in a CBS working paper, but I have been allowed to provide a small overview/summary.

OpenAPI Quality Indicators

The table below summarises CBS’ early findings. All stated measures are divided into two broad categories (Product attributes and Process Attributes). Each category comprises several themes (Documentation, Security etc.), each of which then includes a number of quality indicators, the number of which are included is indicated in brackets.

So how good is your API really?

The interesting part – for me at least – was that on a day-to-day basis my team of OpenAPI Platform managers primarily worry about a small subset of the comprehensive list, and mostly about the attributes on the left in the table.

We constantly ask ourselves:

• Does our API follow the agreed design principles, is it “fit for purpose”, is there consistency within and across domains?
• Is the API stable, performant etc?
• Do we have sufficient documentation, sample code, and support in our demo environments?

But what about all the other stuff? Why are we not thinking about that, and why would we not have come up with these measures, if you had asked any of us how we would assess API quality?

Perhaps it has to do with the "Four Levels of Competence" model, first developed by Noel Buch in the 1970s, which states that in almost all learning we go through four levels, as shown in the table below:

Note that according to this, we will have a blind spot to what we are really poor at (unconscious incompetence) but also what we are really good at (unconscious competence).

This reminds me of a job interview I had a long time ago. The interview went quite well until we got to the stereotypical question of “so what are you good at”? I remember giving a passionate description of a problem I had recently solved (to my own great satisfaction), and I was pretty happy with that answer until a colleague praised me for something, which I actually thought was super easy and did not think much of. I then realised that my answer at the job interview probably was not my real strength – it was just what I was conscious about at the time, and in this case consciously competent at.

Time for an API 360?

The comprehensive and somewhat academic and theoretical work eventually published by CBS is an important reminder for both developers and consumers of OpenAPIs.

It is very natural for a team like mine to sometimes have a too narrow focus. It is easy for any engineering team to be too much “inside out” rather than “outside in” and forget what matters most for our clients. As discussed above, we are by default also limited to focus on what we are conscious about.

For us, a checklist like the above (even in its skeleton form) is a good reminder to consider all of the indicators of good API design and good API development practices.

The same goes for potential OpenAPI consumers, such as our partners. It is quite tempting to only assess the product you can see directly on our OpenAPI developer portal. “Do they have an API for this, does it appear to be well documented, do they have available samples etc?” But they are probably well advised to also consider some of the more invisible qualities: “Have they got adequate security measures in place”, “Do they have proper software development and testing practices”, “Is the underlying system scalable” etc.

In summary, developing and managing a comprehensive OpenAPI supporting hundreds of partners and millions of trades from over 180 countries is a multifaceted challenge. To be successful, both as providers and consumers  one must consider many factors and remember that there is more than meets the eye when it comes to a true assessment of OpenAPI quality.