Functional Areas

[handrews]

In the June 18th meeting (#134) I presented several functional areas as a way to organize the spec and align it with tooling and end-user use cases. The seven areas were sufficiently well-received to warrant further development, so I'm starting a discussion for them.

Functional Areas

Here is the list, slightly modified based on discussions in the call:

0. Resolving import/reference IRIs to URLs and retrieving the target documents

This is "0" because it's a prerequisite, but is in many ways the easiest to split off into a 3rd-party tool. This replaces the current notion of reference "resolvers" (which are often reference removers, which is not always possible and AFAICT usually implemented incorrectly for 3.1) by drawing the boundary in the right place: between the abstract identity (IRI) and a concrete and secure location.

Input:

• One or more IRIs (not relative IRI-references)
• A configuration mapping IRIs, IRI prefixes, or other forms of IRI patterns to corresponding URLs
• An allow/deny configuration regarding what file and network I/O is safe to perform

Extensibility:

• Retrievers can be added for each supported URL scheme (file: and https: being the most obvious / common)

file: support will likely be built-in for most implementations, unless they run in an environment without a filesystem.

Output:

• For a single IRI, it could just return the representation retrieved from the URL
• For a set of IRIs, it should populate a safe document cache, either in-memory or in a filesystem or network location that is believed to be secure and accessible (e.g. in case of network problems) for future access
• Alternatively, a single-document-stream serialization could replace the document cache, or be an alternate way to populate one

Getting this area and its boundaries correct is essential to avoid the pathological non-interoperable mess we currently have with referencing in 3.x. For organizations that have to create descriptions that work across multiple toolchains, this lack of interoperability is one of their most severe challenges.

1. Parsing, manipulating, and serializing OpenAPI Descriptions (OADs)

The critical aspect here is defining a language/environment-neutral representation of a parsed and resolved OAD. This should include both a round-trip safe form that preserves file and order (e.g. line number) mappings, and an efficient form that does not track such information because the OAD will either not be serialized after being parsed, or because there is no need for round-trip safety.

This representation is not any sort of attempt at a generic abstraction of HTTP APIs that would be independent of the OAS. It's just a way to allow testing parsers for compliance with the specification and to support interoperable ways of working with parsed OADs. This allows different tools to share the same parser, even one made by a different tooling vendor.

Input:

• An IRI-indexed document cache with a designated entry document (or a self-contained document stream)
• Optionally, a set of instructions (format TBD) to manipulate the parsed representation

Interfaces:

• A "cache miss" is handled by area 0 (IRI to URL resolver / retriever)
• In theory, one could make the cache write-through to edit in published locations, although that seems a bit scary
• The parsed/ref-resolved (not ref-removed) representation is the interface to all other areas

Output:

• Serialized back to the same set of documents, or to a "normalized" form of some sort

All use cases require at least the non-round-trip-safe parsing aspect of this area.

Round-trip safety is essential for use cases like building an editor / IDE for OADs. Depending on the exact use case, OAD pipelines may or may not need the round-trip-safe form, but definitely need the manipulation and serialization aspects.

2. Modeling data

All data is modeled in the same way, regardless of whether it is:

• Media type-based (HTTP message contents; URL query string contents, at least in some cased)
• Grammar-based (HTTP fields/headers/trailers, URL path and other components)
• Any other mechanism for defining syntax

This likely involves an extensible registry defining how to map non-JSON-compatible media types and grammars to Schema Objects or whatever other Objects are involved (e.g. Encoding Objects). We do not want to attempt to define anything like a generic mapping between ABNF and Schema Objects, just to make that perfectly clear.

This is where a lot of the most difficult design decisions need to be made. The Objects currently involved in this take a variety of different strategies, and it is not clear how to reduce the duplication (or, more confusingly, near-duplication) in the current arrangement without getting too generic to be useful in the concrete usages that we need. Those Objects include:

• Media Type Object
  • groups schema, encoding, and a bit of metadata
  • does not include the actual media type, which is a key in a higher-level object
  • no good way to model media type parameters
• Schema Object
  • currently JSON Schema, a constraint system that is often used, with painful contortions, as a data definition system
• Encoding Object
  • parallel to the Schema Object, but only supports one or two nesting layers (by using arrays and/or explode)
  • conflates mapping the data to media type structures with encoding/escaping data, largely because RFC6570 does this
  • notably, not every place where it uses RFC6570 mechanisms actually aligns with RFC6570's encoding/escaping rules, which are strictly RFC3896-based
  • does not actually even fully support multipart types, despite what the text in versions to date has said
  • in theory, the same Schema Object could be used with an Encoding Object for form data, and without it for JSON data that would otherwise just be the source for the form data
• Parameter Object
  • maps data into a variety of different URL and HTTP header locations
  • somewhat defines inputs to what could be an SDK function, although separate from any request body data
  • conflates mapping with encoding/escaping, largely because RFC6570 does this
  • notably, not every target location works with RFC6570 encoding/escaping, which is confusing and error-prone
• Header Object
  • limited form of Parameter Object inheriting all of its concerns
  • the RFC6570 behavior is almost never what you want, making many configurations de-facto invalid
• XML Object
  • maps schemas to alternate data model similar to parts of the Encoding Object, but..
  • embedded in schemas rather than parallel to them
  • in theory, allows dual use for JSON and XML with the same schema
• Discriminator Object
  • also embedded in schemas
  • assists in projecting the JSON data model into code structures such as inheritance

It's notable that three different ways of augmenting Schema Objects appear here: wrapping the schema from the outside (Media Type Object, Parameter Object), paralleling the schema structure (Encoding Object), and embedding information in subschemas (XML Object, Discriminator Object).

What is clear from the recent 3.x patch release work is that data model mapping and encoding / escaping need to be orthogonal, because (as much as I love RFC6570 on its own), tangling these up creates incredibly complex, subtle, and error-prone interactions.

What is not clear is how to reduce all of these different ways of augmenting schemas to something more consistent and coherent. Nor is it clear that we should continue to (ab)use a constraint validation and annotation system by treating it as a data definition system.

We also need to be sure we can express data semantics sufficiently to meet our declared ambitions around semantics.

3. API Shape: Modeling HTTP interactions

This is mostly where the concept of operation "signatures" lives, although it will depend on data modeling and possibly other thinsg as well.

We need to:

• model HTTP message structure and semantics
• correlate requests and possible responses
• augment HTTP-level descriptions with sub-operation descriptions (RPC support)
• model SDK-level abstractions over (sub-)operations
• support runtime context-sensitive validation (e.g. read/write/create-only, etc.)
• possibly define resource-level relationships (most obvious: collection/item correlation)

Output:

• Something that lets us connect (sub-)operations to security configuration, without having to embed deployment-specific security requiremetns within the shape

4. Securing an API

There are many people who can define this area better than I can.

Output:

• Security configuration that a deployment can associate with a shape

5. Deploying an API

• Servers, server variables, other runtime configuration, discoverability, etc.
• Correlating security configuraitons from the Securing area with the API Shape

6. Organizational and presentation metadata

• Categorizing and grouping various aspects of the spec
• Doing this for different audiences (human docs, SDK generation, AI, etc.)

Tooling and User Use Cases

The point of this is to align areas of the spec with how tools are constructed and used. The current misalignment of ref "resolvers"/removers shows the problems that are caused when we fail to do this up front. Tools that work with the OADs (e.g. editors), generate human-oriented documentation, generate code from data models, or manage the API at runtime use different sets of areas, and should be able to ignore areas that are irrelevant.

Discussion here should, among other things, focus on making those tool and user use cases, and their correlation (or not) with the areas, more clear.

[handrews]

In this morning's Moonwalk meeting, @darrelmiller asked if the functional areas would be sections in the table of contents, or if the relationship to the spec structure is less direct. I replied that in my ideal view, at least some of them would be entirely separate documents that could, if necessary, be released on their own schedule. There are a few ideas wrapped up in that answer:

• Some areas correspond directly to tools or libraries with well-defined interfaces to other areas. The most obvious being Area 0 (corresponding to today's reference resolvers, but with different boundaries) and Area 1 (which is a prerequisite for doing pretty much anything else)
• Some areas are much more stable than others.
  • For example, if we correctly apply the lessons learned from 3.x, Area 0 can probably be written once (OK maybe it will need one update as we are doing a lot of new things), after which it should be stable indefinitely.
  • On the other hand, security changes faster than HTTP, meaning it would be beneficial to be able to update it without needing a revision of everything. Some of that can be handled by making security extensible, and we already have a proposal for that. But it might be beneficial to have even more flexibility.
  • Data modeling is another area where we might want to be able to make more changes, as I feel we really do not have that solved, and probably won't nail it in Moonwalk the first time out either
• Some areas require different skillsets.
  • Security is an obvious one. We get security folks appearing sporadically, but most of our discussions are irrelevant to them. We have created a security SIG which went nowhere. It's not clear to me what it's supposed to do outside of the main spec- what we need is a well-defined area inside of the spec, which will motivate participation
  • In general, this would faciliate "subcommittees" to start breaking this enormous task into manageable pieces

This does not mean wa have to literally have separate specs. There are appealing things about that model, and it has worked very well for CSS. But most of the benefits can be achieved whether they are different specs or different sections or whatever. It's just that the lines are very clear if we have separate documents. It's much harder to give in to the temptation to take shortcuts.

If we were to have separate documents, it would not necessarily be one per area. I could see:

• Area 0 spec (IRI to URL resolver and retriever)
• Area 1 spec (internal representation, parsing, and serializing)
  • It's possible that Area 7 (organizational metadata) might fit here)

After that it gets less clear. It would be easy to make an Area 2 (Data Modeling) spec, and we would need to do that if we want to split up the rest as they all depend to some degree on data modeling. We've already talked splitting shape and deployment, including possibly as separate specs (one of the inspirations for this whole idea - see #66 ). And as noted above, I think security has a strong case for being separate. But with some of these the interfaces are less clear, as are the correlations to tools.

Areas 0 and 1 have obvious target tools or libraries analogous to things that exist today. Before we decide to break up the spec beyond that (if we even do that much), we should be clear on the benefit to tool developers and users.

[miqui]

Looks @handrews . I did a small edit to fix a typo.

[miqui]

Ok, I am going to take a stab at a tough one: securing an API. I'll open a new discussion to capture some of the work, ideas, etc. I'll stick to using yaml.

[rafalkrupinski]

Is there an issue/discussion on replacing JSON Schema?

[handrews]

@rafalkrupinski The closest would be:

• Implementors Feedback on current Alternative Schemas Draft Proposal #122 (originally "alternative schemas" to work around the "superset-subset" problem of pre-OAS 3.1 schema support, but the alternative schemas concept was just as much intended to be for alternatives to JSON Schema as it was alternative versions of JSON Schema
• Support SHACL schema as alternative to json-schema #120 (a specific possible alternative)
• Learning from "competing" formats #47 (includes suggesting CDDL as an alternative)
• Ability to import datatype declarations from XSD files #124 (an idea involving XSD)

The main problem is that we probably need a new approach as none of the available options quite manage it:

• CDDL is promising as a data definition system, but is not sufficiently modular for large-scale use (although CDDL2 might be). We would have to augment it somehow to get anything resembling parity
• I work wiith large organizations using semantic web technologies and even they don't want to only use SHACL - I think it would be a good option but would probably remain niche
• XSD is obviously specific to XML, so it would solve the awkwardness of using XML Objects plus JSON Schema to describe XML, but not much else
• Other options are mostly part of proprietary alternatives to OAS/JSON Schema (that nevertheless "compile" to OAS/JSON Schema)

While I'd personally be happy to work on building an alternative that is designed from the ground up to suitable for OAS's needs, it would be a lot of work and I've been unable to find a source of funding – it's far too large to do purely as a volunteer effort (at least if I'm the volunteer) and the OpenAPI Initiative just doesn't have a large enough budget for it AFAICT. We'd need a corporate sponsor or two, I'm guessing. Or we'd need to make a very strong budgetary case to the Initiative somehow that it's more critical than other priorities. Other folks are also working on how to get more companies to join as dues-paying members of the Initiative, which would expand the budget.

I feel a bit like I'm breaking a taboo here by acknowledging that there are limits to the all-volunteer model, but... OpenAPI is a huge project. And it seems like fewer companies are happy to pay employees to put substantial work hours into contributing to standards. What we see instead are companies getting funding to create products working around problems with the standard. So you have large companies or VC-funded startups making proprietary "front ends" to OAS/JSON Schema instead of funding work to make the OAS more suitable for everyone. It's a little frustrating.

[rafalkrupinski]

Regarding data description languages, I like to joke that I'd rather parse TypeScript types than translate JSON Schema constraints to type declarations. Perhaps there's something to it, it was designed to describe JS objects.

Now you've mentioned budget, open source and sponsors, is there a hired person at OAI responsible for getting sponsors? Such person could talk to corpos and bring these exact arguments.

[handrews]

@rafalkrupinski there's a discussion on "member value proposition" which is about getting more OAI members which would add to the budget. I don't know that there's anything on recruiting direct sponsorships.

• Structuring the thoughts around membership value proposition Outreach#47

[handrews]

@rafalkrupinski thinking on this a bit, there are probably two immediate issues:

• the OAI went very quiet after 3.1, and is only now just "waking up" and starting to publish things. I think a lot of people, including tooling vendors, looked at the 700+ issue backlog stretching back to the mid-2010s and just kinda gave up on us doing things. Publishing Arazzo (already done), 3.0.4 and 3.1.1 (both immanent), and hopefully soon 3.2 will be a clear signal that we're really back in business
• I don't think the companies that could most benefit necessarily understand what the limiting factors are and how additional sponsorship might help. We don't want to give members/sponsors too much control on the spec direction (in fact, our charter forbids it). But we could post a road map and say "if you want to help get one of these things done, we have a person in mind that we could hire to do it given some cash." Maybe? idk, I'm brainstorming, this may or may not make real sense.

BTW I really want to emphasize that the whole OpenAPI Initiative is very sincerely trying to find the best paths forward with both volunteers and, where possible, contractors. And making that division of labor fair. It's a hard problem, and my observation that there's not enough funding should NOT be taken as a criticism of anyone's efforts. There is a lot going on right now, both in terms of the work we're doing now and in terms of how we're trying to make more progress.

The Initiative is also being great about recognizing volunteer contributions through things like blog posts, which is a great way to keep volunteers happy and engaged. We don't want the Initiative to just become a company. But at the same time, if we have volunteer community direction, handing off specific things to contractors to just get it done has worked well where we've done it so far (3.0.4 and 3.1.1 would not be anywhere near as substantial otherwise- the OASComply compliance project got repurposed when it became obvious that the spec needed to be more clear before we could write a compliance tool).

[rafalkrupinski]

I think we all know volunteer resources are usually limited and hiring people is often an effective way to push things forward. Even python hires people!
And BTW, great job culling all those issues.

"if you want to help get one of these things done, we have a person in mind that we could hire to do it given some cash."

I see here a risk of direct influence by sponsors over the way features are realised.

Perhaps a bounty system, where OAI publishes an estimated cost and a currently pledged amount for each task, sponsors pledge full or partial amounts, and when the cost is met, a designated person is contracted by OAI.