Complete coverage should not be a requirement for core inclusion of WordPress REST API endpoints

json-bam

The WordPress REST API is at a bit of a crossroads. There is a proposal on the table by the core team of four contributors — Ryan McCue, Rachel Baker, Daniel Bachhuber, and Joe Hoyle — to ship endpoints to WordPress core iteratively. There is a pushback on this proposal by WordPress Project Lead and co-founder Matt Mullenweg.

The WordPress REST API is at a bit of a crossroads. There is a proposal on the table by the core team of four contributors — Ryan McCue, Rachel Baker, Daniel Bachhuber, and Joe Hoyle — to ship endpoints to WordPress core iteratively. There is a pushback on this proposal by WordPress Project Lead and co-founder Matt Mullenweg.

The API should be iterative, and should not require complete admin parity to be included, as Matt has proposed.

The state of what’s already in core, and what’s pending

The REST API infrastructure is already in core, having been included in WordPress 4.4. The infrastructure provides just that, an underlying infrastructure, to offer plugins a suite of base functionality to utilize, making the development of their own RESTful features easier.

The infrastructure allows developers to get a huge head start for creating their own REST API endpoints and functionality. The endpoints are how a third party client or developer would actually interact with the API, and the endpoints for WordPress core functionality are what are in debate.

Every piece of WordPress functionality — data that can be read, created, updated, or deleted — would ideally be available to the WordPress REST API, so that anything possible in WordPress is also possible via the API, and therefore also possible from a third party application interacting with the API.

However, creating this complete coverage is unrealistic in the current WordPress development structure, and should not be necessary for shipping initial endpoints to WordPress Core.

The proposal from the core REST API team is to begin by shipping four core WordPress objects: posts (meaning the posts table, which includes all post types), terms, comments, and users. These four core objects have been developed as endpoints, are well tested, and are ready for core. There are kinks to work out, but overall, they are an excellent starting point.

Matt’s proposal is that the API should not ship in core, with any endpoints, until complete “wp-admin coverage” is available: meaning that until all actions a user can make in the WordPress admin can be achieved via the API, then it shouldn’t go into core. He said in a meeting about the API, to firm up his proposal, that there should be, “no partial endpoints in core. let’s design a complete API, not half-do it and foist it on millions of sites.”

The primary areas not covered by the API currently are as follows: widget management and display, menu management and display, theme and plugin management, multisite management, and site options management.

So Matt would like to see each of these endpoints be developed and battle tested before any endpoints ship in core. I agree with the REST API team that the first four objects should take priority, and should go in before these other objects are ready. The remaining objects should each be feature plugins of their own, and go in over time, as they are ready, without delaying the four core objects currently under review.

Many use cases covered

Posts, terms, comments, and users endpoints make up practically all WordPress content. This means that a huge swath of use cases for the API are covered with these four objects.

Matt’s thinking that a core WordPress REST API should have all or nothing coverage is a false dichotomy. Applications like Calypso — a WordPress admin replacement based on WordPress.com’s own and separate REST API — are, I believe, the exception for usage of the REST API, not the rule. I believe most people that would take advantage of the REST API’s core inclusion are going to use parts of it only, and I believe most of what they’ll be seeking is content related, versus site management.

The WordPress admin is exhaustive: it’s pretty much anything and everything one would want to do in WordPress. Custom admins that would utilize the REST API are unlikely to completely mimic the current admin; if they were to do so, it’d be difficult to see the point of making such an application interface, versus just using the tried and true default. I’m certain some new full implementations of the admin will show up (similar to Calypso), but I don’t think it’ll be a common use case.

More likely, even custom admins would mostly be in the business of manipulating content — specifically catering content editing experiences for a specific type of website — and would be satisfied creating their own custom endpoints for other site management tasks until those endpoints are available through feature plugins or in core itself.

For tasks such as custom front-end websites that are decoupled from WordPress, the WordPress REST API is immediately useful, using the four proposed objects. As K. Adam White said, delaying the endpoints that are ready, “will effectively block a significant group of potential adopters coming from external platforms.”

The value of core inclusion, verses keeping endpoints in the plugin

Core inclusion of endpoints is a stamp of approval that the endpoints are ready for prime time. Organizations that would use the API for bespoke applications would prefer to know that the endpoints will be ready and available for the long term, and keeping endpoints out of core is a sign — intended or not — that things can change, and will prevent adoption that is important for WordPress’s growth.

Whereas, a procedure of iterative development, vertically focused by endpoint, can be a signal of readiness provided by the WordPress core team, essentially saying that those endpoints in core are stable and will maintain WordPress’s well known commitment to backward compatibility, while those endpoints in the feature plugin may be stable, but are subject to change.

Iteration has been good for WordPress

Iteration has been paramount to nearly all WordPress features, and it doesn’t make sense why it wouldn’t be a part of the REST API’s core inclusion.

In a meeting about the status of the API, Matt Mullenweg said, “I know it’s a minority opinion, but I would be pretty skeptical of merging a partial API into core, I think it would do a lot more damage than benefit.” I fail to understand what kind of damage including these endpoints would cause.

Matt has insinuated that the API as-is would be like shipping a car with no wheels, and I vehemently disagree with the analogy. I think the API is more like a Prius, and the goal is a Tesla. It’s a hybrid, with partial but important coverage, that betters the web (or the environment, for the car analogy). It’s not full coverage (like the fully electric Tesla), but it’s better than nothing (gas guzzler!). Okay, enough analogies, because honestly those don’t help that much. But the API as-is is certainly not a wheel-less car. It’s a perfectly drivable and nice car, it’s just not yet fully the car of tomorrow.

Iteration and WordPress have gone hand in hand for ages. Almost every feature I can think of in WordPress has iterated over time, from the initial release: the Customizer, WordPress media management, custom post types, menu management, and more. It’s important to ship features that work, but that doesn’t mean they have to be exactly what you envision. WordPress itself is far from perfect, yet it’s available for download. Every lead developer could tell you dozens of things they’d like to improve, but they’re perfectly fine shipping new releases without every one of those new features.

Vertically integrated API support

Let’s do a mental exercise on the difference between building horizontal and vertical support in the API. This isn’t how one would really build an API, but it’s how we’ll envision calculating support for it.

If you envision the WordPress dashboard, there are many paths you can take. Every top level admin tab is available to you: Add new post, visit the settings page, choose a theme, etc. The REST API cannot currently mimic every action you can take. I would define horizontal support as visiting the admin, identifying each potential action one step away from that first view, and creating that functionality with the REST API. You’d have a lot more endpoints, but the endpoints would be incomplete — and one layer deep.

The next step, if working horizontally, would be to take each new view you just visited, based on the previous action, and do it again. This would cover a lot more functionality. Change widget order, move menu items around, delete comments, etc. But there are yet further levels you can take each of these screens. However, say the initial screen had 12 possible events, and now each of those 12 has 12 possible events, you now have 144 events to mimic in the API, but you still have layers to go before you have complete support of any particular fully fledged task.

If you continue that process, you’ll be attacking support horizontally, and you’ll be at it for a long time.

Now, let’s think vertically. Let’s choose an end goal: like publish a blog post. Okay, now let’s go back to the admin. This time, we’re going to enable support for adding new posts with the API; that’s one action, and we’re going to ignore other possible actions that don’t have to do with adding a post.

On our next screen, we have new options, but not all of them are explicitly about publishing a post. However, many of these options are important to publishing a post: information like assigning an author (users), and putting the post into a category (terms). So we’re going to do what we need to do with each of these things that enables us to better support posting.

Okay, so we can now publish a post via the API, and we’ve also enabled baseline support for stuff that posts expect. The next step in the flow may be that someone would comment on that post. So we’re going vertically through the process. We’re providing complete coverage for these steps. And it gets more complicated than what I’ve said so far. Posts could have meta data, or be password protected, or they might be multiple post types.

Each of these creates new considerations we must account for to have complete vertical coverage of the process of publishing a post. We haven’t even touched managing site settings or other functionality that doesn’t apply to publishing a post. But as we realize the stuff that interacts with this most important task of publishing content, we begin to form a minimum viable product for our API, to know what we need to ship to fully and well support all things posts with the API. You may still take 144 steps, but at the end, you’ll have more coverage of fewer things, versus light coverage of all things.

This is what the REST API team has done with these four objects. Posts, terms, users, and comments are the content processing experience in WordPress. It’s a fully fledged API experience, without shortcuts. There are challenges of course, and room to improve, but from top to bottom the process of content creation and manipulation is part of the API.

As the REST API team have said in various channels during this debate, the non-supported endpoints are mostly isolated from the publishing experience. Supporting posts is co-dependent on supporting the author of the post, and the terms for the post. Supporting the settings screens for the website, or managing themes and plugins, or managing widget areas are not co-dependent on the post experience.

These other verticals are important, and it is everyone’s goals to create broad, horizontal support of WordPress functionality. But it should be done one vertical feature at a time, and we shouldn’t wait to ship something that is near ready for prime time until we’ve supported absolutely everything. For one, the API is useful to many people right now, but also, the API, unless the entire development strategy of WordPress changes, will never be caught up to support every WordPress feature.

Truly “complete coverage” would require API-first driven development

For the WordPress REST API to truly have “complete coverage” of WordPress features, then the API would have to be the first consideration when building a new feature. This wouldn’t be a bad thing! But it’s not the way WordPress development works right now.

Stripe is a great example. When you hear a developer talk about Stripe, the popular merchant processor, the first thing they’ll say is how good the API is. Well, that’s because Stripe is the Stripe API. No feature ships unless it started with API coverage.

Developers love working with Stripe, because anything that can be accomplished with Stripe is accomplished with the API. With WordPress, the REST API team is running to catch up with core development; core development is not centrally focused around the REST API.

And as long as core development operates outside the scope of API-first, then the API will always be running to catch up, and the smaller team surrounding API development will never maintain pace with core.

WordPress already learned this lesson with unit testing. For years, unit testing was nice to have, but not required for commit. Today, WordPress has unit testing coverage of less than 50%, even though all new commits require tests. Unit testing, though now a first class citizen to WordPress development, is still catching up.

I’m not saying whether or not WordPress should be developed API-first, but I am saying that Matt’s goal of complete coverage before shipping would just mean the API will never ship, because it will never have complete coverage with the current development strategy.

The importance of adoption, and design for the majority

It was recently pointed out that the REST API infrastructure is only used by 20 plugins. It makes it appear as if it’s not an important WordPress feature, and that sentiment is wrong.

Consider nearly any other software. I’ll use MailChimp as an example this time. Loads of people use Mailchimp. Mailchimp has a REST API, and I use it for Post Status. My implementation is not one of their official integrations, which they noted they have 845 of in their 2015 report. Mine is just me interacting with their API — only trackable by counting total API calls.

Well, Mailchimp has millions of users, and those millions of user accounts sent over 204 billion emails in 2015. A fraction of those users (I assume) are responsible for more than 29 billion calls to Mailchimp’s API, either through custom interactions (like me), or through one of the 845 official Mailchimp integrations. It really doesn’t matter that there are only 845 integrations with Mailchimp’s API; because the impact of those integrations and the developers otherwise interacting with the API had huge implications that make Mailchimp a more attractive service.

The second core philosophy of WordPress is to “design for the majority.” It says, “Many end users of WordPress are non-technically minded. They don’t know what AJAX is, nor do they care about which version of PHP they are using.” And while this is true, it doesn’t mean that technical features don’t benefit the majority, even if they aren’t directly interacting with them.

The WordPress REST API is an invisible feature to most users, but it also will eventually impact nearly all users of WordPress. Like Mailchimp and Stripe, the WordPress REST API can and will impact the overall experience of using WordPress itself. Especially as third parties create gateways for interaction with millions of WordPress installs via the API, the API will be very much for the majority. But it will always be the kind of situation where a small number of integrations are responsible for a large number of interactions.

We should not base our decision making process for including the API in core on how many plugins are currently supporting the API. We should base our decision making on the potential impact developers can have developing applications with WordPress via the REST API, by utilizing the power in the API — power that can be utilized to improve publishing for everyone.

The time for the WordPress REST API is now

The time for the WordPress REST API is now. The API is not complete, but it is ready. This is step one. There are real issues to be resolved with the first four objects, but the first four objects should not sit on the sidelines, excluded from WordPress core, because “complete coverage” doesn’t yet exist.

There is a capable and willing team driving the WordPress REST API. We all owe our gratitude to Ryan McCue, Rachel Baker, Daniel Bachhuber, Joe Hoyle, and the other contributors that have worked tirelessly — many of them for many years — to make the WordPress REST API a reality.

We should not take these efforts for granted. A contributor’s time is finite, and everyone that’s worked hard on the API could have spent that time on other things. While time spent shouldn’t be our metric for including a feature in core, we should consider it now. This team has worked for a long time, all along asking for others to provide feedback, and at the 11th hour a vocal (and powerful) minority have decided to throw considerable roadblocks at their efforts.

The WordPress REST API is a good feature for WordPress to have. REST APIs are a staple of modern web applications. The endpoints proposed for core inclusion is a reasonable next step after the infrastructure that went in WordPress 4.4. Whether these first four objects are ready for 4.5, or get postponed until 4.6 isn’t a huge deal; they should be ready before they are merged. But they should not be sidelined until the fantasy of complete coverage is met.

The inclusion of the WordPress REST API should be iterative, smart, and focused. It shouldn’t be all or nothing. It should be now.