The WordPress REST API is at a bit of a crossroads. There is a proposal on the table by the core team of four contributors — Ryan McCue, Rachel Baker, Daniel Bachhuber, and Joe Hoyle — to ship endpoints to WordPress core iteratively. There is a pushback on this proposal by WordPress Project Lead and co-founder Matt Mullenweg.
The API should be iterative, and should not require complete admin parity to be included, as Matt has proposed.
The state of what’s already in core, and what’s pending
The REST API infrastructure is already in core, having been included in WordPress 4.4. The infrastructure provides just that, an underlying infrastructure, to offer plugins a suite of base functionality to utilize, making the development of their own RESTful features easier.
The infrastructure allows developers to get a huge head start for creating their own REST API endpoints and functionality. The endpoints are how a third party client or developer would actually interact with the API, and the endpoints for WordPress core functionality are what are in debate.
Every piece of WordPress functionality — data that can be read, created, updated, or deleted — would ideally be available to the WordPress REST API, so that anything possible in WordPress is also possible via the API, and therefore also possible from a third party application interacting with the API.
However, creating this complete coverage is unrealistic in the current WordPress development structure, and should not be necessary for shipping initial endpoints to WordPress Core.
The proposal from the core REST API team is to begin by shipping four core WordPress objects: posts (meaning the posts table, which includes all post types), terms, comments, and users. These four core objects have been developed as endpoints, are well tested, and are ready for core. There are kinks to work out, but overall, they are an excellent starting point.
Matt’s proposal is that the API should not ship in core, with any endpoints, until complete “wp-admin coverage” is available: meaning that until all actions a user can make in the WordPress admin can be achieved via the API, then it shouldn’t go into core. He said in a meeting about the API, to firm up his proposal, that there should be, “no partial endpoints in core. let’s design a complete API, not half-do it and foist it on millions of sites.”
The primary areas not covered by the API currently are as follows: widget management and display, menu management and display, theme and plugin management, multisite management, and site options management.
So Matt would like to see each of these endpoints be developed and battle tested before any endpoints ship in core. I agree with the REST API team that the first four objects should take priority, and should go in before these other objects are ready. The remaining objects should each be feature plugins of their own, and go in over time, as they are ready, without delaying the four core objects currently under review.
Many use cases covered
Posts, terms, comments, and users endpoints make up practically all WordPress content. This means that a huge swath of use cases for the API are covered with these four objects.
Matt’s thinking that a core WordPress REST API should have all or nothing coverage is a false dichotomy. Applications like Calypso — a WordPress admin replacement based on WordPress.com’s own and separate REST API — are, I believe, the exception for usage of the REST API, not the rule. I believe most people that would take advantage of the REST API’s core inclusion are going to use parts of it only, and I believe most of what they’ll be seeking is content related, versus site management.
The WordPress admin is exhaustive: it’s pretty much anything and everything one would want to do in WordPress. Custom admins that would utilize the REST API are unlikely to completely mimic the current admin; if they were to do so, it’d be difficult to see the point of making such an application interface, versus just using the tried and true default. I’m certain some new full implementations of the admin will show up (similar to Calypso), but I don’t think it’ll be a common use case.
More likely, even custom admins would mostly be in the business of manipulating content — specifically catering content editing experiences for a specific type of website — and would be satisfied creating their own custom endpoints for other site management tasks until those endpoints are available through feature plugins or in core itself.
For tasks such as custom front-end websites that are decoupled from WordPress, the WordPress REST API is immediately useful, using the four proposed objects. As K. Adam White said, delaying the endpoints that are ready, “will effectively block a significant group of potential adopters coming from external platforms.”
The value of core inclusion, verses keeping endpoints in the plugin
Core inclusion of endpoints is a stamp of approval that the endpoints are ready for prime time. Organizations that would use the API for bespoke applications would prefer to know that the endpoints will be ready and available for the long term, and keeping endpoints out of core is a sign — intended or not — that things can change, and will prevent adoption that is important for WordPress’s growth.
Whereas, a procedure of iterative development, vertically focused by endpoint, can be a signal of readiness provided by the WordPress core team, essentially saying that those endpoints in core are stable and will maintain WordPress’s well known commitment to backward compatibility, while those endpoints in the feature plugin may be stable, but are subject to change.
Iteration has been good for WordPress
Iteration has been paramount to nearly all WordPress features, and it doesn’t make sense why it wouldn’t be a part of the REST API’s core inclusion.
In a meeting about the status of the API, Matt Mullenweg said, “I know it’s a minority opinion, but I would be pretty skeptical of merging a partial API into core, I think it would do a lot more damage than benefit.” I fail to understand what kind of damage including these endpoints would cause.
Matt has insinuated that the API as-is would be like shipping a car with no wheels, and I vehemently disagree with the analogy. I think the API is more like a Prius, and the goal is a Tesla. It’s a hybrid, with partial but important coverage, that betters the web (or the environment, for the car analogy). It’s not full coverage (like the fully electric Tesla), but it’s better than nothing (gas guzzler!). Okay, enough analogies, because honestly those don’t help that much. But the API as-is is certainly not a wheel-less car. It’s a perfectly drivable and nice car, it’s just not yet fully the car of tomorrow.
Iteration and WordPress have gone hand in hand for ages. Almost every feature I can think of in WordPress has iterated over time, from the initial release: the Customizer, WordPress media management, custom post types, menu management, and more. It’s important to ship features that work, but that doesn’t mean they have to be exactly what you envision. WordPress itself is far from perfect, yet it’s available for download. Every lead developer could tell you dozens of things they’d like to improve, but they’re perfectly fine shipping new releases without every one of those new features.
Vertically integrated API support
Let’s do a mental exercise on the difference between building horizontal and vertical support in the API. This isn’t how one would really build an API, but it’s how we’ll envision calculating support for it.
If you envision the WordPress dashboard, there are many paths you can take. Every top level admin tab is available to you: Add new post, visit the settings page, choose a theme, etc. The REST API cannot currently mimic every action you can take. I would define horizontal support as visiting the admin, identifying each potential action one step away from that first view, and creating that functionality with the REST API. You’d have a lot more endpoints, but the endpoints would be incomplete — and one layer deep.
The next step, if working horizontally, would be to take each new view you just visited, based on the previous action, and do it again. This would cover a lot more functionality. Change widget order, move menu items around, delete comments, etc. But there are yet further levels you can take each of these screens. However, say the initial screen had 12 possible events, and now each of those 12 has 12 possible events, you now have 144 events to mimic in the API, but you still have layers to go before you have complete support of any particular fully fledged task.
If you continue that process, you’ll be attacking support horizontally, and you’ll be at it for a long time.
Now, let’s think vertically. Let’s choose an end goal: like publish a blog post. Okay, now let’s go back to the admin. This time, we’re going to enable support for adding new posts with the API; that’s one action, and we’re going to ignore other possible actions that don’t have to do with adding a post.
On our next screen, we have new options, but not all of them are explicitly about publishing a post. However, many of these options are important to publishing a post: information like assigning an author (users), and putting the post into a category (terms). So we’re going to do what we need to do with each of these things that enables us to better support posting.
Okay, so we can now publish a post via the API, and we’ve also enabled baseline support for stuff that posts expect. The next step in the flow may be that someone would comment on that post. So we’re going vertically through the process. We’re providing complete coverage for these steps. And it gets more complicated than what I’ve said so far. Posts could have meta data, or be password protected, or they might be multiple post types.
Each of these creates new considerations we must account for to have complete vertical coverage of the process of publishing a post. We haven’t even touched managing site settings or other functionality that doesn’t apply to publishing a post. But as we realize the stuff that interacts with this most important task of publishing content, we begin to form a minimum viable product for our API, to know what we need to ship to fully and well support all things posts with the API. You may still take 144 steps, but at the end, you’ll have more coverage of fewer things, versus light coverage of all things.
This is what the REST API team has done with these four objects. Posts, terms, users, and comments are the content processing experience in WordPress. It’s a fully fledged API experience, without shortcuts. There are challenges of course, and room to improve, but from top to bottom the process of content creation and manipulation is part of the API.
As the REST API team have said in various channels during this debate, the non-supported endpoints are mostly isolated from the publishing experience. Supporting posts is co-dependent on supporting the author of the post, and the terms for the post. Supporting the settings screens for the website, or managing themes and plugins, or managing widget areas are not co-dependent on the post experience.
These other verticals are important, and it is everyone’s goals to create broad, horizontal support of WordPress functionality. But it should be done one vertical feature at a time, and we shouldn’t wait to ship something that is near ready for prime time until we’ve supported absolutely everything. For one, the API is useful to many people right now, but also, the API, unless the entire development strategy of WordPress changes, will never be caught up to support every WordPress feature.
Truly “complete coverage” would require API-first driven development
For the WordPress REST API to truly have “complete coverage” of WordPress features, then the API would have to be the first consideration when building a new feature. This wouldn’t be a bad thing! But it’s not the way WordPress development works right now.
Stripe is a great example. When you hear a developer talk about Stripe, the popular merchant processor, the first thing they’ll say is how good the API is. Well, that’s because Stripe is the Stripe API. No feature ships unless it started with API coverage.
Developers love working with Stripe, because anything that can be accomplished with Stripe is accomplished with the API. With WordPress, the REST API team is running to catch up with core development; core development is not centrally focused around the REST API.
And as long as core development operates outside the scope of API-first, then the API will always be running to catch up, and the smaller team surrounding API development will never maintain pace with core.
WordPress already learned this lesson with unit testing. For years, unit testing was nice to have, but not required for commit. Today, WordPress has unit testing coverage of less than 50%, even though all new commits require tests. Unit testing, though now a first class citizen to WordPress development, is still catching up.
I’m not saying whether or not WordPress should be developed API-first, but I am saying that Matt’s goal of complete coverage before shipping would just mean the API will never ship, because it will never have complete coverage with the current development strategy.
The importance of adoption, and design for the majority
It was recently pointed out that the REST API infrastructure is only used by 20 plugins. It makes it appear as if it’s not an important WordPress feature, and that sentiment is wrong.
Consider nearly any other software. I’ll use MailChimp as an example this time. Loads of people use Mailchimp. Mailchimp has a REST API, and I use it for Post Status. My implementation is not one of their official integrations, which they noted they have 845 of in their 2015 report. Mine is just me interacting with their API — only trackable by counting total API calls.
Well, Mailchimp has millions of users, and those millions of user accounts sent over 204 billion emails in 2015. A fraction of those users (I assume) are responsible for more than 29 billion calls to Mailchimp’s API, either through custom interactions (like me), or through one of the 845 official Mailchimp integrations. It really doesn’t matter that there are only 845 integrations with Mailchimp’s API; because the impact of those integrations and the developers otherwise interacting with the API had huge implications that make Mailchimp a more attractive service.
The second core philosophy of WordPress is to “design for the majority.” It says, “Many end users of WordPress are non-technically minded. They don’t know what AJAX is, nor do they care about which version of PHP they are using.” And while this is true, it doesn’t mean that technical features don’t benefit the majority, even if they aren’t directly interacting with them.
The WordPress REST API is an invisible feature to most users, but it also will eventually impact nearly all users of WordPress. Like Mailchimp and Stripe, the WordPress REST API can and will impact the overall experience of using WordPress itself. Especially as third parties create gateways for interaction with millions of WordPress installs via the API, the API will be very much for the majority. But it will always be the kind of situation where a small number of integrations are responsible for a large number of interactions.
We should not base our decision making process for including the API in core on how many plugins are currently supporting the API. We should base our decision making on the potential impact developers can have developing applications with WordPress via the REST API, by utilizing the power in the API — power that can be utilized to improve publishing for everyone.
The time for the WordPress REST API is now
The time for the WordPress REST API is now. The API is not complete, but it is ready. This is step one. There are real issues to be resolved with the first four objects, but the first four objects should not sit on the sidelines, excluded from WordPress core, because “complete coverage” doesn’t yet exist.
There is a capable and willing team driving the WordPress REST API. We all owe our gratitude to Ryan McCue, Rachel Baker, Daniel Bachhuber, Joe Hoyle, and the other contributors that have worked tirelessly — many of them for many years — to make the WordPress REST API a reality.
We should not take these efforts for granted. A contributor’s time is finite, and everyone that’s worked hard on the API could have spent that time on other things. While time spent shouldn’t be our metric for including a feature in core, we should consider it now. This team has worked for a long time, all along asking for others to provide feedback, and at the 11th hour a vocal (and powerful) minority have decided to throw considerable roadblocks at their efforts.
The WordPress REST API is a good feature for WordPress to have. REST APIs are a staple of modern web applications. The endpoints proposed for core inclusion is a reasonable next step after the infrastructure that went in WordPress 4.4. Whether these first four objects are ready for 4.5, or get postponed until 4.6 isn’t a huge deal; they should be ready before they are merged. But they should not be sidelined until the fantasy of complete coverage is met.
The inclusion of the WordPress REST API should be iterative, smart, and focused. It shouldn’t be all or nothing. It should be now.
WordPress unit testing coverage is now 53.59%, the reference you used is ~20 months old, for up to date code coverage stats see:
https://codecov.io/github/aaronjorbin/wordpress.codecov
“The WordPress REST API is at a bit of a crossroads”
Phew, I thought it was at a gridlock.
🙂
#SpotOn
I think you’ve argued all the points very well and clearly justified why full feature parity is almost impossible to achieve within an acceptable timeframe.
I wonder what Matt will come back with…
You’ve brilliantly communicated what those of us in the trenches of WordPress (as opposed to those of us biased by our self-interest in a minority demographic of wordpress.com users) instinctively know. The REST API team have done a fantastic and selfless job and we want these endpoints in core.
You also didn’t need to obfuscate a weak argument with tenuous metaphors like those on the other side of the debate.
If you’re going to blurt out opinions before you’ve fully thought through your argument (and we all do it from time to time) then you need to be capable of hearing the counter argument and putting on your big boy pants to admit that perhaps your initial thoughts were wrong. This is surely now the position Matt finds himself in.
Oh, and Calypso doesn’t have feature parity with wp-admin. wordpress.com wp-admin maybe, but not wp-admin as the majority of us know it…
I agree. It’s not all or nothing, it should be now.
“The inclusion of the WordPress REST API should be iterative, smart, and focused. It shouldn’t be all or nothing. It should be now.”
Um. Well. Ha!
Actually, you’ve just made the case for why it should, perhaps forever, be a plugin. If the rationale for stuffing the REST API into core is so it IS the product – new & improved, etc – and it’s not going to be that, well then what’s the new reasoning? “Because X said so…” or “Because we need a shiny new objective to rub…”?
The simple – and smart answer – is to NOT add it to core, at least no at this time. Clearly it’s not “core” to everyone. What exactly is gained by giving more installs more (core) code than they actually need?
WordPress is sold as a mature and world-class product with MASSIVE market share. In short, LOTS of people depend on WP. Rah rah emotional “rationalizations” don’t belong in this conversation. Please reconsider. Thx.
I think you misunderstood Brian’s point about . He isn’t arguing the rationale for adding theAPI to core is because it “IS the product”. He wrote that “wouldn’t be a bad thing! But it’s not the way WordPress development works”. Rather, he’s using examples of APIs that *are* the product to show how if WP development worked that way, then the API would remain a plugin for a long, long time, and that would be a bad thing.
You seem to be arguing the API doesn’t belong in core and should always be a plugin. I suppose there’s room for legitimate disagree around that point, although I don’t see it.
Yes, WordPress has been sold as a world-class product. But, to continue to be seen that way, it needs the API. The argument for adding the API isn’t because it’s “shiny”. It’s because of how powerful it is, and what it allows you to do with WordPress that’s never been possible before. WordPress isn’t the best choice for a lot of applications. The API dramatically opens up the possibilities of what it’s good for.
Adding parts of the API to core as soon as they are solid, before it has complete admin functionality coverage, benefits a lot of developers (and, by extension, users), without hurting developers (and users) who won’t need it.
Holding those features back in a plugin discourages adoption, as Brian mentioned. Committing features to core now will maximize WordPress’ benefit.
I think Matt was challenging the idea that the current offering fell short of the number of endpoints he, and others, believe necessary to be remarkably useful to the greater dev community.
As much as I want REST API in core, I don’t think that is a poor argument.
I believe I later read (but could be wrong) that Matt said that parity (which given the current speed of development surely is unattainable in my opinion) with wp-admin should be the goal, but that what the first inclusion into core beyond API infrastructure finally includes isn’t known.
It bothers me that passionate debate about the best way forward for WordPress so often devolves into angry rants and conspiracy theories. We all want the same thing, which is a worthwhile and reasonably solid REST API. It just may take a little longer to realize that goal than those superb project devs, and their supporters want. And they have done remarkable work on this.
Hopefully, it will be sooner than later because my company can’t fully commit to forward compatible, stable larger app builds while the WP-REST API is a plugin. And for very good reasons I think.
I know you think most people are looking to use the REST API for content and not management, I’m apparently the minority. I could care less about the content side of it, all I want is the management side of it. Until that’s implemented, not interested.
This is a great post Brian, but your argument caves in on itself because of what you wrote here:
Why does this ruin your argument?
Because you also admit that the REST API is not complete:
So, to summarize your stance:
• Incomplete things should be kept as feature plugins.
• The REST API is an incomplete thing (and currently a feature plugin).
• The incomplete feature that is the REST API should not remain a feature plugin.
Do you see the inconsistency?
Now, you might argue my points by saying that it’s okay for incomplete things to be shipped, and that the value of the REST API is such that any effort to get it into Core sooner rather than later should be made. You have done this in your article already, essentially. For the most part, you are right that it’s okay to ship incomplete things in Core. But the REST API is a different animal.
Why is it different?
It is the most hyped-up feature of WordPress ever, with the highest expectations placed upon it of any WordPress feature to date.
Can you name one other feature in WordPress that has had conferences made about it before it’s even been completed? Has any other feature had so many thinkpieces, blog posts, WordCamp talks, and podcast chatter about it “changing the future of WordPress” and all of that?
In other words, while it is often appropriate to ship incomplete things, the expectations are so high for the REST API that the “outrage” over it not being included into Core is not consistent with the expectations placed upon it.
Much of the arguments seem to come down to plain old impatience.
Matt’s point of wanting a complete API before inclusion into core is legit, mostly if you look at WP on the long run. It’s far better to deal with difficult unexpected problems before pushing solutions that would risk otherwise to be half baked.
Still, I agree with the author. A complete API coverage is not realistic on any given timeline, that’s asking too much. It’s more probable that development is just abandoned or refocused on an independent fork. WordPress is famous for keeping backward compatibility, but there is a price to be paid: it’s really hard for developers to add innovative features. When systems become too complex after a long phase of incremental development, it’s time to dismantle and rebuild with a more comprehensive architecture.
The risk for our community (if no compromise is reached) is a fork, one for the “old” WordPress and another for a headless WP data server strongly based on the API. Many like to say that WP is the operating system of the web, and its story can very well be similar to Linux distributions. There was one, they became many.
Piero