On one Saturday evening I went to backyard with a cup of nice freshly brewed coffee to enjoy life and good weather (it was not raining in Tampere). The love towards open source among the developers started to bother me a bit. Perhaps bother is a bit too strong word, so I might say wonder . Once again I started thinking about APIs and API Economy. I’m a visual person and thus I started to draw a picture of my thoughts. I came up with the below picture and tons of questions in my head.
Questions did not stop but kept on coming. More and more. Some of the questions were
- “What is the role of open source in API economy? What will happen to open source once the APIs become more popular?”
- “Why the hell should I open source my API?” kept on spinning in my head and started to bother me.
I love open source and I have learned 95% of my feeble skills from the open source community. This post became rather lengthy due to the reason that both APIs and open source are close to my heart.
APIs have become the default building blocks
Classical value of open source defined by Weber (“The Political Economy of F/OSS Software”) is “Avoid wasteful duplication of programming efforts by reusing existing software”.
In the age of APIs that value is still valid, but now the duplication is avoided by using APIs developed and often maintained by someone else.
APIs operated over http or other protocol (popular term is web-APIs) have become the default building blocks of services much like you would build a castle from lego blocks. The analogy is not perfect since with legos you don’t need glue at all, but with APIs you often need a little bit of own code (glue) to keep the castle in one piece. Nevertheless, more and more services are built by utilizing APIs built by 3rd parties.
As a service developer you take the APIs in use (some commercial) to solve some problems in achieving your goal instead of developing the solution from scratch. That makes sense. I was scrathing an itch and I thought that some other people would have the same questions in head and needed answers.
API solves are problem for someone else. It solves it well and is so easy to onboard that some even pay money for it. The same “scratch an itch” was and still is one of the ideas and driving forces of open source. Someone else has had the same itch to scratch and has decided to share the solution code with some permissive license. Similarities do not end there.
With APIs you have no need to maintain code
With APIs you don’t maintain the code and same is with open source components. You don’t sell open source code. Instead you build services by stitching together pieces of open source code and sell the result to consumers. APIs are sold as services too. Both of them are monetised as services. Or course open source can be used in closed source apps which are then sold to public as software. Some of the open source licenses permit this but some dont’. But still the idea of scratching an itch is there in both.
Closed source code != everything is closed
Source code of APIs are quite commonly closed source nowadays. Does that mean all APIs are closed by nature? No it does not. As a consumer of the API, you are not consuming the API per se, but the logic behind it - value proposition. Thus having the API code open most likely does not benefit anyone. Why so? Well, normally APIs are facades in front of the backend and the logic is in the backend, not in the API. Even the integration might be in between the API and backend and handled with yet another solution. API is just a access point to value proposition.
They idea on an API is that the consumer does not need to care about what’s behind it and how it manages given tasks. It just does and that is what you pay for. As a consumer I have no need to copy API anywhere. Why would open sourcing that benefit any 3rd party API consumer? It normally doesn’t and thus API code is not open source normally.
Exceptions of course exists and public sector is one of those. They want to build API ones and replicate it to other places which are running the same backend. Events around one city area are stored in one (or multiple systems). Events for another city are in different systems. Now the two cities want to make developer’s life easier and provide same API for the events. Solution is that they add the same API on top of separate backend systems. They can do so with ease since it’s open source and shared in Github. Of course the logic is still in the backend and some minor adjustments might be needed, but the promise looks solid. Of course if they want to make developer’s life much more easier, they agree on the data models used too. Otherwise they solve the API hell problem (multiple different APIs for same purpose), but create data model hell. Another case for open sourcing API is that if you contribute it to some existing open source solution, application, service, framework, SDK, library.
Open machine-readable specifications
Significant amount of API providers (like Platform of Trust) have decided to share machine-readable specs of the open APIs for anyone to use. Most common format is OpenAPI spec which was previously known as swagger. Next to OpenAPI spec is still RAML and API Blueprint, but the tooling and buzz around those is fading since even the pushers of RAML have joined OpenAPI initiative under Linux Foundation. It seems evident that we are on the road towards OpenAPI spec monopoly regarding REST API specs.
APIs fuel the platform economy
Anyone following the global IT and business scene has probably read millions of times that we live in the age of platforms. I will not go deep into the subject but leave this task for reader. Excellent books about platform economy are for example Modern Monopolies: What It Takes to Dominate the 21st Century Economy by Moazed and Johnson and Platform Revolution: How Networked Markets Are Transforming the Economy–and How to Make Them Work for You by Parker et al.
All the major platforms are fueled by APIs…and utilize open source as well. Some see the platform giants as leeches - taking advantage of open source but not contributing (enough) back to it. This is pushing companies away from open source. As new solutions are developed as open source, the giants take the code and sell it as a service for their platform consumers. In other words they get the profits, but do not give back. Or at least that is the somewwhat general claim as can be seen in “Amazon Has Gone From Neutral Platform to Cutthroat Competitor, Say Open Source Developers”.
AWS is striking at the Achilles’ heel of open source: lifting the work of others, and renting access to it.
long-standing trend of AWS rolling out managed services of popular open source technology, or replicating such technologies… This move is a text-book commoditization move — providing Elastic’s premium services for free.
In short, elacticsearch and alike open source driven companies build successful products, sell it as subscription. Then comes bigger fish in the sea (any of the giants) and adds the product in the catalogue for their customers for free. That really does not encourage any business people to choose open source for very long time.
Rise of network connectivity
Another phenomenon which has increased the success of APIs compared to open source is the rise of network connectivity and lowered costs for that. Internet with high quality speed is available more than 10 years ago, which was probably the prime time of open source.
Connectivity lowers the need for local processing and APIs offer easy way to expand applications to operate in realtime around the internet. Local application might be just a skeleton while data and part of the logic is distributed around the internet behind APIs.
Of course the availability of high speed and reliable internet connection varies from location to location. Building services on top of APIs requires constant internet connection and minimal amount of processing or storing of information is at the device at the moment. Edge computing has emerged and more of the data processing is handled in the devices and thus internet connection is not always needed in data crunching.
Rising security threaths of open source…and APIs alike
Some of the developers I’ve interviewed during the past years have indicated distrust towards complete frameworks and runtime environments due to poor code quality and badly managed quality control including security. The examples for example with NPM have increased the mistrust and turned developers towards other solutions.
Malicious code is now injected to packages which are installed in seconds onto local machine by single developer. Same package might be installed in tens of thousands of applications per day as was witnessed for example Feb 2018 when a core contributor to the conventional-changelog ecosystem had their npm credentials compromised and a malicious version of the package was published. Package was installed 28,000 times in 35 hours and executed a Monero crypto miner.
Previously open source code was “closer” to the developer and often reviewed before using or at least the code was in immediate proximity. Now the code is behind another step and component is installed with single line of code in commandline (for example npm and pypi). More often code is even further away from sight as packages installed are configured in separate file. Of course close to similar situation has been as long as the libraries have been added to applications as includes. Yet the ease of taking someone elses code into use and go to production with it, is a lot easier now. Also the update of components is nowadays more automised and handled by software, not developers. That opens a new attack vector for malicious code if using open source.
Now the situation seems to be a bit problematic. Developers love open source because it enables modifications to fit own purposes, reduces redundant work and so on. At the same time they wittness security flaws and risk which causes mistrust towards open source. Developers’s love APIs due to ease of use, but again the security risks cause friction. Security risks causing for example customer data breaches is probably the biggest thread to both open source and APIs. In that sense they are standing side by side.
Position of open source changed when APIs expanded to http. Previously APIs were used inside the applications to enable component reuse and exposing capabilities for other components. After so called web APIs which interact with other applications via http came along, the situation changed dramatically.
Tools development is often open source driven. The OpenAPI spec is a prime example of this. It has grown a vast toolset around itself. But here as in platform business - winner takes all. Rest of the REST API spec formats are pretty much taking the scraps. Even the bashed AWS has opened the boto3 core code. And if you google as little more you’ll find huge army of open source tools around platforms and APIs.
Expose API specifications instead of source code. Instead of pushing source code to Github or other public source code management solutions, API providers publish machine-readable specifications in various formats. Currently most popular format is OpenAPI spec which was previously known as Swagger. Other specification options are RAML and Blueprint, but the ecosystem and tooling around those two is declining. Swagger “won” the API spec war some years ago and has become the de facto API specification.
But are the spec files really open? In cases when there’s no license attached to it, then no. In those cases spec files are just publicly available. If there would be or is a open source license or permnissive Creative commons license next to it, then the spec files would be open data/source. Open in the spec files is more equivalent to “Open API”, which is not mandated to be open source or contain open data - it’s just available for all to utilise.
Utilize expanding tool base. Exposure of source code is not necessary to ensure great developer experience for the API consumers. A plethora of tools have emerged around the machine-readable specifications enabling automatic duplication of the API inside own development environment or anywhere else. Of course the duplicated API does not have the same content, but often specification contains example values and return sets to emulate API well enough for development purposes.
Some more to read from 100 Days DX
- #100 - The biggest open resource on Developer eXperience so far
- #99 - Hackers - the top 1% of engineers
- #98 - Invalid Open API spec files ruins the developer experience
- #97 - Lightweight API evaluation framework
- #96 - Embracing open source community driven tools development
- #95 - Exploring the multilevel nature of API Design Guides
While you are here...
Check out Platform of Trust in which I've worked to enable best possible developer experience!