API contracts are essentially set in stone, and adding, changing, or iterating on them is often cumbersome and difficult. API change processes themselves can be frustrating and slow, and mistakes can be extremely costly. But as a product’s functionality grows, so should its API—so having well-defined and well-working change processes is vital to a business.
Two years ago, the internal API review and development process at PagerDuty left much to be desired. A former employee had laid a great foundation, but participation had waned. By late 2018, our internal advisory group, the API Guild, hadn’t met for months. Moreover, we hadn’t published any new publicly available API endpoints in more than a year.
In this article, I’ll share how we kick-started PagerDuty’s API development with little new software development and some simple process changes. While this initiative started as a joint effort between me, the engineering manager of PagerDuty’s developer ecosystem, and the team’s product manager, it quickly became a broader collaborative effort.
A plan is formed
We set out to achieve two simple goals:
Inspire more PagerDuty engineering and product folks to care about APIs.
Foster the creation and maintenance of our APIs.
We had some serious roadblocks to overcome in order to create an environment where more APIs were being built and more people at PagerDuty cared about our APIs. First, our delivery team had very little room on their roadmap for additional development work. This meant we couldn’t code ourselves out of the problem. Second, PagerDutyʼs public APIs have distributed ownership and responsibility. Our developer ecosystem team owns the infrastructure that powers our APIs and the schema definitions, but each endpoint is owned by its respective delivery team. The delivery teams are responsible for maintaining their endpoint and fixing bugs. This distributed ownership model meant we needed to have the entire product and engineering group on board with our API process and tooling, and we needed to ensure they were empowered to contribute to it.
What’s in a name?
To inspire and build a community around API development, we needed to start by cleaning up our internal API interest group, the API Guild. We called a meeting for anyone who was still in the group and shared our intentions. The meeting went extremely well; those in attendance understood and recognized the need for change. In order to reboot the group and sustain that enthusiasm, we opted for a fresh start: We got rid of old processes and the backlog of discussion topics, and made some impactful changes to the group’s format and presentation.
We knew we needed to capture people’s attention in order to generate momentum, so our first change was to rename the group API Allies. This simple change gave members a new perspective and made it clear this was a new group with new intentions.
When it came time to create new paths and processes, we opted to keep it simple, aiming to make the group accessible to new people who wanted to participate or follow along. Since weʼre a relatively small organization, just over 500 people, we were able to avoid putting formal processes in place in cases where no more than two people or four steps were involved. We established clear accountability by assigning a delivery team as the fallback for all unclaimed action items, including new API proposal reviews, new schema structure investigation, and documentation tasks.
To streamline communication and participation while increasing company-wide visibility, we decided to adopt the following practices:
Rather than a mailing list, weʼd use only our Slack channel (#api-allies) for communication.
Anyone would be able to propose an agenda item by putting it on the groupʼs Trello board.
Developers could request a review for any change using
@herein our #api-allies channel.Anyone could report bugs or inconsistencies using
@herein our #api-allies channel.
A focus on empowerment
Getting more people to contribute was about more than merely getting the word out about API Allies; it was an exercise in empowering our developers, product owners, and technical support team through simplified workflows. To get started, we made extensive use of automation in our schema repository using pull request templates and inline linting through GitHub Actions. We also introduced our internal success and support teams to Stoplight Studio, a GUI for the OpenAPI schema, so they could contribute to our API documentation without even having to open a text editor.
Throughout the months of renewal, we heard feedback from delivery teams that publishing a new API endpoint was daunting, because at PagerDuty we’ve agreed that once you publish an API endpoint, you can’t make breaking changes to it. To help teams feel more comfortable building new API endpoints, we developed a new tool called the Early Access Framework. The tool makes it easy for teams to flag their APIs as early access, which lets consumers of the API know the contract may change. It also provides valuable usage data to the team so they can iterate on their design.
Streamlining processes
With API Allies up and running smoothly and some light tooling in place, we decided we needed to focus on getting more APIs published. We wanted to enable teams to do so with great documentation and a straightforward process for participation. We set out to create a light framework for getting reviews on new API additions and changes. We didn’t want API Allies to become a gatekeeper for our APIs, so we designed the framework to be recommended but not required. The framework is straightforward:
Create a proposal using our API Change Proposal template.
Ask for a review of the proposal from API Allies via Slack, and two reviewers will be assigned.
Refine the proposal with the assigned reviewers.
Ship it.
Since launching this recommended framework, we’ve had over 10 delivery teams use it, and only one choose not to. We believe the high enrollment is the result of trusting delivery teams to follow best practices when they’re laid out. The one team that didn’t use the framework wound up running into problems designing their API schema, which delayed their project.
Modernizing our tooling
Having tooling that simply works is important for reducing friction around publishing APIs, and itʼs an area where we still have a lot of work to do.
During our revitalization efforts, we determined that upgrading our schema definition tooling would be one of the best ways to support developers. Our schema was defined in Swagger 2.0, with custom tooling for packaging and deploying, and was hosted in nonstandard ways. It was extremely unfriendly to our internal developers: The continuous integration tooling would constantly break because it relied on four external dependencies, and there were many manual deployment steps. An upgrade would minimize friction by reducing touch points with outdated and poorly documented tooling; it would also bring our tooling in line with the rest of the open-source community, making it easier to get support.
We upgraded to modern tooling, leaned on Stoplight to serve our documentation, and turned to the open-source community. In keeping with our pursuit of simplicity, we shrunk the amount of documentation in the repository by moving any verbose explanations to our proposal format, maintaining an FAQ document for new developers, and referencing external documentation for more complicated concepts such as OpenAPI. We also documented the upgrade of our tooling using our proposal format, ensuring future developers would be able to go back and see why we made a particular decision.
The new tooling enabled us to deploy changes to our API schema in just three steps:
Merge into the internal repository.
Merge into our public schema repository.
Manually update our Postman account (sadly, this step couldnʼt be automated).
What still needs to be done
We still have a lot of room for improvement, especially when it comes to providing feedback to delivery teams on the user stories their endpoints are enabling. However, since we revamped API Allies 18 months ago, PagerDuty has published four new endpoints, continued to host API Allies meetings biweekly, and started the discovery phase on a broader change to our internal API architecture, which is intended to supercharge delivery teamsʼ ability to ship API endpoints. During this process, we’ve learned a lot about how keeping processes simple and giving your team the tools they need can lead to increased involvement and enthusiasm throughout an organization.
If you have an internal API interest group that could use some refreshing, try applying the lessons we learned along the way:
Examine the process first; coding a solution is often not the first thing you should do.
Think of process as a funnel for involvement rather than a way to control the outcome.
Encourage people outside the group to contribute by making quality-of-life improvements such as automating repetitive workflows.
Keep it simple. Complicated processes and documentation discourage participation.
Keep these lessons in mind, and youʼll be well on your way to a rewarding refresh.