How does your organization think about resiliency and reliability writ large?

Our engineering principles reflect our mission to prioritize high-quality code and emphasize our commitment to learning from mistakes. We conduct production incident reviews in a safe, nonjudgmental environment to understand both the root cause of a problem and the necessary steps to prepare the system and organization to successfully cope with similar problems in the future.

That’s also why we instrument our systems for observability. You don’t know you have a problem if you don’t monitor the health and performance of your systems—and have actionable, well-documented steps to mitigate issues. No service or system goes to production unless it has some basic monitoring in place.

Finally, we build to last. When building products and solutions, we ensure services will perform and scale given the business doubles in size annually. We run experiments and make data-driven decisions, which also means building services and products that can tolerate frequent change and are easy to extend or build on.

— Victoria Puscas, Deliveroo

We fall back on the resilience of our systems and people when reliability fails, so both are critical to our success. While some elements of a resilient team mirror a resilient network—avoiding single points of failure, for example—we know a resilient team’s foundation requires uniquely human traits like empathy, vulnerability, and understanding.

— Laura Thomson, Fastly

We encourage a culture of learning to continually get better at preventing incidents, reducing impact, and shortening recovery times. Incidents are an opportunity to grow and improve resiliency—we run blameless postmortems that focus on lessons learned and preventative measures.

Scaling reliability requires a focus on technology, people, and process. In terms of technology, we architect and design in a way that’s conducive to more resilience and fault tolerance when things go wrong. Time to recover after failure detection is crucial, so the incident management and recovery process must be easy to deploy. Finally, it’s important to ensure people are well-prepared and invested in making the overall system better.

— Al Sene, DigitalOcean

Our roadmap has two categories: innovation (product features) and tune-up items, which drive reliability and scale to speed of innovation. Tune-up items comprise the backbone of our innovation and include investing in automation of quality gates in order to release frequently, fast incident detection and response, and refactoring to ensure components are secure, scalable, and available, with low latency and continuous delivery.

This amounts to a self-perpetuating flywheel. Investing in tune-up items—and therefore reliability and speed of innovation—leads to higher development velocity, which improves product innovation, which leads to scaling the team, which then leads to more investment in the speed of innovation.

— Bhavini Soneji, Headspace

What measures or metrics do you use to capture investment in reliability?

At a team level, we look at a prioritized list of repair items and reliability-related tickets every sprint. We also track our teams’ service uptime, SEVs by level, time in SEV, and time to restore. At a higher level, when we plan for the quarter or longer time horizons, we create engineering goals around making our platform more stable and scalable. These become part of our roadmap.

There are also times when we need to take a stance on some of our systems’ older parts. For example, we might have a company-level initiative to improve the tools we give restaurants to manage their commercials or menus. For such initiatives, we typically prioritize quality over profitability as metrics. These changes are intended to significantly improve the experiences of our restaurants, riders, or internal partners by giving them useful, reliable, mature tools and systems to grow and work with.

— Victoria Puscas, Deliveroo

We measure common metrics like time to detect and time to recover from issues. We’re also always looking for ways to reframe base reliability metrics around customer impact. It’s important not to default to only measuring easy things, but to really dig in and ask teams, “What are your goals? What do you need to do to get there?” Metric choice is critical to driving the right system optimizations.

— Laura Thomson, Fastly

We strive to monitor and measure everything we can to encapsulate all aspects of the process, find errors, determine the health of our systems, and identify opportunities for future growth. We enact our commitment to reliability by making it everyone’s responsibility.

Typically, we measure overall system availability, subsystems health, and customer experience against internal objectives. We also take into account industry-standard metrics like time to detect, time to recover, change fail rates, and outage durations. As we continue to make investments, we monitor how these metrics trend over time.

— Al Sene, DigitalOcean

We evaluate reliability investment by asking:

• What’s the customer impact? (e.g., bugs, latency, time to value)

• What’s the business impact? (e.g., brand trust, revenue impact due to downtime)

• What’s the impact on the operational efficiency of the business? (e.g., developer and staff productivity)

We use different sets of metrics. The first is a high-level reliability view, which includes count of incidents and customer impact, revenue implications, crash rate, and app store review rating. The second is around production incidents and follow-through, which includes mean time to resolution, fix rate for production bug fixes and incident action items, percentage of incidents reported through customer support versus detected through alerting, and more. The third is proactive software development life cycle pipeline strengthening, including releases canceled or rolled back, quality gates, count of end-to-end test automation and coverage, pre-production environment uptime, and more.

— Bhavini Soneji, Headspace

How do you think about and/or fund projects to address low-probability but high-risk events?

Some events can be mitigated with a clear procedure and line of escalation. We have a few documents that describe what we might need to do and who we might need to involve in case of a data breach or security vulnerability. (Fortunately, I can’t remember anything like that happening in the four years I’ve been here.)

There’s also a risk that your third-party provider goes down or a data center experiences an outage. Then, the question becomes about investing the time to mitigate these issues now versus later. We’ve started planning to build an infrastructure that’s resilient to local failures in the next few years by isolating issues to a specific region, rather than letting them affect customers elsewhere.

— Victoria Puscas, Deliveroo

These events are best prevented at the architectural level, whether in the design phase or via the systems thinking and hacking projects our resilience engineering team takes on. We also work through simulations to explore prevention and mitigation strategies, including the tabletop exercises mentioned prior, as well as network simulations using in-house tools.

— Laura Thomson, Fastly

Just because an event is low-probability doesn’t mean you shouldn’t prepare for it. You should invest in contingency plans to stay ahead of any incident that might pop up. This ensures you’re taking care of customers regardless of what’s happening behind the scenes.

We continue to invest heavily in building reliable, secure and highly available services across our portfolio of products. This is table stakes for the cloud industry.

— Al Sene, DigitalOcean

For cloud region and data breaches, the first order is architecting and designing the system correctly, ensuring we have data encryption and data backups, and having stateless services driven through configuration. We ensure our roadmap and strategic planning prioritize strengthening our quality gates to mitigate incidents proactively with fast detection and response, chaos simulation, and cloud region and data security. Strengthening cross-functional teams, processes, and foundations isn’t necessarily exciting, but it’s key to innovation.

— Bhavini Soneji, Headspace