As a product manager (PdM), you have a lot going on and it’s not your job to manage development or ops/SRE/platform engineering. That said, it’s useful to be clued in about the work of your colleagues.
DORA (DevOps Research & Assessment Team) is a group of industry analysts who first started benchmarking product team performance in 2013. They focus specifically on the efficacy of what we now call ‘DevOps’: the combined processes of how a team goes from design to code with a focus. In particular, DevOps practice concerns itself with how good a team is at making small, incremental changes to their applications. One of the prominent DORA members, Jez Humble, published a book called ‘Continuous Delivery’ in 2010, which was foundational for the DevOps movement. You can find the latest version of their report here: DORA Report.
If you’re new to digital product development, skimming the report might help give you a sense of how your colleagues in the above areas define success for themselves and the challenges they’re working on to get there.
Here are five quick things about the report I think every PdM should know.
1. 2012-2014 was Big for DevOps
Prior to the first version of the report, DevOps was much more of a niche community, heavily concentrated in Silicon Valley and other large centers of general practice. Before DevOps, the conventional wisdom in digital engineering was that you should avoid releasing new code a lot because it was risky and disruptive. The DORA reports were a kind of coming out vehicle for DevOps because they delivered a surprising and highly provocative inference: with these new practice you could both release more often and have less downtime.
2. There are Four DORA Metrics
- Deployment Frequency (DF)
• How often code is deployed to production.
• High performers deploy more frequently (daily or on-demand). - Lead Time for Changes (LT)
• The time from code commit to code running in production.
• Shorter lead times indicate faster delivery and iteration. - Change Failure Rate (CFR)
• The percentage of deployments that cause a failure (e.g., outage, rollback, degraded service).
• Lower is better; elite teams keep this below 15%. - Mean Time to Recovery (MTTR)
• How long it takes to restore service after a failure in production.
• High performers recover in under an hour.
3. Performance Still Varies Enormously between Firms
As late as 2024, the report still shows enormous differences in product team performance. This excerpt shows the difference between the lowest performing cohort and the highest.
source: DORA Report (2024)
4. It Uses Clusters
While some question parts of their methods, the report basically clusters firms and then looks at differences in their performance. The graph below shows some of the metrics across their four clusters.
This table shows the same thing, but with more explicit description.
5. You Can Help!
For the time it took you to notionally understand this, you’ve already become a better, more understanding PdM. Most development and platform teams are trying to work toward these goals and probably know about the DORA metrics. As a PdM, you can help them by:
- Putting fewer things into the product pipeline, but expecting better results.
- Leading with a testable point of view on how a given user experience should work (which is central to Hypothesis-Driven Development).
