Notes on the 2024 DORA Report

This is the 11th time DORA (acquired by Google in 2018) publishes their yearly Accelerate State of DevOps report. I went through the 100+ pages, and listed a few points that surprised me — surprised often not in the sense of radically new ideas, but more like earlier beliefs that are confirmed by the survey's data.

I’m structuring my notes according to the original report.

Artifical Intelligence

• Developers (especially security professionals, sysadmins and full-stack devs) relying on AI to help perform their core responsibilities are experiencing an increase in productivity, flow and job satisfaction.
• Surprisingly, these job satisfaction increases are paired with less time spent with valuable-, and more with toilsome work! I like the authors’ theory, saying that the productivity increase in valuable work means it’s done faster, and the freed-up time might need to be spent on more boring, not (yet) AI-assisted tasks. Still, respondents experience increased job satisfaction because it’s coming from seeing the end-user impact of their work - which is achieved regardless of AI assistance or not.
• Trust in AI-generated code is somewhat low, requiring further work before reaching production.
• On a team level, AI helps improve code quality and documentation - but also, hurts delivery throughput and stability! This seemingly contradictory finding might be attributed to batch sizes: AI can enable deploying bigger changesets in the same amount of time, which, as previous DORA reports showed, are more prone to create instability.
• While individuals, teams and organizations benefit from AI use, there are no corresponding benefits (yet?) in product performance.

Platform Engineering

• The positive impact of Internal Developer Platforms (that enable self-service workflows) on individual and team productivity is 8% and 10% respectively. This improvement even impacts overall organizational performance (+6%). The data also shows that these gains are smaller when feedback is not collected and used about the platform.
• The underlying reason of the success of internal developer platforms is the increased developer independence via self-service capabilities.
• While the impact of a dedicated platform team is negligible on individual productivity, on the team level, it brings a +6% gain compared to organizations with developer platforms but without a dedicated team owning it.
• While productivity increased, the study found that throughput suffers an 8% decrease with the use of these platforms. Reasons could be found around the complexity these systems introduce, and the way they are obligatory to use for even non-ideal scenarios. To combat this, again, a user-centered approach in platform development can help.
• The report found a surprising 14% decrease in change stability (change failure rate and the recently introduce rate of rework metric), whic can be explained by increased throughput combined with the trust that the platform will allow errors to be remediated quickly, increasing the tolerance of instability.
• Another weird finding is the higher burnout in organizations using internal developer platforms. Here, lack of a more plausible explanation, maybe the causation is reversed: those organizations might create developer platforms who want to mitigate their already existing burnout.

Developer Experience

• Stable priorities and focusing on the end-user experience increase productivity, product quality, and developer satisfaction — and decrease burnout. I love that there’s data behind this gut feeling!
• Focusing on the end-user experience brings clear priorities, increased sense of purpose, and better cross-functional collaboration.
• Product performance is increased the most when the focus on the users is paired with a strong culture of documentation. (My comment: this is where fully distributed companies have another advantage: they have no choice but maintain a mature documentation culture due to their async work!)
• The survey showed that frequently changing organization priorities slightly decrease productivity, and substantially increase burnout. What’s even worse is that performing well in other areas like strong leadership, good internal documents and a user-centered approach still cannot offset this risk on employee well-being that unstable priorities mean. (Note: It’s not the change itself that’s problematic, but the frequency of it, causing chronic stress.)
• On the other hand, stable priorities correlated with lower software delivery performance (in both speed and stability)! Maybe organizations with mature products and stable priorities ship code less frequently and in larger chunks.

Other notes

• Transformative leadership positively impacts team, product and organizational performance, productivity, job satisfaction and burnout metrics (this is the second highest impact area!).
• Cloud infra only increases organizational performance if the approaches, processes, and technologies are also changed. Only moving from a data center to the cloud can even hurt organizational performance without these changes.

I recommend spending an hour or two going through the entire report, I'm pretty sure there are other areas that might surprise you.