April 1, 2024

Missing dimensions in development ops & productivity metrics

by Mark Greene


Engineering Managers have a variety of metrics they can draw from to assist in understanding their teams' efficiency & operational performance. What’s missing is a precise measurement of Risk (i.e. complexity) and Resilience (i.e. mitigation effort).

These dimensions rightfully categorize changes so that macro level metrics like DORA can be acted upon with precision. 

Shepherdly Metrics Explained

Shepherdly introduces a novel approach to measuring the risk associated with pull requests and the mitigations necessary to ensure resilience. At its core, a predictive model is used to assess the risk of introducing bugs through code changes. 

The model calculates a Risk Score, which then determines the Resilience Coverage needed—essentially, the mitigations required to guard against potential bugs. This approach not only quantifies the risk but also guides developers on specific actions to enhance codebase resilience.

Comparative Analysis

Risk Management and Mitigation

While Shepherdly zeroes in on the risk at the pull request level, providing actionable insights for mitigation, DORA offers a more macro perspective. DORA's Change Failure Rate, for example, assesses the overall risk posture of the deployment process but does not offer granular insights into individual code changes or categorizes change failure rate by risk.

Resilience and Reliability

Shepherdly's Resilience Coverage actively promotes proactive mitigation where it’s needed, encouraging developers to implement best practices directly in the PR. In contrast, DORA's Time to Restore Service focuses on the team's ability to respond to and recover from incidents, highlighting operational resilience rather than change resilience.

Efficiency and Productivity

Since most changes are objectively lower risk, there is justification to treat these differently in the review process by accelerating their deployment. While DORA offers insight into Deployment Frequency and Lead Time for Changes, Shepherdly disambiguates the driving factors by its risk classification.

Furthermore, by measuring the resilience of a change, teams can isolate which areas of their development process are contributing to Change Defect Rate.

Strategic Decision-Making and Visibility

By capturing innate risk and resilience (a proxy for effort), teams can have a nuanced understanding of how to balance speed & resilience before changes are shipped. For teams seeking deeper automation, the desired balance can be codified per repository.

DORA metrics guide broader strategic decisions around DevOps practices and investments, providing visibility into the health and performance of the software delivery lifecycle.

Main Takeaway

Having a broad understanding is great but typically leads to questions that require deeper analysis in order to act. Shepherdly is the tool that allows you to identify the main drivers behind DORA metric trends.

Table Of Contents

A. IdentifiersContact details, such as real name, alias, postal address, telephone or mobile contact number, unique personal identifier, online identifier, Internet Protocol address, email address, and account nameYES
B. Personal information categories listed in the California Customer Records statuteName, contact information, education, employment, employment history, and financial informationNO
C. Protected classification characteristics under California or federal lawGender and date of birthNO
D. Commercial informationTransaction information, purchase history, financial details, and payment informationNO
E. Biometric informationFingerprints and voiceprintsNO
F. Internet or other similar network activityBrowsing history, search history, online behavior, interest data, and interactions with our and other websites, applications, systems, and advertisementsNO
G. Geolocation dataDevice location
H. Audio, electronic, visual, thermal, olfactory, or similar informationImages and audio, video or call recordings created in connection with our business activitiesNO
I. Professional or employment-related informationBusiness contact details in order to provide you our Services at a business level or job title, work history, and professional qualifications if you apply for a job with usNO
J. Education InformationStudent records and directory informationNO
K. Inferences drawn from other personal informationInferences drawn from any of the collected personal information listed above to create a profile or summary about, for example, an individual’s preferences and characteristicsNO
L. Sensitive Personal InformationNO