We heard it straight from one seasoned CTO’s mouth, who said, “this isn’t conjecture, I’ve been told very specifically – [the engineering team is] strongly averse to any type of metrics evaluation or estimations. I’d say about 70% of them have fallen on the wrong side of a spreadsheet at some point in time in a previous job. Historically people have been bitten by some effectively random industry norms that have been assigned for a team.”
We hear time and time again that engineers have a visceral reaction to overly metrics-driven engineering orgs. Not only do engineers hate it, but management does too because they want to keep their teams happy.
With every other function normalizing, or even happily accepting quantitative measurement, why has engineering remained outside of this practice?
The case against engineering metrics
Metrics can be misleading
Software engineering is a combo of technical execution, designing for scale, collaboration across functions (design, product, sales, etc), decisions, and communication between people. There are too many factors that contribute to the success or failure of a project. Metrics can provide a simplified view of a project’s progress or quality without capturing the full picture. There isn’t yet a way to capture all of the complexity here across the spectrum of people, processes, and technical components.
Metrics can be misused
Metrics are often used as a way to measure individual or team performance, which can create a culture of fear and competition that can be counterproductive to collaboration and creativity. One CTO took a clear stance on this, “I’m not interested in nor will I tolerate any notion of adopting metrics evaluation for personal performance because that’s not the intent of understanding data.”
Metrics can be arbitrary
Determining what metrics to use and how to measure them can be difficult, and can lead to metrics that are arbitrary or not relevant to the project or team. “One of the reasons that I left my last job is they ended up hiring a whole bunch of new C-level folks who were overexcited about metrics. I agreed mostly with the engineering bias [against metrics]. Unless the valuation of the company is specifically tied to my hitting my estimated date, we’re all just sitting around wasting a lot of time asking questions about whether I missed a Monday or a Tuesday. Engineers will basically ask: ‘why is it important? If you tell me the company is going to fall off a cliff because I missed my Wednesday deadline, I’ll make the damn Wednesday deadline. But if this is just an exercise, then get off my back. Let me write the code.’ That’s what I hear from the engineering side.” Engineers can sense urgency for urgency’s sake, and the back and forth around the metrics can be distracting to actually getting work done.
The case for engineering metrics
Understanding the hesitations around metrics can help leaders have open discussions with their teams about the risks and benefits. The counters to the above are that a metrics-driven culture can open doors to a new level of productivity and flow across the team.
Data lets you investigate:
One leader described how data enables him have better conversations cross-functionally. “Data to me provides an opportunity to ask smarter questions. Without data, what I tend to see is a lot of this: ‘we’re moving well.’ I think there are a lot of organizations where that’s okay. I think for me if you have the data, then you get into what I think is the proper conversation, which is how the construction and the operations of this squad is working cross-functionally.”
Data gives you the assurance that things are going well
Not all data points to something wrong, someone not doing their job, or something going awry. Engineering metrics can also just help leaders who aren’t in the day-to-day gain visibility and assurance that things are going as planned. “Data provides me the opportunity to sometimes ask super stupid questions that I get really insightful, great answers that allows me to then say, well, then that’s not a problem. I get to move over here.”
What data have you found is beneficial to how your team functions, and what data have you found does more harm than good?
