Measuring Eng Impact

This became Measures of engineering impact.

At today’s engineering leader breakfast (an extremely virtual breakfast, these days), we chatted about measuring engineering impact, which is quite different from measuring engineering velocity, even though velocity is a major input into impact.

Velocity metrics are the sort of things that you probably want to use to understand rather than to grade outcomes. Impact metrics should be things that you can grade against performance.

A few fascinating notes on how other companies do this:

• Amazon: # press releases
• Square: # new billable features
• Gusto: # competitive advantages created, # competitive accelerants, # table stake features

At Calm I’ve been thinking about this too. We’ve been using:

• # big bets, these are new, differentiated features
• # experiments (with a target distribution for winning, neutral, and losing outcomes)
• # fires (not trying to drive this to zero, rather learning from its fluxuations)
• 1-2 technical investments: we don’t want to incentivize numerous tech-driven technical changes, but we also want to avoid starving out innovation, so we specify a fixed target

This has been working out pretty well so far.