James Gregory has an entry up called Metrics, on metrics as used
to measure the performance of your software product: is it as fast as we need,
as stable as we need, as tested as we need? He then goes on to talk about
measuring the performance of your coders:
There's other metrics you should watch as well: the deficit between a
programmer's estimate on time to fix a bug and actual time taken is honestly
the only way to start getting good time-estimates. Which is another important
point: metrics don't just let you improve your product now, they help your
staff get better at their jobs.
Similarly, it's imperfect, but trends from time-lines of bugs open, and
rates of bugs opened per unit time show interesting trends. I've found these to
be good indicators of project completeness in the past, but these metrics are
dangerous: it's easy to be misled by this stuff, so treat such metrics with
suitably large salt-grains.
He goes on to note Joel Spolsky's objections to measuring programmer
performance like this, which are in Spolsky's article The Econ 101 Management
Method. James says:
The example [Spolsky] gave was in reference to his company's
bug-tracking software, and he submitted that once you can measure the
bugs-closed per unit time value, programmers will start artificially inflating
their count by lying to the bug-tracker and to you. I frankly find this
position a bit offensive, but if such situations exist it reflects poorly on
management rather than the engineers. You need to be accepting of metrics that
show problems, because if people hide the problems, you can't solve them, but
similarly: if you can't measure, or, don't look for the problems, you can't
solve them either.
Spolsky has an argument from psychology too, which I'll get to, but first I
thought it was worth noting that while James who is (let's stipulate) a good
manager needs to work with, optimise around or fire bad engineers occasionally,
we as (let's stipulate) good engineers need to defend ourselves against bad
managers. I can come up with all kinds of amusing or horrifying stories about
managers who've gotten bees in their bonnet about various metrics or other
'signs' that a project is going well or badly, or that they're being undermined
or are about to get a bonus or whatever. For every manager who is trying to
notice patterns like every time you tell me a bug fix is going to take a
day, it actually takes a month
there's one who is roaming around reminding
people triumphantly that all version control commit messages must have at least
100 words in them. [I should note that this example is hypothetical: no actual
crazy managers were harmed in the making of this blog entry. That said, I'm
sure there's some manager out there doing this right now.]
That's a bad manager!
isn't much consolation when you work under said
manager.
Anyway, as far as it goes, I suppose Spolsky's point is that programmer
performance is unlikely to be measured by a single metric any more than product
quality is in James's post, with increases in speed needing to be balanced
against security, stability and maintainability metrics. But, if you
do make the mistake of relying on a single metric for product quality, as James
notes, you get a very fast piece of software which barely works. That's one
thing. But Spolsky is arguing that if you rely on a single metric for
programmer quality, you not only get say, a low bug count at the expense of
tests, you also get worse programmers:
Intrinsic motivation is your own, natural desire to do things well...
Extrinsic motivation is a motivation that comes from outside, like when you’re
paid to achieve something specific.
Intrinsic motivation is much stronger than extrinsic motivation. People work
much harder at things that they actually want to do. That’s not very
controversial.
But when you offer people money to do things that they wanted to do, anyway,
they suffer from something called the Overjustification Effect. “I must be
writing bug-free code because I like the money I get for it,” they think, and
the extrinsic motivation displaces the intrinsic motivation. Since extrinsic
motivation is a much weaker effect, the net result is that you’ve actually
reduced their desire to do a good job.
So Spolsky has a point beyond you've got to be careful that you don't
mistake the metric for what it's measuring,
although that's part of his
point. This is a point that scientific method training hammers into you: you
need metrics to help you decide what reality is doing, but the metric is
not the same thing as reality, it is only ever an approximation to it, and
is only good insofar as it is a useful approximation.
Spolsky's other point is that regardless of whether you, the
stipulated good manager, mistake the metric for reality, the people judged
based on the metric will: they'll even start mistaking the metric for their
conscience. This, I think, is a point James hasn't addressed yet.