Long comment from http://agilepainrelief.com/notesfromatooluser/2010/02/misuse-of-velocity-in-agile-projects.html has turned into a post here.
Scrum and Agile is something I'm particularly interested in and enthused about. But it is strange how many people throw the terms around or even claim to work on Agile teams without a background in the reasons and common applications of Agile. I'm not saying you have to do everything in your Agile book, but at least read one!
Some of these comments make my eyes bulge out a little bit. I think even the most elementary book on scrum makes it really clear why we use story points and why they are the basis of all sprint planning.
Now granted, this is *really* hard to use when you are responsible to a client and have to invoice weekly with the # of hours of work performed and estimate ahead of time what they will get for X. Time and Materials or fixed bid, client's have expectations. But outside of that, if you can get a client who is happy to use agile that basically means they hire X full-time people for N number of days - and hope to fullfil part of an agreed upon roadmap.
In that case, the point is that you will never have the fastest, most accurate, most consistent, perfect engineers. However, you will always have customers - internal or external. And the hope is that you can at least set realistic expectations with the customers which your engineers have ownership over meeting. How do you do that? You let your engineers estimate what they can accomplish, and you check that estimate against what they said they could do in the past. That's what velocity is about. Not holding people accountable, but letting people hold themselves accountable.
Think of it as pacing yourself in a marathon against your average time, not trying to race others. You can't race others because:
a). This ain't a marathon and there are a 100 finish lines
b). This ain't a marathon, more like a decathlon and some teams will excel in certain areas and flop in others.
Any measuring you do should be sujective, not quantitative.
Since estimation is subjective, comparing quantitatively between teams is useless.
Give smart people tools to do their job, communicate with customers and get out of the way!
You could measure estimation ability by seeing how often a team meets their targets. But then, you are also measuring the amount of crap their customers make them deal with, company politics, internet connectivity, coffee quality, etc :) Just let good engineers get to work. If you're paying attention, you'll know is performing and who is committed to performance.