A friend of mine manages one of those large home improvement stores that are in most communities. He told me that they are bombarded with metrics from “headquarters” judging their performance and a lot of the metrics don’t make sense. He said “headquarters” will decide to push a particular product and set up product displays at the front of the store. The store personnel are instructed to push the products in the display. If his store doesn’t sell the displayed product well, according to his metric target, he is essentially punished as it affects his performance rating.
My friend solves this problem by purchasing some of the displayed product himself and then returning it to one of the other stores across town to get his money back. He always makes his metric goal and it gives him the opportunity to chat with a fellow store manager while returning his items. Yes, the other store managers do the same thing. My friend is happy, "headquarters" thinks their happy, and a lot of time and money is wasted. Brian L. Joiner stated:
“When people are pressured to meet a target value there are three ways they can proceed. 1. They can work to improve the system. 2. They can distort the system. 3. They can distort the data."
I am a strong believer in the use of data and metrics. I also believe they must be used with careful judgment and in context. Too many times organizations set a metric in order to judge performance and by doing so create unproductive behavior like the kind my friend displayed by distorting the system or the data. When a target value is not met, the first response should be to seek an understanding as to why, based on the context, but a lot of organizations skip the understanding step and jump straight to criticism and punishment, essentially guaranteeing metric manipulation. This is a major contributor as to why some leaders don't want to report status for a project as yellow or red for the overall status metric and then everyone is "surprised" when the real status finally comes out.
I was attending my first meeting in a new organization and the contractor for this organization was presenting the metric data that was supposed to be reflective of their performance. They were using control charts to show the data. After looking at a few charts I asked what their control limits were. They said three standard deviations above and three below the center line (average). I said “no they are not.” They said “yes they are.” Being younger at this point in my career, and less diplomatic (unlearned), I said “no they are not” and they said “yes they are” after we went through this small child method of problem resolution a few more times, my director said get with James after the meeting.
After the meeting I went to talk to them about their control limits and no one wanted to talk. Finally, one of them said as the others looked down, "the control limits are not plus or minus three sigma. They are plus or minus five sigma." They had altered the control limits so their process appeared to be more stable than they actually were to avoid management attention and this had been going on for years. Metric manipulation!
I don’t proclaim to be a genius at analyzing data but I can look at data on a control chart and quickly identify the basic forms of unnatural process variation and so can you. This is easy and quickly learned. In the aforementioned example, all of the contractors data points for the process were close to the mean. In a properly constructed control chart, some of the points should occur away from the mean.
All of the important processes in your organization should have control charts for the key process indicators for the process.
If you do not have control charts for your processes, you have money flying out the window. You are heating the house with the front door open. If you do not have an understanding of process variation and/or Sick Sigma™, then a must read book as a starting point is Understanding Variation: the Key to Managing Chaos by David J. Wheeler. Wheeler calls control charts process variation charts.
In the second edition the Lessons on page 112-113, What You Should Do Starting Tomorrow on page 118-119 and Wheeler’s end of chapter summaries provide the best quick overview I have encountered as most books make it too abstract/complex for practitioners that don’t have a mathematical background. I love Wheeler's quote in the book of Myron Tribus:
"Managing a company by means of the monthly report is like trying to drive a car by watching the yellow line in the rear view mirror"
We should use data and metrics to manage. However we must use judgment in context to minimize metric manipulation and have an understanding of the basic principles of process variation. Always question data to ensure it was collected correctly and is being interpreted correctly, otherwise it could be creating more problems than it is solving.