How Good is Your Benchmarking?

With a recovery in the offing, a company's executives want to make sure they can fill open positions in a timely manner. They discover that for a given occupation in a company their size in their industry, it takes an average of two weeks to fill a job opening, but they are taking three weeks. What does that mean?

"The knee-jerk reaction is that we're not being very efficient. It is taking us way too long to bring in these people," says Jay Jamrog, senior vice president of research for the Institute for Corporate Productivity. "But you have to do some more research. Are we bringing in a better-quality person by taking more time? Does this person come up the learning curve faster? Are they productive faster? Do they stay with the organization longer? Do they exhibit high potential?"

Jamrog's example illustrates just one of the major pitfalls of benchmarking -- failing to clearly understand why you are conducting the study. "Companies measure too much," says Jamrog. "They are overloaded with numbers and so they don't do the 'so-what' test. It is not about the numbers; it is about the story they tell. The story is the impact on the organization."

Of course, there are plenty of stories where benchmarking has had a profound impact on a company. Perhaps the most celebrated is Xerox's Leadership Through Quality effort in the 1980s after the company's leaders discovered that Japanese manufacturers could sell copiers for the same price as Xerox spent just manufacturing its machines. With the help of benchmarking and other initiatives, Xerox cut its manufacturing costs by 50% while slashing inventory costs and greatly improving parts quality.

Last year, the Global Benchmarking Network (GBN) commissioned a study of benchmarking practices. More than 450 organizations participated. While the study showed the primary benefit of benchmarking is improved process performance and that 20% of the organizations reported an average financial benefit of $250,000 from a benchmarking project, it also found some major concerns.

"Worryingly, there are a sizeable proportion of organizations (approximately 30%) that are using Best Practice Benchmarking and not obtaining the full benefits. This appears to be because many organizations have not been trained in benchmarking, do not follow a proven benchmarking methodology or use a benchmarking code of conduct, and are not using standard project management practices to manage their benchmarking projects," GBN noted. "In these circumstances it is not surprising they are not obtaining the full benefits."

Best in Class

Benchmarking studies can be formal or informal, large or small, but the gold standard for these studies compares performance to best in class. Cindy Jutras, a senior vice president and research fellow at Aberdeen Group, said her firm always determines best in class for its studies. "We use a competitive framework that divides all survey respondents into best in class, industry average and laggard. We define best in class as the top 20% of aggregate performance. You have to set the bar to compare things against. Without that, it may be good research but it probably isn't a benchmark."

Jutras said it is critical to define the terms of best in class correctly. She noted that new analysts sometimes fall into the trap of making a best-in-class determination entirely on degree of improvement. "My analogy is, it is very easy to lose 20 pounds if you weigh 300 pounds. It is very difficult to lose 20 pounds if you weigh 100 pounds," she explains. "If all you do is make best-in-class metrics based on improvement, then people who are very good and shooting for the last few points of improvement will automatically be laggards."

Perspective is an important aspect of any benchmarking exercise. Dick Van Liew, a senior consultant with ABB Reliability Services, has participated in 150 benchmark studies in various industries. "Not everybody wants to be at the top of the heap for reliability practices. If you are an airline, you want to be very near the top because when you have an incident, people die. If you are making candy and have an incident, you ruin a batch," he says. The business question to consider, he notes, is determining if pursuing a change in practice will result in a meaningful payback.

Like many benchmarking professionals, Van Liew recommends that companies include firms outside their own industry when seeking best-in-class practices. "One main driver for benchmarks against other industries is that you don't find best practices in any one industry segment. If you are really interested in best practices, you want to find the best of the best," he says. "By including a wide group in your benchmarking, you start picking up more of the best practices."

Doug Olander, a senior director for BBK with more than 25 years of experience in manufacturing and performance improvement, recalled his experience with the greenfield start-up of an automotive plastics plant. The industry average for its blow molding technology process was 700 ppm of quality delivered and the best in class for North America was 400 ppm. "We set our goal at 10 times better than best in class, or 40 ppm, and we achieved 16 ppm the first year," he says. "That gave us the basis for competition." Olander's team then faced the question of what goal to set next. They knew that injection-molding operations were more advanced, so they reset their goal against that technology. "In year two, when we reached single digits, we were now the best plastic molder from a quality perspective."

Too Much Change?

Best-in-class benchmarking can lead to profound change in an organization. That change may require reorganizing people's roles and responsibilities, introducing new targets against which they will be measured, and major investments in technology, processes and people. For such changes, say experts, it is vital to get executive buy-in to support the change process.

"The farther you delegate benchmarking, the more likely you are to end up justifying the current state," says Olander. "It has to be done at a level where there is enough self-confidence to be able to say there are flaws here. My experience is that the farther down the organization, the more threatening that communication becomes."

Best-in-class benchmarking also can test an organization's commitment to change. As the GBN study reported, all too many companies find out what is needed to be the best and then back away for a variety of reasons. "Inside many organizations, benchmarking falls by the wayside, not because they wouldn't benefit from it but because they don't have the intestinal fortitude to make the changes," observes David Caruso, vice president of manufacturing at Endeca Technologies Inc., a search applications company. But he adds, for those companies that take the plunge, benchmarking is "a speedometer for creating competitive advantage."