Statline article from The Quality Magazine, 3(3), June 1994, p92

Doug Shaw

The practice of benchmarking is currently very much in vogue. Poorly executed, it supports the maxim ``Comparisons are odious''. Properly executed and interpreted, it is a very powerful tool for an organization pursuing Quality goals.

Proper execution requires that two questions be asked.

1. Are we comparing like with like?
2. Does the comparison tell us anything about how to improve our process?

Benchmarking often starts as an exercise in comparing numbers. A company, UVW Inc., obtains (perhaps through an industry association, perhaps through an informal network) some performance data on a key process. For example, ``Our competitor, XYZ Pty Ltd, is filling 95% of orders within 15 days''. UVW Inc. then looks at its own performance and finds that it takes 23 days to fill 95% of its orders.

Management may react to this comparison by exhorting UVW Inc. staff to reduce the time to fill 95% of orders to 15 days or less, thinking that they have evidence that this target is achievable.

But are they comparing like with like?

Perhaps XYZ Pty Ltd handles a slightly different range of products, services a market with different characteristics, or uses a different definition of the term ``fill an order''.

When benchmarking against processes in other organizations, we have to be very careful that we know what our data represent, and that we know how our benchmarking partners arrived at their data.

Some examples:

• An Australian manufacturing company compared its labour productivity (tonnes per hour worked) with that of a US company inthe same business, and felt somewhat depressed.Then they found that, while they had included ancillary staff in their manhours (reasoning that production was not possible without maintenance staff, administrative staff, and so on), their US counterpart had included only direct production staff.
• Comparisons of reject rates in manufacturing require great care. An Australian company found its reject rates to be much higher than an Austrian company making the same product. When they imported a pallet of the Austrian product, their workers all said ``We couldn't pass that stuff for dispatch!''

Benchmarking should be more than comparing like with like so that productivity targets can be set. It should not just be about comparing numbers, it should be about comparing processes. UVW Inc. can most obviously benefit from the benchmarking exercise if the comparison provides some insight into how XYZ Pty Ltd is able to fill most orders within 15 days, so that they might apply similar ideas.

In general, answering the question ``Does the comparison tell us anything about how to improve our process?'' requires cooperation from the other company. Companies have most to gain from benchmarking if they are part of a network of ``benchmarking partners'' and allow sharing of process information. This implies that they should expect to contribute to the benchmarking partnership, as well as extracting gain.

Companies should be careful in deciding which organizations and processes they include in their benchmarking. There's no point to benchmarking against processes clearly not as good as your own, and benchmarking against organizations vastly superior in performance is likely to show that the processes are not comparable.

Benchmarking, done properly, is a major exercise. It needs to be restricted to processes which are critical to business success. It should provide input to the organization's strategic planning so that objectives can be set, plans made and resources allocated to meet the goal of continued competitiveness.

Comparing numbers is useless unless you are comparing like with like and thinking about how to improve your process.