Benchmarking IT

‘It is not the strongest of the species that survives, nor the most intelligent, but the one that is most responsive to change.’ Charles Darwin

Benchmarking is the process of comparing one’s business processes and performance metrics to industry bests and/or best practices from other industries. Benchmarking involves management identifying the best firms in their industry, or any other industry where similar processes exist, and comparing the results and processes of those studied (the “targets”) to one’s own results and processes to learn how well the targets perform and, more importantly, how they do it.

I have been reading, The Real business of IT – How CIOs create and communicate value and as I was reading chapter 3, Show value for money, I thought to myself that I had the title for my next post. The chapter discusses, well, value for money and the importance of benchmarks, especially for CIOs who have just joined or are thinking of joining/moving to pastures anew.

Benchmarking an organisation’s IT is important whether conducted internally or externally. As the cost is quite high for conducting benchmarking via the established players, such as Gartner, many smaller organisations may initially decide to do it internally. Benchmarking has evolved now to the extent that even universities have started to run benchmarking courses, such as Stanford university’s IT benchmarking certificate, aimed at, yep, CIOs!

As quoted by; ‘in today’s business environment, says Bechtel CIO Geir Ramleth, IT needs to benchmark itself against a new set of peers: successful technology companies that built their IT systems in the Internet era. Doing so is a painful exercise for the ego. “Corporate IT is trying to break the sound barrier, and the Googles and Amazons are NOT supersonic. They’re hypersonic,” says Howard Rubin.’

My research has shown that Gartner has created a niche in IT benchmarking, as Gartner currently holds one of the largest global IT Trends and Benchmark Database. Dr Howard Rubin, created this global database and is a world authority on IT benchmarking and he offers the following thoughts and advice (Courtesy of Computer Aid Inc – CAI):

‘CAI: How do organizations interested in benchmarking best determine what they should be measuring and how they should be measuring?

Howard Rubin: I think the key thing for organizations is bi-directionality. That means your approach to benchmarking must come from both the top and from the bottom. From the top, you really have to understand your technology costs- the costs of your technology goods and services- almost as if you were a manufacturing company. You have to understand the cost structure of technology, what its impact is on your margin and what the impact of your technology investment is on growth, shrinkage and market share. And you have to integrate your understanding of the cost structure and performance structure of technology directly into the company’s financials.

You also have to figure out who you want to be looking at, in terms of comparisons. Is it direct peers or is it organizations that have a business performance structure that you aspire to meet? Another point I should make about the choice of measurements from the top is that there is this thing called the balance scorecard, in which people look at their finance measures, customer related measures, profit measures and organizational measures; but these are just static measures. That means that if a company’s strategic objective is to be the number one player within a given market, or to have the most comprehensive view of the customer, the balance scorecard isn’t going to cut it.

It is directional measures, as opposed to static measures, which will tell you where you are moving versus where you would like to be and what your corresponding rate of change is. And there are basically three kinds of directional measures: positional measures, directional measures, and velocity measures. In short, you need to be benchmarking where you are, where your targets are, how fast your organization is moving and how fast the world is changing. And all of this must be done within the context of strategy.

Approaching things from the bottom, you really have to understand a lot about technologies and about the technology organization itself. That means much more than just knowing how long it takes to develop an application, or the quality of your software, or the customer service component of your technology.

It means you need to look at technology as a commodity, at the unit costs. You need to be able to understand, almost like having a technology catalogue in front of you, what all of the technology components of your business consist of. What are your volumes? What are your unit costs? What are the costs to your competitors? What other alternatives are available out on the street in the open market?

And there are some other aspects, too. If you are a CFO, for instance, you really ought to understand where technology hits your P&L, where it impacts your salaries, your expense, and your depreciation. It is very important to understand how fixed or how variable your technology costs are.

Finally, there is a kind of ethereal dimension that sits on top of all of this, one which involves how well you are using technology to innovate and change your business, as compared to your competitors.

In the end, what companies really need is a full navigational system. Something that will give them the instrumentation to get them where they want to go, as well as the external calibration to see if someone is going to get there first, second, better, cheaper, or faster.

CAI: What are some of the major challenges that most organizations encounter when they first get started with measurements and benchmarking? What are some of the most common mistakes made? Do you have any caveats for organizations that are undertaking this for the first time?

Howard Rubin: When you first get started with benchmarking, and you haven’t done it before, you are basically going to be comparing data that you have internally with external data. Consequently, people will get their internal numbers and then they will get their external numbers and try to compare the two things right away. They will be looking for insights and conclusions and hypotheses. However, after the first round of benchmarking, you should really be making an effort not to look for insights and conclusions. You should be focused on rationalization. First time starters need to understand that rationalization is part of the benchmarking process. It is not a precursor to the process.

The other issue with first timers is the availability of data. It is very important to overcome the fact that you may not have a complete set of data available internally. This is always going to be an issue. Consequently, my recommendation is to look at your benchmark program as if it were a step function program: take a small core, build out, step up, sort of ratchet, take the key questions needed to answer the first, and have the benchmarking provider map your structure. You don’t need to do everything at once. You can build things up throughout the process.

A final caveat involves management by numbers. For example, you will find many large organizations that have gone through multiple mergers and that haven’t shed any of their redundant systems or redundant technologies. Certainly they can do better. But the path upwards is not going to be visible just by looking at the numbers. There may be a whole lot of other things that have to happen first. This is especially true if you are using benchmarking for internal target setting.

My brother is a really fine physician and he always advises his students not to look at the numbers but rather, to look at the patient. That’s an important caveat in benchmarking, too. The numbers will give you calibration. They will help you understand what side of the benchmark you may be on. But the goal is not to be better or worse than the benchmark. On either side of the benchmark, you can be learning how to improve your position.

CAI: You are known, among other things, for having collected and organized data into one of the world’s largest information technology databases. Could you give us more information about this repository? For example, what kinds of metrics get tracked? How broad is the technological and geographical representation?

Howard Rubin: The Worldwide IT Trend and Benchmark Database was really formalized in 1994. It was a project, as I mentioned before, which started out within the Canadian government. They were trying at the time to develop a global view of technology utilization in business.

In its current form, the Worldwide Benchmark Database maintains data on more than 10,000 large companies, each typically over 500 million dollars in revenue. It covers companies that are based across 100 countries, so it has a really massive geographic spread. There is also a large diversity of data, everything from basic business and IT spending data, to detailed data on technology platforms, programming languages, application development productivity, application quality, size and number of personnel, compensation, practices and processes, and process maturity. You will even find customer service related data.

The database is also updated continuously. We use internet based surveys for this as well as data collection mechanisms that originate from within our own consulting engagements. Consequently, we are able to keep the data fresh, on a daily basis, and we are able to update major trend levels on a quarterly basis. What that means is that if we see a major business or political change, we can sample thousands of companies within a 24 hour period to see if there is any movement. I don’t think anyone else in the world right now has the capability to determine, within 24 hours, the effect on business decision-making and technology that a world event may have.

You originally asked me about how benchmarking has changed over time. Traditionally, benchmarking has been used to compare current data to historical data. What we are seeing now with the worldwide benchmarking database, however, is the comparing of current data with current data. That’s an important development in my opinion because data is kind of like produce: it gets rotten after a very short period of time’.

An article in, said: ‘Two decades of research by Howard Rubin, president at Rubin Systems, reveals two key concepts that can enable CIOs to see whether their IT investments are really adding up. He found that measuring IT spend against two factors – operating expense and net revenue – is a more accurate gauge of IT effectiveness than the metric of measuring solely against net revenue.

In addition, Rubin discovered that enterprises spending slightly more than their peers tend to have better business results. But after a certain point, that extra spending does no good. Rubin calls the sweet spot of extra but not exorbitant spending “optimal IT intensity.” He calculates IT intensity by comparing the IT spend to both the operating expense and net revenue, and has developed IT intensity curves that help CIOs see if they are under-investing, investing an optimal amount or over-investing.’ Another good article, I recently read was Using Benchmarking Metrics to Uncover Best Practices and is worth reading if you want to embark on benchmarking your IT.

I would like to conclude with a quote from The Real business of IT – How CIOs create and communicate value – Randy Spratt, CIO, McKesson: ‘We opened up our finances and made them transparent. In mid 2006, we delivered a one line allocation to the business. Now we deliver a complete invoice. Between transparency, benchmarking, and competitive bid efforts, we have strengthened the view that our finances are under control, we’re driving to continual improvement on a per unit cost basis, and we hold ourselves accountable for delivering to service levels. “We don’t hear, ‘Why does IT cost so much?’ now. Do we still have expense level conversations? Yes, but they’re more about how we can jointly reduce costs.”

Further resources:

NCC IT Department Accreditation

NCC Benchmark Surveys

Benchmarking IT services

CIO Infrastructure Benchmark Assessment Tool

FREE IT Infrastructure Benchmark

Get a free instant benchmark of your SAP system

IT Benchmark Blog Blog


About mubbisherahmed
I am passionate about IT and its ability to deliver cost effective, value for money solutions that can enhance performance and in many cases provide competitive advantage by using a range of solutions and approaches in innovative ways.

6 Responses to Benchmarking IT

  1. mubbisherahmed says:

    Hamad Lone, CIO, offered the following feedback on a social media site:

    I have found that using external benchmarks can be misleading and cause extra problems , since no two organizations are the same. However, setting internal benchmarks using some external benchmarks as guidelines can help you improve performance and offer valuable feedback.

    I replied:

    Hamad, thanks for providing real world input towards my endeavours. I would agree on two fronts:

    1. Internal benchmarks can work just as well. The secret is to to then setup appropriate metrics to measure them and then to think about how the CIO can help the business achieve its objectives by providing a better service or assist in an area of business. Its the application that counts. For example a financial business, supermarket and online bookstore may all be using data mining but will use it differently. The financial house may use it to find out which products appeal to which segments of the market, the supermarket may use it to control, the supply chain better and the online bookstore may use it to push books to the customer based on their purchase history.

    2. Internal benchmarking is very cost effective and may return the same results if used correctly.

  2. mubbisherahmed says:

    Thai Ngo, Distributor, ERP Project Manager & VN IDS Leader provided the following feedback on a social media site:

    it’s a good topic.
    from what i observed in my company, we usually set our benchmark at a higher level in the industry (like cost – quality of service – …).

    I replied:

    Thai, I thought it was worth bringing it in for a discussion. It’s good to see that your company sets a higher threshold for benchmarking. Equally important is to ensure that once the benchmarking is done to think about how the CIO can help the business achieve its objectives by providing a better service or assist in an area of business. Its the application that counts. For example a financial business, supermarket and online bookstore may all be using data mining but will use it differently. The financial house may use it to find out which products appeal to which segments of the market, the supermarket may use it to control, the supply chain better and the online bookstore may use it to push books to the customer based on their purchase history.

  3. mubbisherahmed says:

    Jim Arnold, Corporate Marketing / Web 2.0 ROI Strategist / CMO at MetricsBoard / Author / Speaker / Open Networker added the following comment on a social media site:

    Mabbisher, great article on IT benchmarking. It is a valuable practice that should be used more often by the CIO.

    I replied:

    Jim, thanks for lending your support and I agree hence this article. It’s the analysis and eventual application that happens that assists the organisation and the CIO. If used correctly, it can move the CIO from a position of being misunderstood and under valued by his/her peers to a much valued peer. A peer who understands the business and helps it through better application of technology and emphasises IT’s contribution to the organisations’s ability to serve it better.

    Jim replied:

    We have a couple of blogs related to benchmarking if anyone is interested. We also have a free crowd-verified IT benchmark that people can take

  4. mubbisherahmed says:

    Ken Gorf, Director Finance & Legal, West Trax International provided the folowing feedback on a social media site:

    Hi Mubbisher, good question! Please take a look at the Benchmark KPI’s for SAP users at Hope this is useful input for your research, and of practical value to SAP users.

    I replied:

    Ken, thanks for forwarding your company’s link to me. It is relevant for SAP ERP and good to bump into another Brit on LI.

    I am going to add your site’s link to the ‘Further resources’ on the post ‘Benchmarking IT.’

  5. Rick Toma says:

    Mubbisher, I was texted to join this discussion and read your perspective with great interest. You are on spot with the reasons to benchmark, how to interpret data and the changing climate to which you get your data.

    Having been over 15 years in I/T services on the customer sale side, I have found a significant barrier to objectively “seeing where you are at”. Frankly put, it is a fear that your baby might just not be, nicely said, the most attractive baby in the bunch. Reasons and rationalization of the current state abound , to feed this fear. So, as you put it, it not about just knowing. That has little value. It’s about having the courage to face up to the unkown with the conviction and the muscle to do something about it, when it makes real business sense. But….that could lead to another whole topic.

    Great discussion!


    • mubbisherahmed says:

      Rick, thanks for joining the discussion. Yes, all too often organisations don’t quite know where they are in terms of their IT and don’t even take the small steps that can improve their IT. This is down to a variety of reasons ranging from autocratic CIOs who want to do things their way to organisation cultures where ‘fire fighting’ is so much the norm that the CIOs never get the chance to spend time on improving their IT.

      You are also absolutely correct that CIOs need to be courageous. Courage alone though can get them into strife. As you said, they need to be courageous when it makes real business sense. They need to ensure that the organisation can appreciate that real business sense through real IT value.

      That’s where benchmarking starts to deliver and make real contribution. Great to read such a thoughtful commment. Thanks, Rick.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: