Develop Meaningful Service Metrics

• CIO
• IT VPs

• Align business/IT objectives (design top-down or outside-in)
• Significantly improve the relationship between the business and IT aspects of the organization
• Reinforce desirable actions and behaviors

• Service Level Managers
• Service Owners
• Program Owners

• Identify unusual deviations from the normal operating state
• Drive service improvement to maximize service value
• Validate the value of performance improvements while quantifying and demonstrating benefits realization

What are service metrics?

Service metrics measure IT services in a way that relates to a business outcome. IT needs to measure performance from the business perspective using business language.

To ensure the business cares about the metrics that IT produces, start with business needs to make sure you’re measuring the right things. This will give IT the opportunity talk to the right stakeholders and develop metrics that will meet their business needs.

Service metrics are designed with the business perspective in mind, so they are fully aligned with business objectives.

Different stakeholders will require different types of metrics. A CEO may require metrics that provide a snapshot of the critical success of the company while a business manager is more concerned about the performance metrics of their department.

Service metrics help IT communicate with the business in business terms and enables IT to articulate how and where they provide business value. Business stakeholders can also easily understand how IT services contribute to their success.

N=364

1. Too Many MetricsToo many metrics are provided and business leaders don’t know what to do with these metrics.
2. Metrics Are Too TechnicalIT provides technical metrics that are hard to relate to business needs, and methods of calculating metrics are not clearly understood, articulated, and agreed on.
3. Metrics Have No Business ValueService metrics are not mapped to business goals/objectives and they drive incorrect actions or spend.

1. Reinforce the wrong behaviorThe wrong metrics drive us-against-them, siloed thinking within IT, and meeting metric targets is prioritized over providing meaningful outcomes.
2. Do not reflect user experienceMetrics don’t align with actual business/user experience, reinforcing a poor view of IT services.
3. Effort ≠ ValueInvesting dedicated resources and effort to the achievement of the wrong metrics will only leave IT more constrained for other important initiatives.

For each business activity that supports the completion of a business process, define the success criteria that must be met in order to produce the desirable outcome.

Identify the IT services that are used by business stakeholders for each business activity. Measure the performance of these services from a business perspective to arrive at the appropriate service metrics.

Determine Business Success

Business metrics are derived from a pure business perspective. These are the metrics that the business stakeholders will measure themselves on, and business success is determined using these metrics.

Manage Service Value to the Business

Service metrics are used to measure IT service performance against business outcomes. These metrics, while relating to IT services, are presented in business terms and are tied to business goals.

Enable Operational Excellence

IT metrics are internal to the IT organization and used to manage IT service delivery. These metrics are technical, IT-specific, and drive action for IT. They are not presented to the business, and are not written in business language.

Business Relationship Management

BRM works to understand the goals and objectives of the business and inputs them into the design of the service metrics.

Service Metrics

BRM leverages service metrics to help IT organizations manage the relationship with the business.

BRM articulates and manages expectations and ensures IT services are meeting business requirements.

Service Level Management

SLM works with the business to understand service requirements, which are key inputs in designing the service metrics.

Service Metrics

SLM leverages service metrics in overseeing the day-to-day delivery of IT services. It ensures they are provided to meet expected service level targets and objectives.

Conventional Wisdom

Info-Tech Perspective

1

2

3

Target Outcome

Related Metrics

Procedure to validate the usefulness of IT metrics

# / % of service metrics added/removed per year

Target Outcome

Related Metrics

% of service metrics tied to business goals

% of service metrics reviewed on schedule

• 88% customer satisfaction rate
• 60% service profitability
• 15% increase in workforce productivity over the last 12 months

(Source: Aberdeen Group. “Service Benchmarking and Measurement.”)

The expected response time has not been achieved and this is visible in the service metrics. The reduced performance has been identified as having an impact of $250,000 per month in lost revenue potential.

The service metric drove an action to perform a root-cause analysis, which identified a network switch issue and drove a resolution action to fix the technology and architect redundancy to ensure continuity.

The fix eliminated the performance impact, allowing for recovery of the $250K per month in revenue, improved end-user confidence in the organization, and increased use of the application, creating additional revenue.

Industry: Manufacturing | Source: CIO interview and case material

The manufacturing business operates within numerous countries and requires a lot of coordination of functions and governance oversight. The company has monthly meetings, both regional and national, and key management and executives travel to attend and participate in the meetings.

While the meetings provide a lot of organizational value, the business has grown significantly and the cost of business travel has started to become prohibitive.

It was decided that only a few core meetings would require onsite face-to-face meetings, and for all other meetings, the company would look at alternative means. The face-to-face aspect of the meetings was still considered critical so they focused on options to retain that aspect.

The IT organization identified that they could provide a video conferencing service to meet the business need. The initiative was approved and rolled out in the organization.

Result:

IT service metrics needed to be designed to confirm that the expected value outcome of the implementation of video conferencing was achieved.

Under the direction of the CIO, the business goals and needs driving use of the service (i.e. reduction in travel costs, efficiency, no loss of positive outcome) were used to identify success criteria and key questions to confirm success.

With this information, the service manager was able to implement relevant service metrics in business language and confirmed an 80% adoption rate and a 95% success rate in term meetings running as expected and achieving core outcomes.

DIY Toolkit

Guided Implementation

Workshop

Consulting

Best-Practice Toolkit

1. Defining stakeholder needs for IT based on their success criteria
2. Derive meaningful service metrics based on identified IT services and validate with business stakeholders
3. Validate metrics can be collected and measured
4. Determine calculation methodology

1. Presentation format selected based on stakeholder needs and preference for information
2. Presentation format validated with stakeholders

1. Identify metrics that will be presented first to the stakeholders based on urgency or impact of the IT service
2. Determine the process to collect data, select initial targets, and integrate with SLM and BRM functions
3. Roll out the metrics implementation for a broader audience
4. Establish roles and timelines for metrics maintenance

Guided Implementations

• Design metrics based on business needs
• Validate the metrics

• Select presentation format
• Review metrics presentation design

• Select and implement pilot metrics
• Determine rollout process and establish maintenance/tracking mechanism

Onsite Workshop

• Meaningful service metrics designed from stakeholder needs

• Appropriate presentation format selected for each stakeholder

• Metrics implemented and process established to maintain and track program success

Design the Metrics

Determine Presentation Format and Implement Metrics

Gather Service Level Requirements

Monitor and Improve Service Levels

Activities

• 1.1 Determine stakeholder needs
• 1.2 Determine success criteria and key performance indicators
• 1.3 Derive metrics
• 1.4 Validate the metric collection

• 2.1 Discuss stakeholder needs/preference for data and select presentation format
• 2.2 Select and design the metric report

• Requirements
• 3.1 Determine the business requirements
• 3.2 Negotiate service levels
• 3.3 Align operational level agreements (OLAs) and supplier contracts

• 4.1 Conduct service report and perform service review
• 4.2 Communicate service review
• 4.3 Remediate issues using action plan
• 4.4 Proactive prevention

Deliverables

1. Metrics Development Workbook

1. Metrics Presentation Format Selection Guide
2. Metrics Tracking Tool

1. Service Level Management SOP
2. Service Level Agreement

1. Service Level Report
2. Service Level Review
3. Business Satisfaction Report

2.1

3.1

3.2

• Determine the stakeholder and their needs
• Identify IT services that are tied to the delivery of business outcomes
• Derive the service metrics

• For the selected metrics, identify the data source for collection
• Validate whether or not the data can be created
• Create a calculation method for the metrics

• Using the methodology provided, identify additional stakeholders and map out their success criteria, including KPIs to determine the appropriate service metrics

• Determine whether the designed metrics are measurable, and if so, how

• Metrics Development Workbook

• Metrics Development Workbook

Metrics Worksheet

Metrics Worksheet

Metrics Tracking Sheet

Defined IT Service Metrics

Thus, IT must start by identifying specific stakeholder needs. The more IT understands about the business, the more relevant the metrics will be to the business stakeholders.

1. Who are your stakeholders?
2. What are their goals and pain points?
3. What do the stakeholders need to know?
4. What do I need to measure?
5. Derive your service metrics

1. Who are the relevant stakeholders?
2. What are the goals and pain points of your stakeholders?
3. What do the stakeholders need to know?
4. What does IT need to measure?
5. What are the appropriate IT metrics?

Download the Metrics Development Workbook.

Who are your stakeholders?

1. Identify the primary stakeholders of your service metrics. Stakeholders are the people who have a very specific need to know about how IT services affect their business outcomes. Different stakeholders can have different perspective on the same IT service metric.Most often, the primary target of service metrics are the business stakeholders, e.g. VP of a business unit.
2. Identify any additional stakeholders. The CIO is also a stakeholder since they are effectively the business relationship manager for the senior leaders.

Video Conferencing Case Study
Manufacturing company

For this phase, we will demonstrate how to derive the service metrics by going through the steps in the methodology.

At a manufacturing company, the CIO’s main stakeholder is the CEO, whose chief concern is to improve the financial position of the company.

What are their goals and pain points?

1. Clearly identify each stakeholder’s business goals and outcomes. These would be particular business goals related to a specific business unit.
2. Identify particular pain points for each business unit to understand what is preventing them from achieving the desirable business outcome.

VC Case Study

One of the top initiatives identified by the company to improve financial performance was to reduce expense.

Because the company has several key locations in different states, company executives used to travel extensively to carry out meetings at each location.

Therefore, travel expenses represent a significant proportion of operational expenses and reducing travel costs is a key goal for the company’s executives.

What do the stakeholders need to know?

1. Identify the key things that the stakeholders would need to know based on the goals and pain points derived from the previous step.These are your success criteria and must be met to successfully achieve the desired goals.

VC Case Study

The CEO needs to have assurance that without executives traveling to each location, remote meetings can be as effective as in-person meetings.

These meetings must provide the same outcome and allow executives to collaborate and make similar strategic decisions without the onsite, physical presence.

Therefore, the success criteria are:

• Reduced travel costs
• Effective collaboration
• High-quality meetings

What does IT need to measure?

1. Identify the IT services that are leveraged to achieve the business goals and success criteria.
2. Identify the users of those services and determine the nature of usage for each group of users.
3. Identify the key indicators that must be measured for those services from an IT perspective.

VC Case Study

The IT department decides to implement the video conferencing service to reduce the number of onsite meetings. This technology would allow executives to meet remotely with both audio and video and is the best option to replicate a physical meeting.

The service is initially available to senior executives and will be rolled out to all internal users once the initial implementation is deemed successful.

To determine the success of the service, the following needs to be measured:

1. Outcomes of VC meetings
2. Quality of the VC meetings
3. Reduction in travel expenses

Derive your service metrics

1. Derive the service metrics that are meaningful to business stakeholders based on the IT services and the key indicators identified in the previous steps.
2. Distinguish between service metrics and business metrics. You may identify some business metrics in addition to the IT metrics, and although these are important, IT doesn’t own the process of tracking and reporting business metrics.

VC Case Study

In the previous step, IT identified that it must measure the outcomes of VC meetings, quality of the VC meetings, and the reduction in travel expenses. From these, the appropriate service metrics can be derived to answer the needs of the CEO.

IT needs to measure:

1. Percent of VC meetings successfully delivered
2. Growth of number of executive meetings conducted via VC

IT also identified the following business metrics:

1. Reduction in percent of travel expense/spend
2. Reduction in lost time due to travel

INSTRUCTIONS

1. Determine what data sources are available. Make sure that you know where the information you need is captured, or will need to be captured. This would include:
   • A ticket/request system
   • An auto discovery tool
   • A configuration management database ( CMDB)
2. Confirm that IT has the ability to collect the information.
   • If the necessary data is already contained in an identified data source, then you can proceed.
   • If not, consider whether it’s possible to gather the information using current sources and systems.
   • Understand the constraints and cost/ROI to implement new technology or revise processes and data gathering to produce the data.

VC Case Study

Using the metric derived from the video conferencing service example, IT wants to measure the % of VC meetings successfully delivered.

What are the data sources?

• Number of VC meetings that took place
• Number of service incidents
• User survey

INSTRUCTIONS

1. Understand your data requirements
   • To produce relevant metrics from your data, you need to ensure the level of quality and currency that provides you with useful information. You need to define:
     • The level of detail that has to be captured to make the data useful.
     • The consistency of the data, and how it needs to be entered or gathered.
     • The accuracy of the data. This includes how current the data needs to be, how quickly changes have to be made, and how data quality will be verified.

VC Case Study

Data requirement for percent of successful VC meetings:

• Level of detail – user category, location, date/time,
• Consistency – how efficiently are VC-related incidents opened and closed? Is the data collected and stored consistently?
• Accuracy – is the information entered accurately?

1. Develop the calculations that will be used for each accepted metric. The measurement needs to be clear and straightforward.
2. Define the scope and assumptions for each calculation, including:
   • The defined measurement period (e.g. monthly, weekly)
   • Exclusions (e.g. nonbusiness hours, during maintenance windows)

VC Case Study

Metric: Percent of VC meetings delivered successfully

IT is able to determine the total number of VC meetings that took place and the number of VC service requests to the help desk.

That makes it possible to use the following formula to determine the success percentage of the VC service:

((total # VC) – (# of VC with identified incidents)) / (total # VC) * 100

INSTRUCTIONS

• Scenario 1: Ad hoc remedial action and root-cause investigation. If the reason for the result is unknown, determining root cause or identifying trends is required to determine required actions.
• Scenario 2: Predefined remedial action. A set of predetermined actions associated with different results. This is useful when the meaning of the results is clear and points to specific issues within the environment.
• Scenario 3: Nonremedial action. The metrics may produce a result that reinforces or supports company direction and strategy, or identifies an opportunity that may drive a new initiative or idea.

VC Case Study

If the success rate of the VC meetings is below 90%, IT needs to focus on determining if there is a common cause and identify if this is a consistent downward trend.

A root-cause analysis is performed that identifies that network issues are causing difficulties, impacting the connection quality and usability of the VC service.

Industry: Manufacturing | Source: CIO

Standard IT metrics were technical and did not provide a business context that allowed for easy understanding of performance and decision making.

They discussed the goals for the meeting and defined the success criteria for those goals in the context of video conference meeting outcomes.

The success criteria that were discussed were then translated into a set of questions (key performance indicators) that if answered, would show that the success criteria were achieved.

The remaining metrics were identified as the possible service metrics, and the ability to gather the information and produce the metric was confirmed.

Service metrics were defined for:

1. Percent of video conference meetings delivered successfully
2. Growth in the number of executive meetings conducted via video conference

• To accelerate this project, engage your IT team in an Info-Tech workshop with an Info-Tech analyst team.
• Info-Tech analyst will join you and your team onsite at your location or welcome you to Info-Tech's historic Toronto office to participate in an innovative onsite workshop.
• Contact your account manager (www.infotech.com/account), or email Workshops@InfoTech.com for more information.

1.1

The onsite analyst will help you select key stakeholders and analyze their business objectives and current pain points.

1.2

The analyst will facilitate a discussion to uncover the information that these stakeholders care about. The group will also identify the IT services that are supporting these objectives.

1.5

Based on the key performance indicators obtained in the previous page, derive meaningful business metrics that are relevant to the stakeholders.

1.6

The analyst will help the workshop group determine whether the identified metrics can be collected and measured. If so, a calculation methodology is created.

1.7

Establish a feedback mechanism to have business stakeholders validate the meaningfulness of the metrics.

1.1

1.2

3.1

3.2

• Review the different format of metrics presentation and discuss the pros/cons of each format
• Discuss stakeholder needs/preference for data
• Select the presentation format

• Discuss stakeholder feedback based on selected presentation format
• Modify and adjust the presentation format as needed

• Design the metrics using the selected format

• Finalize the design for metrics presentation

• Metrics Presentation Format Selection Guide

• Metrics Presentation Format Selection Guide

Presentation Format Selection

Validated Service Reports

Typically web-based

Dynamic data that is updated in real time

Aggregates a lot of information into a single view

Presents metrics in a simplistic style that is well understood

Provides a quick point-in-time view of performance

Easy to consume visual presentation style

Complicated to set up well.
Requires additional technology support: programming, API, etc.

Promotes a short-term outlook – focus on now, no historical performance and no future trends. Doesn’t provide the whole picture and story.

Existing dashboard tools are often not customized enough to provide real value to each stakeholder.

Turns dry into attractive –transforms data into eye-catching visual memory that is easier to retain

Can be used as the intro to a formal report

There are endless types of infographics

Easily consumable

Easy to retain

Eye catching

Easily shared

Spurs conversation

Customizable

Require design expertise and resources

Can be time consuming to generate

Could be easily misinterpreted

Message can be lost with poor design

Metrics can be presented as a component of a periodic, formal report

A physical document that presents detailed information to a particular audience

More detailed, more structured and broader reporting period

Formal, shows IT has put in the effort

Effectively presents a broader and more complete story

Targets different stakeholders at the same time

Requires significant effort and resources

Higher risk if the report does not meet the expectation of the business stakeholder

Done at a specific time and only valuable for that specific time period

Harder to change format

Incorporates multiple metrics effectively.

Scores services against the most important organizational goals and objectives. Scorecards may tie back into strategy and different perspectives of success.

Quick view of performance against objectives

Measure against a set of consistent objectives

Easily consumable

Easy to retain

Requires a lot of forethought

1. Who is the relevant stakeholder?
2. What are the defined actions for the metric?
3. How frequently does the stakeholder need to see the metric?
4. How does the stakeholder like to receive information?

Industry: Manufacturing | Source: CIO Interview

The business had a clear need to understand if the implementation of video conferencing would allow previously onsite meetings to achieve the same level of effectiveness.

Provided reports had always been generated from an IT perspective and the business rarely used the information to make decisions.

The metrics needed to help the business understand if the meetings were remaining effective and be tied into the financial reporting against travel expenses, but there would be limited visibility during the executive meetings.

Approach

The service manager reviewed the information that he had gathered to confirm how often they needed information related to the service. He also met with the CIO to get some insight into the reports that were already being provided to the business, including the ones that were most effective.

Considerations

The conversations identified that there was no need for a dynamic real-time view of the performance of the service, since tracking of cost savings and utility would be viewed monthly and quarterly. They also identified that the item would be discussed within a very small window of time during the management meetings.

The Solution

It was determined that the best style of reporting for the metric was an existing scorecard that was produced monthly, using some infographics to ensure that the information is clear at a glance to enable quick decision making.

• To accelerate this project, engage your IT team in an Info-Tech workshop with an Info-Tech analyst team.
• Info-Tech analyst will join you and your team onsite at your location or welcome you to Info-Tech's historic Toronto office to participate in an innovative onsite workshop.
• Contact your account manager (www.infotech.com/account), or email Workshops@InfoTech.com for more information.

2.1

The onsite analyst will introduce the group to the communication vehicles of infographic, scorecard, formal report, and dashboard.

2.1

For selected stakeholders, the analyst will facilitate a discussion on how stakeholders would like to view information and how the metrics can be presented to aid decision making.

2.2

Based on the discussion, the working group will select the most appropriate presentation format and create a rough draft of how the report should look.

1.1

1.2

2.1

3.2

• Identify metrics that will be presented first to the stakeholders based on urgency or impact of the IT service
• Determine the process to collect data, select initial targets, and integrate with SLM and BRM functions

• Review the success of metrics and discuss feedback from stakeholders
• Roll out the metrics implementation to a broader audience
• Establish roles and timelines for metrics maintenance

• Document the first batch of metrics
• Document the baseline, initial targets
• Create a plan to integrate with SLM and BRM functions

• Create a document that defines how the organization will track and maintain the success of the metrics program
• Review the metrics program periodically

• Metrics Tracking Tool

• Metrics Tracking Tool

Metrics Tracking Tool

Metrics Tracking Tool

Metrics Tracking Tool

Active Service Metrics Program

1. Metrics Tracking Plan. Identify the metrics to be tracked and their purpose.
2. Metrics Tracking Actuals. Monitor and track the actual performance of the metrics.
3. Remediation Tracking. Determine and document the steps that need to be taken to correct a sub-performing metric.

A new metric is required or an existing metric needs large-scale changes (example: calculation method or scope).
Triggers metrics design as shown in phases 1 and 2.

A minor change is required to the presentation format or data. Note: a major change in a metric would be performed through the Add option.

The metric is no longer required, and it needs to be removed from reporting and data gathering. A final report date for that metric should be determined.

The metric is still useful and no changes are required to the metric, its measurement, or how it’s reported.

Industry: Manufacturing | Source: CIO Interview

Reviewing the value of active metrics

When the video conferencing service was initially implemented, it was performed as a pilot with a group of executives, and then expanded for use throughout the company. It was understood that prior to seeing the full benefit in cost reduction and increased efficiency and effectiveness, the rate of use and adoption had to be understood.

The primary service metrics created for the service were based on tracking the number of requests for video conference meetings that were received by the IT organization. This identified the growth in use and could be used in conjunction with financial metrics related to travel to help identify the impact of the service through its growth phase.

Once the service was adopted, this metric continued to be tracked but no longer showed growth or expanded adoption.

The service manager was no longer sure this needed to be tracked.

Key Activity

The metrics around requests for video conference meetings were reviewed at the annual metrics review meeting with the business. The service manager asked if the need for the metric, the goal of tracking adoption, was still important for the business.

The discussion identified that the adoption rate was over 80%, higher than anticipated, and that there was no value in continuing to track this metric.

Based on the discussion, the adoption metrics were discontinued and removed from data gathering and reporting, while a success rate metric was added (how many meetings ran successfully and without issue) to ensure the ongoing value of the video conferencing service.

• To accelerate this project, engage your IT team in an Info-Tech workshop with an Info-Tech analyst team.
• Info-Tech analyst will join you and your team onsite at your location or welcome you to Info-Tech's historic Toronto office to participate in an innovative onsite workshop.
• Contact your account manager (www.infotech.com/account), or email Workshops@InfoTech.com for more information.

3.1

The onsite analyst will help the workshop group select the metrics that should be first implemented based on the urgency and impact of these metrics.

3.2

The analyst will help the group create a process to gather data, measure baselines, and set initial targets.

3.5

The Info-Tech analyst will help the group establish a process to receive feedback from the business stakeholders once the report is generated.

3.6

The analyst will facilitate a discussion on how to implement the metrics program across the organization.

3.7

Set up a mechanism to ensure the success of the metrics program by assessing process adherence and process validity.