Secure IT/OT Convergence

Create a holistic IT/OT security culture.

Analyst Perspective

Are you ready for secure IT/OT convergence?

IT/OT convergence is less of a convergence and more of a migration. The previously entirely separate OT ecosystem is migrating into the IT ecosystem, primarily to improve access via connectivity and to leverage other standard IT capabilities for economic benefit.

In the past, OT systems were engineered to be air gapped, relying on physical protection and with little or no security in design, (e.g. OT protocols without confidentiality properties). However, now, OT has become dependent on the IT capabilities of the organization, thus OT inherits IT’s security issues, that is, OT is becoming more vulnerable to attack from outside the system. IT/OT convergence is complex because the culture, policies, and rules of IT are quite foreign to OT processes such as change management, and the culture, policies, and rules of OT are likewise foreign to IT processes.

A secure IT/OT convergence can be conceived of as a negotiation of a strong treaty between two systems: IT and OT. The essential initial step is to begin with communication between IT and OT, followed by necessary components such as governing and managing OT security priorities and accountabilities, converging security controls between IT and OT environments, assuring compliance with regulations and standards, and establishing metrics for OT security.

Ida Siahaan Research Director, Security and Privacy Practice Info-Tech Research Group

Executive Summary

Info-Tech Insight

Returning to isolated OT is not beneficial for the organization, so IT and OT need to learn to collaborate, starting with communication to build trust and to overcome their differences. Next, negotiation is needed on components such as governance and management, security controls on OT environments, compliance with regulations and standards, and establishing metrics for OT security.

Consequences of unsecure IT/OT convergence

OT systems were built with no or little security design 90% of organizations that use OT experienced a security incident. (Fortinet, 2021. Ponemon, 2019.)

(Source: Fortinet, 2021.)

Lack of visibility 86% of OT security-related service engagements lack complete visibility of OT network in 2021 (90% in 2020, 81% in 2019). (Source: “Cybersecurity Year In Review” Dragos, 2022.)

The need for secure IT/OT convergence

Important Industrial Control System (ICS) cyber incidents

2000 Target: Australian sewage plant. Method: Insider attack. Impact: 265,000 gallons of untreated sewage released.	2012 Target: Middle East energy companies. Method: Shamoon. Impact: Overwritten Windows-based systems files.	2014 Target: German Steel Mill. Method: Spear-phishing. Impact: Blast furnace failed to shut down.	2017 Target: Middle East safety instrumented system (SIS). Method: TRISIS/TRITON. Impact: Modified SIS ladder logic.	2022 Target: Viasat’s KA-SAT network. Method: AcidRain. Impact: Significant loss of communication for the Ukrainian military, which relied on Viasat’s services.
1903 Target: Marconi wireless telegraph presentation. Method: Morse code. Impact: Fake message sent “Rats, rats, rats, rats. There was a young fellow of Italy, Who diddled the public quite prettily.”	2010 Target: Iranian uranium enrichment plant. Method: Stuxnet. Impact: Compromised programmable logic controllers (PLCs).	2013 Target: ICS supply chain. Method: Havex. Impact: Remote Access Trojan (RAT) collected information and uploaded data to command-and-control (C&C) servers	2016 Target: Ukrainian power grid. Method: BlackEnergy. Impact: For 1-6 hours, power outages for 230,000 consumers.	2021 Target: Colonial Pipeline. Method: DarkSide ransomware. Impact: Compromised billing infrastructure halted the pipeline operation.

(Source: US Department of Energy, 2018.

”Significant Cyber Incidents,” CSIS, 2022

MIT Technology Review, 2022.)

Info-Tech Insight

Most OT incidents start with attacks against IT networks and then move laterally into the OT environment. Therefore, converging IT and OT security will help protect the entire organization.

The Secure IT/OT Convergence Framework

Phases Plan Enhance Monitor & Optimize

Plan Outcomes Mapping business goals to IT/OT security goals RACI chart for priorities and accountabilities Compliance obligations register Readiness checklist Enhance Outcomes Security strategy for IT/OT convergence Risk management framework Security policies & procedures IR, DR, BCP Monitor & Optimize Outcomes Security awareness and training Security architecture and controls

Plan Benefits Improved flexibility and less divided IT/OT Improved compliance Enhance Benefits Increased strategic common goals Increased efficiency and versatility Monitor & Optimize Benefits Enhanced security Reduced costs

Plan

Initiate communication

Map organizational goals to IT/OT security goals

Input: Corporate, IT, and OT strategies

Output: Your goals for the security strategy

Materials: Secure IT/OT Convergence Requirements Gathering Tool

Participants: Executive leadership, OT leader, IT leader, Security leader, Compliance, Legal, Risk management

1. As a group, brainstorm organization goals.
   1. Review relevant corporate, IT, and OT strategies.
2. Record the most important business goals in the Secure IT/OT Convergence Requirements Gathering Tool. Try to limit the number of business goals to no more than 10 goals. This limitation will be critical to helping focus on your secure IT/OT convergence.
3. For each goal, identify one to two security alignment goals. These should be objectives for the security strategy that will support the identified organization goals.

Download the Secure IT/OT Convergence Requirements Gathering Tool

Record organizational goals

Refer to the Secure IT/OT Convergence Framework when filling in the following elements.

1. Record your identified organization goals in the Goals Cascade tab of the Secure IT/OT Convergence Requirements Gathering Tool.
2. For each of your organizational goals, identify IT alignment goals.
3. For each of your organizational goals, identify OT alignment goals.
4. For each of your organizational goals, select one to two IT/OT security alignment goals from the drop-down lists.

Establish scope and boundaries

It is important to know at the outset of the strategy: What are we trying to secure in IT/OT convergence ?
This includes physical areas we are responsible for, types of data we care about, and departments or IT/OT systems we are responsible for.

Record scope and boundaries

Refer to the Secure IT/OT Convergence Framework when filling in the following elements: Record your security-related organizational scope, physical location scope, IT systems scope, and OT systems scope in the Scope tab of the Secure IT/OT Convergence Requirements Gathering Tool. For each item scoped, give the rationale for including it in the comments column. Careful attention should be paid to any elements that are not in scope.

Plan

Define roles and responsibilities

Input: List of relevant stakeholders

Output: Roles and responsibilities for the secure IT/OT convergence program

Materials: Secure IT/OT Convergence RACI Chart Tool

Participants: Executive leadership, OT leader, IT leader, Security leader

There are many factors that impact an organization’s level of effectiveness as it relates to IT/OT convergence. How the two groups interact, what skill sets exist, the level of clarity around roles and responsibilities, and the degree of executive support and alignment are only a few. Thus, it is imperative in the planning phase to identify stakeholders who are:

• Responsible: The people who do the work to accomplish the activity; they have been tasked with completing the activity and/or getting a decision made.
• Accountable: The person who is accountable for the completion of the activity. Ideally, this is a single person and will often be an executive or program sponsor.
• Consulted: The people who provide information. This is usually several people, typically called subject matter experts (SMEs).
• Informed: The people who are updated on progress. These are resources that are affected by the outcome of the activities and need to be kept up to date.

Download the Secure IT/OT Convergence RACI Chart Tool

Define RACI Chart

Plan

Establish governance and build cross-functional team

To establish governance and build an IT/OT cross-functional team, it is important to understand the operation of OT systems and their interactions with IT within the organization, e.g. ad hoc, centralized, decentralized.

Info-Tech Insight

To determine IT/OT convergence maturity level, Info-Tech provides the IT/OT Convergence Self-Evaluation Tool.

Centralized security governance model example

Plan

Identify convergence elements and compliance obligations

Identify compliance obligations

Source: ENISA, 2013 DHS, 2009.

OT system has compliance obligations with industry regulations and security standards/regulations/guidelines. See the lists given. The lists are not exhaustive. OT system owner can use the standards/regulations/guidelines as a benchmark to determine and manage the security level provided by third parties. It is important to understand the various frameworks and to adhere to the appropriate compliance obligations, e.g. IEC/ISA 62443 - Security for Industrial Automation and Control Systems Series.

IEC/ISA 62443 - Security for Industrial Automation and Control Systems Series

(Source: Cooksley, 2021)

IEC/ISA 62443 is a comprehensive international series of standards covering security for ICS systems, which recognizes three roles, namely: asset owner, system integrator, and product manufacturer. In IEC/ISA 62443, requirements flow from the asset owner to the product manufacturer, while solutions flow in the opposite direction. For the asset owner who owns and operates a system, IEC 62443-2 enables defining target security level with reference to a threat level and using the standard as a benchmark to determine the current security level. For the system integrator, IEC 62443-3 assists to evaluate the asset owner’s requirements to create a system design. IEC 62443-3 also provides a method for verification that components provided by the product manufacturer are securely developed and support the functionality required.

Record your compliance obligations

Refer to the “Goals Cascade” tab of the Secure IT/OT Convergence Requirements Gathering Tool. Identify your compliance obligations. Most organizations have compliance obligations that must be adhered to. These can include both mandatory and voluntary obligations. Mandatory obligations include: Laws Government regulations Industry standards Contractual agreements Voluntary obligations include standards that the organization has chosen to follow for best practices and any obligations that are required to maintain certifications. Organizations will have many different compliance obligations. For the purposes of your secure IT/OT convergence, include only those that have OT security requirements. Record your compliance obligations, along with any notes, in your copy of the Secure IT/OT Convergence Requirements Gathering Tool. Refer to the “Compliance DB” tab for lists of standards/regulations/guidelines.

Plan

Assess readiness

Readiness checklist for secure IT/OT convergence

(Source: “Grid Modernization: Optimize Opportunities And Minimize Risks,” Info-Tech)

Enhance

Update security strategy

To update security strategy, it is important to actively encourage visible sponsorship across management and to provide regular updates.

(Source: NIST SP 800-82 Rev.3, “Guide to Operational Technology (OT) Security,” NIST, 2022.)

OT system life cycle is like the IT system life cycle, starting with architectural design and ending with decommissioning. Currently, IT only gets involved from installation or maintenance, so they may not fully understand the OT system. Therefore, if OT security is compromised, the same personnel who commissioned the OT system (e.g. engineering, electrical, and maintenance specialists) must be involved. Thus, it is important to have the IT team collaborate with the OT team in each stage of the OT system’s life cycle. Finally, it is necessary to have propositional sharing of responsibilities between IT leaders, security leaders, and OT leaders who have broader responsibilities.

Enhance

Update risk management framework

The need for asset and threat taxonomy

One of issues in IT/OT convergence is that OT systems focus on production, so IT solutions like security patching or updates may deteriorate a machine or take a machine offline and may not be applicable. For example, some facilities run with reliability of 99.999%, which only allows maximum of 5 minutes and 35 seconds or less of downtime per year. Managing risks requires an understanding of the assets and threats for IT/OT systems. Having a taxonomy of the assets and the threats cand help. Applying normal IT solutions to mitigate security risks may not be applicable in an OT environment, e.g. running an antivirus tool on OT system may remove essential OT operations files. Thus, this approach must be avoided; instead, systems must be rebuilt from golden images.

(Source: ENISA, 2018.)

Enhance

Update security policies and procedures

Policy hierarchy example

This example of a policy hierarchy features templates from Info-Tech’s Develop and Deploy Security Policies and Identify the Best Framework for Your Security Policies research.

Enhance

Update IR, DR, and BCP

A proactive approach to security is important, so actions such as updating and testing the incident response plan for OT are a must. (“Cybersecurity Year In Review” Dragos, 2022.)

1. Customize organizational chart for IT/OT IR, DR, BCP based on governance and management model.
   E.g. ad hoc, internal distributed, internal centralized, combined distributed, and decentralized. (Software Engineering Institute, 2003)
2. Adjust the authority of the new organizational chart and decide if it requires additional staffing.
   E.g. full authority, shared authority. (Software Engineering Institute, 2003)
3. Update IR plan, DR plan, and BCP for IT/OT convergence.
   E.g. incorporate zero trust principles for converge network
4. Testing updated IR plan, DR plan, and BCP.

Optimize

Implement awareness, induction, and cross-training

Awareness may not correspond to readiness

Certifications

One of issues in certification is the complexity on relevancy in topics with respect to roles and levels. An example solution is the European Union Agency for Cybersecurity (ENISA)’s approach to analyzing existing certifications by orientation, scope, and supporting bodies, grouped into specific certifications, relevant certifications, and safety certifications. Specific cybersecurity certification of ICS/SCADA Example: ISA-99/IEC 62443 Cybersecurity Certificate Program, GIAC Global Industrial Cyber Security Professional (GICSP), Certified SCADA Security Architect (CSSA), EC-Council ICS/SCADA Cybersecurity Training Course. Other relevant certification schemes Example: Network and Information Security (NIS) Driving License, ISA Certified Automation Professional (CAP), Industrial Security Professional Certification (NCMS-ISP). Safety Certifications Example: Board of Certified Safety Professionals (BCSP), European Network of Safety and Health Professional Organisations (ENSHPO).

(Source: ENISA, 2015.)

Optimize

Design and deploy converging security architecture and controls

IT/OT convergence architecture can be modeled as a layered structure based on security. In this structure, the bottom layer is referred as “OT High-Security Zone” and the topmost layer is “IT Low-Security Zone.” In this model, each layer has its own set of controls configured and acts like an additional layer of security for the zone underneath it. The data flows from the “OT High-Security Zone” to the topmost layer, the “IT Low-Security Zone,” and the traffic must be verified to pass to another zone based on the need-to-know principle. In the normal control flow within the “OT High-Security Zone” from level 3 to level 0, the traffic must be verified to pass to another level based on the principle of least privilege. Remote access (dotted arrow) is allowed under strict access control and change control based on the zero-trust principle with clear segmentation and a point for disconnection between the “OT High-Security Zone” and the “OT Low-Security Zone” This model simplifies the security process, as if the lower layers have been compromised, then the compromise can be confined on that layer, and it also prevents lateral movement as access is always verified.

(Source: “Purdue Enterprise Reference Architecture (PERA) model,” ISA-99.)

Off-the-shelf solutions

Getting the right recipe: What criteria to consider?

Visibility and Asset Management Passive data monitoring using various protocol layers, active queries to devices, or parsing configuration files of OT, IoT, and IT environments on assets, processes, and connectivity paths. Threat Detection, Mitigation, and Response (+ Hunting) Automation of threat analysis (signature-based, specification-based, anomaly-based, sandboxing) not only in IT but also in relevant environments, e.g. IoT, IIoT, and OT on assets, data, network, and orchestration with threat intelligence sharing and analytics. Risk Assessment and Vulnerability Management Risk scoring approach (qualitative, quantitative) based on variables such as behavioral patterns and geolocation. Patching and vulnerability management. Usability, Architecture, Cost The user and administrative experience, multiple deployment options and extensive integration capabilities, and affordability.

Optimize

Establish and monitor IT/OT security metrics for effectiveness and efficiency

Further information

Related Info-Tech Research

Build an Information Security Strategy Info-Tech has developed a highly effective approach to building an information security strategy – an approach that has been successfully tested and refined for over seven years with hundreds of organizations. This unique approach includes tools for ensuring alignment with business objectives, assessing organizational risk and stakeholder expectations, enabling a comprehensive current-state assessment, prioritizing initiatives, and building a security roadmap.

Preparing for Technology Convergence in Manufacturing Information technology (IT) and operational technology (OT) teams have a long history of misalignment and poor communication. Stakeholder expectations and technology convergence create the need to leave the past behind and build a culture of collaboration.

Implement a Security Governance and Management Program Your security governance and management program needs to be aligned with business goals to be effective. This approach also helps provide a starting point to develop a realistic governance and management program. This project will guide you through the process of implementing and monitoring a security governance and management program that prioritizes security while keeping costs to a minimum.

Bibliography

Bibliography

Research Contributors and Experts

Jeff Campbell Manager, Technology Shared Services Horizon Power, AU Jeff Campbell has more than 20 years' experience in information security, having worked in both private and government organizations in education, finance, and utilities sectors. Having focused on developing and implementing information security programs and controls, Jeff is tasked with enabling Horizon Power to capitalize on IoT opportunities while maintaining the core security basics of confidentiality, integrity and availability. As Horizon Power leads the energy transition and moves to become a digital utility, Jeff ensures the security architecture that supports these services provides safer and more reliable automation infrastructures.

Christopher Harrington Chief Technology Officer (CTO) Carolinas Telco Federal Credit Union Frank DePaola Vice President, Chief Information Security Officer (CISO) Enpro Kwasi Boakye-Boateng Cybersecurity Researcher Canadian Institute for Cybersecurity

Build a Security Metrics Program to Drive Maturity

Good metrics come from good goals.

ANALYST PERSPECTIVE

Metrics are a maturity driver.

"Metrics programs tend to fall into two groups: non-existent and unhelpful.

The reason so many security professionals struggle to develop a meaningful metrics program is because they are unsure of what to measure or why.

The truth is, for metrics to be useful, they need to be tied to something you care about – a state you are trying to achieve. In other words, some kind of goal. Used this way, metrics act as the scoreboard, letting you know if you’re making progress towards your goals, and thus, boosting your overall maturity."

– Logan Rohde, Research Analyst, Security Practice Info-Tech Research Group

Executive summary

Situation

• Many security leaders put off adding metrics to their program because they don't know where to start or how to assess what is worth measuring.

Complication

• Sometimes, this uncertainty causes the belief that their security programs are not mature enough for metrics to be worthwhile.
• Because metrics can become very technical and precise, it's easy to think they're inherently complicated (not true).

Resolution

• A metric, really, is just a measure of success against a given goal. Gradually, programs will achieve their goals and set new, more specific goals, and with them comes more specific metrics.
• It is not necessary to jump into highly technical metrics right away. A lot can be gained from metrics that track behaviors.
• A metrics program can be very simple and still effectively demonstrate the value of security to the organization. The key is to link your metrics to the goals or objectives the security team is pursuing, even if they are simple implementation plans (e.g. percentage of departments that have received security training).

Info-Tech Insight

1. Metrics lead to maturity, not vice versa
   • Tracking metrics helps you assess progress and regress in your security program. This helps you quantify the maturity gains you’ve made and continue to make informed strategic decisions.
2. The best metrics are tied to goals
   • Tying your metrics to goals ensures that you are collecting metrics for a specific purpose rather than just to watch the numbers change.

Our understanding of the problem

This Research is Designed For:

• CISO

This Research Will Help You:

• Understand the value of metrics.
• Right-size a metrics program based on your organization’s maturity and risk profile.
• Tie metrics to goals to create meaningful KPIs.
• Develop strategies to effectively communicate the right metrics to stakeholders.

This Research Will Also Assist:

• CIO
• Security Manager
• Business Professionals

This Research Will Help Them:

• Become informed on the metrics that matter to them.
• Understand that investment in security is an investment in the business.
• Feel confident in the progress of the organization’s security strategy.

Info-Tech’s framework integrates several best practices to create a best-of-breed security framework

Information Security Framework

Governance

• Context and Leadership
  • Information Security Charter
  • Information Security Organizational Structure
  • Culture and Awareness
• Evaluation and Direction
  • Security Risk Management
  • Security Policies
  • Security Strategy and Communication
• Compliance, Audit, and Review
  • Security Compliance Management
  • External Security Audit
  • Internal Security Audit
  • Management Review of Security

Management

• Prevention
  • Identity Security
    • Identity and Access Management
  • Data Security
    • Hardware Asset Management
    • Data Security & Privacy
  • Infrastructure Security
    • Network Security
    • Endpoint Security
    • Malicious Code
    • Application Security
    • Vulnerability Management
    • Cryptography Management
    • Physical Security
    • Cloud Security
  • HR Security
    • HR Security
  • Change and Support
    • Configuration and Change Management
    • Vendor Management
• Detection
  • Security Threat Detection
  • Log and Event Management
• Response and Recovery
  • Security Incident Management
  • Information Security in BCM
  • Security eDiscovery and Forensics
  • Backup and Recovery
• Measurement
  • Metrics Program
  • Continuous Improvement

Metrics help to improve security-business alignment

While business leaders are now taking a greater interest in cybersecurity, alignment between the two groups still has room for improvement.

Key statistics show that just...

5% of public companies feel very confident that they are properly secured against a cyberattack.

41% of boards take on cybersecurity directly rather than allocating it to another body (e.g. audit committee).

19% of private companies do not discuss cybersecurity with the board.

(ISACA, 2018)

Info-Tech Insight

Metrics help to level the playing field

Poor alignment between security and the business often stems from difficulties with explaining how security objectives support business goals, which is ultimately a communication problem.

However, metrics help to facilitate these conversations, as long as the metrics are expressed in practical, relatable terms.

Security metrics benefit the business

Executives get just as much out of management metrics as the people running them.

1. Metrics assuage executives’ fears
   • Metrics help executives (and security leaders) feel more at ease with where the company is security-wise. Metrics help identify areas for improvement and gaps in the organization’s security posture that can be filled. A good metrics program will help identify deficiencies in most areas, even outside the security program, helping to identify what work needs to be done to reduce risk and increase the security posture of the organization.
2. Metrics answer executives’ questions
   • Numbers either help ease confusion or signify other areas for improvement. Offering quantifiable evidence, in a language that the business can understand, offers better understanding and insight into the information security program. Metrics also help educate on types of threats, staff needed for security, and budget needs to decrease risk based on management’s threat tolerance. Metrics help make an organization more transparent, prepared, and knowledgeable.
3. Metrics help to continually prove security’s worth
   • Traditionally, the security team has had to fight for a seat at the executive table, with little to no way to communicate with the business. However, the new trend is that the security team is now being invited before they have even asked to join. This trend allows the security team to better communicate on the organization’s security posture, describe threats and vulnerabilities, present a “plan of action,” and get a pulse on the organization’s risk tolerance.

Common myths make security metrics seem challenging

Security professionals have the perception that metrics programs are difficult to create. However, this attitude usually stems from one of the following myths. In reality, security metrics are much simpler than they seem at first, and they usually help resolve existing challenges rather than create new ones.

Tie your metrics to goals to make them worthwhile

SMART metrics are really SMART goals.

Specific

Measurable

Achievable

Realistic

Timebound

Achievable: What is an achievable metric?

When we say that a metric is “achievable,” we imply that it is tied to a goal of some kind – the thing we want to achieve.

How do we set a goal?

1. Determine what outcome you are trying to achieve.
   • This can be small or large (e.g. I want to determine what existing systems can provide metrics, or I want a 90% pass rate on our monthly phishing tests).
2. Decide what indicates that you’ve achieved your goal.
   • At what point would you be satisfied with the progress made on the initiative(s) you’re working on? What conditions would indicate victory for you and allow you to move on to another goal?
3. Develop a key performance indicator (KPI) to measure progress towards that goal.
   • Now that you’ve defined what you’re trying to achieve, find a way to indicate progress in relative or relational terms (e.g. percentage change from last quarter, percentage of implementation completed, ratio of programs in place to those still needing implementation).

Info-Tech’s security metrics methodology is repeatable and iterative to help boost maturity

Security Metric Lifecycle

Start:

Review current state and decide on priorities.

Set a SMART goal for improvement.

Develop an appropriate KPI.

Use KPI to monitor program improvement.

Present metrics to the board.

Revise metrics if necessary.

Metrics go hand in hand with your security strategy

A security strategy is ultimately a large goal-setting exercise. You begin by determining your current maturity and how mature you need to be across all areas of information security, i.e. completing a gap analysis.

As such, linking your metrics program to your security strategy is a great way to get your metrics program up and running – but it’s not the only way.

Check out the following Info-Tech resource to get started today:

Build an Information Security Strategy

The value of security metrics goes beyond simply increasing security

This blueprint applies to you whether you need to develop a metrics program from scratch or optimize and update your current strategy.

Value of engaging in security metrics:

• Increased visibility into your operations.
• Improved accountability.
• Better communication with executives as a result of having hard evidence of security performance.
• Improved security posture through better understanding of what is working and what isn’t within the security program.

Value of Info-Tech’s security metrics blueprint:

• Doesn’t overwhelm you and allows you to focus on determining the metrics you need to worry about now without pressuring you to do it all at once.
• Helps you develop a growth plan as your organization and metrics program mature, so you continue to optimize.
• Creates effective communication. Prepares you to present the metrics that truly matter to executives rather than confusing them with unnecessary data. Pay attention to metric accuracy and reproducibility. No management wants inconsistent reporting.

Impact

Short term: Streamline your program. Based on your organization’s specific requirements and risk profile, figure out which metrics are best for now while also planning for future metrics as your organization matures.

Long term: Once the program is in place, improvements will come with increased visibility into operations. Investments in security will be encouraged when more evidence is available to executives, contributing to overall improved security posture. Potential opportunities for eventual cost savings also exist as there is more informed security spending and fewer incidents.

Info-Tech offers various levels of support to best suit your needs

DIY Toolkit

“Our team has already made this critical project a priority, and we have the time and capability, but some guidance along the way would be helpful.”

Guided Implementation

“Our team knows that we need to fix a process, but we need assistance to determine where to focus. Some check-ins along the way would help keep us on track.”

Workshop

“We need to hit the ground running and get this project kicked-off immediately. Our team has the ability to take this over once we get a framework and strategy in place.”

Consulting

“Our team does not have the time or the knowledge to take this project on. We need assistance through the entirety of this project.”

Diagnostics and consistent frameworks used throughout all four options

Link Security Metrics to Goals to Boost Maturity – Project Overview

Workshop overview

Contact your account representative or email Workshops@InfoTech.com for more information.

Phase 1

Link Security Metrics to Goals to Boost Maturity

Phase 1

1.1 Review current state and set your goals

1.2 Develop KPIs and prioritize your goals

1.3 Implement and monitor KPIs

This phase will walk you through the following activities:

• Current state assessment
• Setting SMART goals
• KPI development
• Goals prioritization
• KPI implementation

This phase involves the following participants:

• Security Team

Outcomes of this phase

• Goals-based KPIs
• Security Metrics Determination and Tracking Tool

Phase 1 outline

Complete these steps on your own or call us to complete a guided implementation. A guided implementation is a series of two to three advisory calls that help you execute each phase of a project. They are included in most advisory memberships.

Guided Implementation 1: Link Security Metrics to Goals to Boost Maturity

Proposed Time to Completion: 2-4 weeks

Step 1.1: Setting Goals

Start with an analyst kick-off call:

• Determine current and target maturity for various security programs.
• Develop SMART Goals.

Then complete these activities…

• CMMI Assessment

Step 1.2 – 1.3: KPI Development

Review findings with analyst:

• Prioritize goals
• Develop KPIs to track progress on goals
• Track associated metrics

Then complete these activities…

• KPI Development

With these tools & templates:

• KPI Development Worksheet
• Security Metrics Determination and Tracking Tool

Phase 1 Results & Insights:

• Basic Metrics program

1.1 Review current state and set your goals

120 minutes

Let’s put the security program under the microscope.

Before program improvement can take place, it is necessary to look at where things are at presently (in terms of maturity) and where we need to get them to.

In other words, we need to perform a security program gap analysis.

Info-Tech Best Practice

The most thorough way of performing this gap analysis is by completing Info-Tech’s Build an Information Security Strategy blueprint, as it will provide you with a prioritized list of initiatives to boost your security program maturity.

Completing an abbreviated gap analysis...

• Security Areas
• Network Security
• Endpoint Security
• Vulnerability Management
• Identity Access Management
• Incident Management
• Training & Awareness
• Compliance, Audit, & Review
• Risk Management
• Business Alignment & Governance
• Data Security

1. Using the CMMI scale on the next slide, assess your maturity level across the security areas to the left, giving your program a score from 1-5. Record your assessment on a whiteboard.
2. Zone in on your areas of greatest concern and choose 3 to 5 areas to prioritize for improvement.
3. Set a SMART goal for improvement, using the criteria on goals slides.

Use the CMMI scale to contextualize your current maturity

Use the Capability Maturity Model Integration (CMMI) scale below to help you understand your current level of maturity across the various areas of your security program.

1. Initial
   • Incident can be managed. Outcomes are unpredictable due to lack of a standard operating procedure.
2. Repeatable
   • Process in place, but not formally implemented or consistently applied. Outcomes improve but still lack predictability.
3. Defined
   • Process is formalized and consistently applied. Outcomes become more predictable, due to consistent handling procedure.
4. Managed
   • Process shows signs of maturity and can be tracked via metrics. Moving towards a predictive approach to incident management.
5. Optimizing
   • Process reaches a fully reliable level, though improvements still possible. Regularity allows for process to be automated.

(Adapted from the “CMMI Institute Maturity Model”)

Base your goals around the five types of metrics

Choose goals that make sense – even if they seem simple.

The most effective metrics programs are personalized to reflect the goals of the security team and the business they work for. Using goals-based metrics allows you to make incremental improvements that can be measured and reported on, which makes program maturation a natural process.

Info-Tech Best Practice

Before setting a SMART goal, take a moment to consider your maturity for each security area, and which metric type you need to collect first, before moving to more ambitious goals.

Security Areas

• Network Security
• Endpoint Security
• Vulnerability Management
• Identity Access Management
• Incident Management
• Training & Awareness
• Compliance, Audit & Review
• Risk Management
• Business Alignment & Governance
• Data Security

Set SMART goals for your security program

Specific

Measurable

Achievable

Realistic

Timebound

Now that you have determined which security areas you’d like to improve, decide on a goal that meets the SMART criteria.

Examples of possible goals for various maturity levels:

1. Perform initial probe to determine number of systems capable of providing metrics by the end of the week.
2. Take baseline measurements each month for three months to determine organization’s baseline state.
3. Implement a vulnerability management program to improve baseline state by the end of the quarter.
4. Improve deployment of critical patches by applying 90% of them within the set window by the end of the year.
5. Demonstrate how vulnerability management affects broad organizational trends at quarterly report to senior leadership.

Compare the bolded text in these examples with the metric types on the previous slide

Record and assess your goals in the Security Metrics Determination and Tracking Tool

1.1 Security Metrics Determination and Tracking Tool

Use tab “2. Identify Security Goals” to document and assess your goals.

To increase visibility into the cost, effort, and value of any given goal, assess them using the following criteria:

• Initial Cost
• Ongoing Cost
• Initial Staffing
• Ongoing Staffing
• Alignment w/Business
• Benefit

Use the calculated Cost/Effort Rating, Benefit Rating, and Difference Score later in this project to help with goal prioritization.

Info-Tech Best Practice

If you have already completed a security strategy with Info-Tech resources, this work may likely have already been done. Consult your Information Security Program Gap Analysis Tool from the Build an Information Security Strategy research.

1.2 Develop KPIs and prioritize your goals

There are two paths to success.

At this time, it is necessary to evaluate the priorities of your security program.

Option 1: Progress to KPI Development

• If you would like practice developing KPIs for multiple goals to get used to the process, move to KPI development and then assess which goals you can pursue now based on resources available, saving the rest for later.

Option 2: Progress to Prioritization of Goals

• If you are already comfortable with KPI development and do not wish to create extras for later use, then prioritize your goals first and then develop KPIs for them.

Phase 1 Schematic

• Gap Analysis
• Set SMART Goals (You are here.)
  • Develop KPIs
• Prioritize Goals
• Implement KPI & Monitor
• Phase 2

Develop a key performance indicator (KPI)

Find out if you’re meeting your goals.

Terms like “key performance indicator” may make this development practice seem more complicated than it really is. A KPI is just a single metric used to measure success towards a goal. In relational terms (i.e. as a percentage, ratio, etc.) to give it context (e.g. % of improvement over last quarter).

KPI development is about answering the question: what would indicate that I have achieved my goal?

To develop a KPI follow these steps:

1. Review the case study on the following slides to get a sense of how KPIs can start simple and general and get more specific and complex over time.
2. Using the example to the right, sort your SMART goals from step 1.1 into the various metric types, then determine what success would look like for you. What outcome are you trying to achieve? How will you know when you’ve achieved it?
3. Fill out the KPI Development Worksheets to create sample KPIs for each of the SMART goals you have created. Ensure that you complete the accompanying KPI Checklist.

KPIs differ from goal to goal, but their forms follow certain trends

Explore the five metric types

1. Initial Probe

Focused on determining how many sources for metrics exist.

• Question: What am I capable of knowing?
• Goal: To determine what level of insight we have into our security processes.
• Possible KPI: % of systems for which metrics are available.
• Decision: Do we have sufficient resources available to collect metrics?

2. Baseline Testing

Focused on gaining initial insights about the state of your security program (what are the measurements?).

• Question: Does this data suggest areas for improvement?
• Goal: To create a roadmap for improvement.
• Possible KPI: % of systems that provide useful metrics to measure improvement.
• Decision: Is it necessary to acquire tools to increase, enhance, or streamline the metrics-gathering process?

Info-Tech Insight

Don't lose hope if you lack resources to move beyond these initial steps. Even if you are struggling to pull data, you can still draw meaningful metrics. The percent or ratio of processes or systems you lack insight into can be very valuable, as it provides a basis to initiate a risk-based discussion with management about the organization's security blind spots.

Explore the five metric types (cont’d)

3. Program Implementation

Focused on developing a basic program to establish basic maturity (e.g. implement an awareness and training program).

• Question: What needs to be implemented to establish basic maturity?
• Goal: To begin closing the gap between current and desired maturity.
• Possible KPI: % of implementation completed.
• Decision: Have we achieved a formalized and repeatable process?

4. Improvement

Focused on attaining operational targets to lower organizational risk.

• Question: What other related activities could help to support this goal (e.g. regular training sessions)?
• Goal: To have metrics operate above or below a certain threshold (e.g. lower phishing-test click rate to an average of 10% across the organization)
• Possible KPI: Phishing click rate %
• Decision: What other metrics should be tracked to provide insight into KPI fluctuations?

Info-Tech Insight

Don't overthink your KPI. In many cases it will simply be your goal rephrased to express a percentage or ratio. In others, like the example above, it makes sense for them to be identical.

5. Organizational Impact

Focused on studying several related KPIs (Key Performance Index, or KPX) in an attempt to predict risks.

• Question: What risks does the organization need to address?
• Goal: To provide high-level summaries of several metrics that suggest emerging or declining risks.
• Possible KPI: Likelihood of a given risk (based on the trends of the KPX).
• Decision: Accept the risk, transfer the risk, mitigate the risk?

Case study: Healthcare example

Let’s take a look at KPI development in action.

Meet Maria, the new CISO at a large hospital that desperately needs security program improvements. Maria’s first move was to learn the true state of the organization’s security. She quickly learned that there was no metrics program in place and that her staff were unaware what, if any, sources were available to pull security metrics from.

After completing her initial probe into available metrics and then investigating the baseline readings, she determined that her areas of greatest concern were around vulnerability and access management. But she also decided it was time to get a security training and awareness program up and running to help mitigate risks in other areas she can’t deal with right away.

Info-Tech Insight

There is very little variation in the kinds of goals people have around initial probes and baseline testing. Metrics in these areas are virtually always about determining what data sources are available to you and what that data actually shows. The real decisions start in determining what you want to do based on the measures you’re seeing.

Metric development example: Vulnerability Management

See examples of Maria’s KPI development on the next four slides...

Implementation

Goal: Implement vulnerability management program

KPI: % increase of insight into existing vulnerabilities

Associated Metric: # of vulnerability detection methods

Improvement

Goal: Improve deployment time for patches

KPI: % of critical patches fully deployed within target window

• Associated Metric 1: # of critical vulnerabilities not patched
• Associated Metric 2: # of patches delayed due to lack of staff
• Associated Metric X

Metric development example: Identity Access Management

Implementation

Goal: Implement MFA for privileged accounts

KPI: % of privileged accounts with MFA applied

Associated Metric: # of privileged accounts

Improvement

Goal: Remove all unnecessary privileged accounts

KPI: % of accounts with unnecessary privileges

• Associated Metric 1: # of privileged accounts
• Associated Metric 2: # of necessary privileged accounts
• Associated Metric X

Metric development example: Training and Awareness

Implementation

Goal: Implement training and awareness program

KPI: % of organization trained

Associated Metric: # of departments trained

Improvement

Goal: Improve time to report phishing

KPI: % of phishing cases reported within target window

• Associated Metric 1: # of phishing tests
• Associated Metric 2: # of training sessions
• Associated Metric X

Metric development example: Key Performance Index

Organizational Trends

Goal: Predict Data Breach Likelihood

• KPX 1: Insider Threat Potential
  • % of phishing cases reported within target window
    • Associated Metrics:
      • # of phishing tests
      • # of training sessions
  • % of critical patches fully deployed within target window
    • Associated Metrics:
      • # of critical vulnerabilities not patched
      • # of patches delayed due to lack of staff
  • % of accounts with unnecessary privileges
    • Associated Metrics:
      • # of privileged accounts
      • # of necessary privileged accounts
• KPX 2: Data Leakage Issues
  • % of incidents related to unsecured databases
    • Associated Metrics:
      • # of unsecured databases
      • # of business-critical databases
  • % of misclassified data
    • Associated Metrics:
      • # of misclassified data reports
      • # of DLP false positives
  • % of incidents involving data-handling procedure violations.
    • Associated Metrics:
      • # of data processes with SOP
      • # of data processes without SOP
• KPX 3: Endpoint Vulnerability Issues
  • % of unpatched critical systems
    • Associated Metrics:
      • # of unpatched systems
      • # of missed patches
  • % of incidents related to IoT
    • Associated Metrics:
      • # of IoT devices
      • # of IoT unsecure devices
  • % of incidents related to BYOD
    • Associated Metrics:
      • # of end users doing BYOD
      • # of BYOD incidents

Develop Goals-Based KPIs

1.2 120 minutes

Materials

• Info-Tech KPI Development Worksheets

Participants

• Security Team

Output

• List of KPIs for immediate and future use (can be used to populate Info-Tech’s KPI Development Tool).

It’s your turn.

Follow the example of the CISO in the previous slides and try developing KPIs for the SMART goals set in step 1.1.

• To begin, decide if you are starting with implementation or improvement metrics.
• Enter your goal in the space provided on the left-hand side and work towards the right, assigning a KPI to track progress towards your goal.
• Use the associated metrics boxes to record what raw data will inform or influence your KPI.
  • Associated metrics are connected to the KPI box with a segmented line. This is because these associated metrics are not absolutely necessary to track progress towards your goal.
  • However, if a KPI starts trending in the wrong direction, these associated metrics would be used to determine where the problem has occurred.
• If desired, bundle together several related KPIs to create a key performance index (KPX), which is used to forecast the likelihood of certain risks that would have a major business impact (e.g. potential for insider threat, or risk for a data breach).

Record KPIs and assign them to goals in the Security Metrics Determination and Tracking Tool

1.2 Security Metrics Determination and Tracking Tool

Document KPI metadata in the tool and optionally assign them to a goal.

Tab “3. Identify Goal KPIs” allows you to record each KPI and its accompanying metadata:

• Source
• Owner
• Audience
• KPI Target
• Effort to Collect
• Frequency of Collection
• Comments

Optionally, each KPI can be mapped to goals defined on tab “2. Identify Security Goals.”

Info-Tech Best Practice

Ensure your metadata is comprehensive, complete, and realistic. A different employee should be able to use only the information outlined in the metadata to continue collecting measurements for the program.

Complete Info-Tech’s KPI Development Worksheets

1.2 KPI Development Worksheet

Use these worksheets to model the maturation of your metrics program.

Follow the examples contained in this slide deck and practice creating KPIs for:

• Implementation metrics
• Improvement metrics
• Organizational trends metrics

As well as drafting associated metrics to inform the KPIs you create.

Info-Tech Best Practice

Keep your metrics program manageable. This exercise may produce more goals, metrics, and KPIs than you deal with all at once. But that doesn’t mean you can’t save some for future use.

Build an effort map to prioritize your SMART goals

1.2 120 minutes

Materials

• Whiteboard
• Sticky notes
• Laptop

Participants

• Security team
• Other stakeholders

Output

• Prioritized list of SMART goals

An effort map visualizes a cost and benefit analysis. It is a quadrant output that visually shows how your SMART goals were assessed. Use the calculated Cost/Effort Rating and Benefit Rating values from tab “2. Identify Security Goals” of the Security Metrics Determination and Tracking Tool to aid this exercise.

Steps:

1. Establish the axes and colors for your effort map:
   1. X-axis (horizontal) - Security benefit
   2. Y-axis (vertical) - Overall cost/effort
   3. Sticky color - Business alignment
2. Create sticky notes for each SMART goal and place them onto the effort map based on your determined axes.
   • Goal # Example Security Goal - Benefit (1-12) - Cost (1-12)

1.3 Implement and monitor the KPI to track goal progress

Let’s put your KPI into action!

Now that you’ve developed KPIs to monitor progress on your goals, it’s time to use them to drive security program maturation by following these steps:

1. Review the KPI Development Worksheets (completed in step 1.2) for your prioritized list of goals. Be sure that you are able to track all of the associated metrics you have identified.
2. Track the KPI and associated metrics using Info-Tech’s KPI Development Tool (see following slide).
3. Update the data as necessary according to your SMART criteria of your goal.

A Word on Key Risk Indicators...

The term key risk indicator (KRI) gets used in a few different ways. However, in most cases, KRIs are closely associated with KPIs.

1. KPIs and KRIs are the same thing
   • A KPI, at its core, is really a measure of risk. Sometimes it is more effective to emphasize that risk rather than performance (i.e. the data shows you’re not meeting your goal).
2. KRI is KPI going the wrong way
   • After achieving the desired threshold for an improvement goal, our new goal is usually to maintain such a state. When this balance is upset, it indicates that settled risk has once again become active.
3. KRI as a predictor of emerging risks
   • When organizations reach a highly mature state, they often start assessing how events external to the organization can affect the optimal performance of the organization. They monitor such events or trends and try to predict when the organization is likely to face additional risks.

Track KPIs in the Security Metrics Determination and Tracking Tool

1.3 Security Metrics Determination and Tracking Tool

Once a metric has been measured, you have the option of entering that data into tab “4. Track Metrics” of the Tool.

Tracking metric data in Info-Tech's tool provides the following data visualizations:

• Sparklines at the end of each row (on tab “4. Track Metrics”) for a quick sense of metric performance.
• A metrics dashboard (on tab “5. Graphs”) with three graph options in two color variations for each metric tracked in the tool, and an overall metric program health gauge.

Info-Tech Best Practice

Be diligent about measuring and tracking your metrics. Record any potential measurement biases or comments on measurement values to ensure you have a comprehensive record for future use. In the tool, this can be done by adding a comment to a cell with a metric measurement.

If you want additional support, have our analysts guide you through this phase as part of an Info-Tech workshop

Book a workshop with our Info-Tech analysts:

Workshops offer an easy way to accelerate your project. While onsite, our analysts will work with you and your team to facilitate the activities outlined in the blueprint.

Getting key stakeholders together to formalize the program, while getting started on data discovery and classification, allows you to kickstart the overall program.

In addition, leverage over-the-phone support through Guided Implementations included in advisory memberships to ensure the continuous improvement of the classification program even after the workshop.

Logan Rohde

Research Analyst – Security, Risk & Compliance Info-Tech Research Group

Ian Mulholland

Senior Research Analyst – Security, Risk & Compliance Info-Tech Research Group

Call 1-888-670-8889 for more information.

Phase 2

Adapt Your Reporting Strategy for Various Metric Types

Phase 2

2.1 Review best practices for presenting metrics

2.2 Strategize your presentation based on metric type

2.3 Tailor your presentation to your audience

2.4 Use your metrics to create a story about risk

2.5 Revise Metrics

This phase will walk you through the following activities:

• Develop reporting strategy
• Use metrics to create a story about risk
• Metrics revision

This phase involves the following participants:

• Security Team

Outcomes of this phase

• Metrics Dashboard
• Metrics Presentation Deck

Phase 2 outline

Call 1-888-670-8889 or email GuidedImplementations@InfoTech.com for more information.

Complete these steps on your own or call us to complete a guided implementation. A guided implementation is a series of two to three advisory calls that help you execute each phase of a project. They are included in most advisory memberships.

Guided Implementation 2: Adapt Your Reporting Strategy for Various Metric Types

Proposed Time to Completion: 2-4 weeks

Step 2.1 – 2.3: Best Practices and Reporting Strategy

Start with an analyst kick-off call:

• Do’s and Don’ts of reporting metrics.
• Strategize presentation based on metric type.

Then complete these activities…

• Strategy development for 3-5 metrics

Step 2.4 – 2.5: Build a Dashboard and Presentation Deck

Review findings with analyst:

• Review strategies for reporting.
• Compile a Key Performance Index.
• Revise metrics.

Then complete these activities…

• Dashboard creation
• Presentation development

With these tools & templates:

• Security Metrics Determination and Tracking Tool Template
• Security Metrics KPX Dashboard Tool

Phase 2 Results & Insights:

• Completed reporting strategy with presentable dashboard

2.1 Review best practices for presenting metrics

Avoid technical details (i.e. raw data) by focusing on the KPI.

• KPIs add context to understand the behavior and associated risks.

Put things in terms of risk; it's the language you both understand.

• This usually means explaining what will happen if not addressed and what you recommend.
• There are always three options:
  • Address it completely
  • Address it partially
  • Do not address it (i.e. accept the risk)

Explain why you’re monitoring metrics in terms of the goals you’re hoping to achieve.

• This sets you up well to explain what you've been doing and why it's important for you to meet your goals.

Choose between KPI or KRI as the presentation format.

• Base your decision on whether you are trying to emphasize current success or risk.

Match presentation with the audience.

• Board presentations will be short; middle-management ones may be a bit longer.
• Maximize your results by focusing on the minimum possible information to make sure you sufficiently get your point across.
• With the board, plan on showing no more than three slides.

Read between the lines.

• It can be difficult to get time with the board, so you may find yourself in a trial and error position, so pay attention to cues or suggestions that indicate the board is interested in something.
• If you can, make an ally to get the inside scoop on what the board cares about.

Read the news if you’re stuck for content.

• Board members are likely to have awareness (and interest) in large-scale risks like data breaches and ransomware.

Present your metrics as a story.

• Summarize how the security program looks to you and why the metrics lead you to see it this way.

2.2 Strategize your presentation based on metric type (1 of 5)

Metric Type: Initial Probe

Scenario: Implementing your first metrics program.

• All metrics programs start with determining what measurements you are capable of taking.

Decisions: Do you have sufficient insight into the program? (i.e. do you need to acquire additional tools to collect metrics?)

Strategy: If there are no barriers to this (e.g. budget), then focus your presentation on the fact that you are addressing the risk of not knowing what your organization's baseline state is and what potential issues exist but are unknown. This is likely the first phase of an improvement plan, so sketching the overall plan is a good idea too.

• If budget is an issue, explain the risks associated with not knowing and what you would need to make it happen.

Possible KPIs:

• % of project complete.
• % of systems that provide worthwhile metrics.

Strategize your presentation based on metric type (2 of 5)

Metric Type: Baseline Testing

Scenario: You've taken the metrics to determine what your organization’s normal state is and you're now looking towards addressing your gaps or problem areas.

Decisions: What needs to be prioritized first and why? Are additional resources required to make this happen?

Strategy: Explain your impression of the organization's normal state and what you plan to do about it. In other words, what goals are you prioritizing and why? Be sure to note any challenges that may occur along the way (e.g. staffing).

• If the board doesn't like to open their pocketbook, your best play is to explain what stands to happen (or is happening) if risks are not addressed.

Possible KPIs:

• % of goals complete.
• % of metrics indicating urgent attention needed.

Strategize your presentation based on metric type (3 of 5)

Metric Type: Implementation

Scenario: You are now implementing solutions to address your security priorities.

Decisions: What, to you, would establish the basis of a program?

Strategy: Focus on what you're doing to implement a certain security need, why, and what still needs to be done when you’re finished.

• Example: To establish a training and awareness program, a good first step is to actually hold training sessions with each department. A single lecture is simple but something to build from. A good next step would be to hold regular training sessions or implement monthly phishing tests.

Possible KPIs:

• % of implementation complete (e.g. % of departments trained).

Strategize your presentation based on metric type (4 of 5)

Metric Type: Improvement

Scenario: Now that a basic program has been established, you are looking to develop its maturity to boost overall performance (i.e. setting a new development goal).

Decisions: What is a reasonable target, given the organization's risk tolerance and current state?

Strategy: Explain that you're now working to tighten up the security program. Note that although things are improving, risk will always remain, so we need to keep it within a threshold that’s proportionate with our risk tolerance.

• Example: Lower phishing-test click rate to 10% or less. Phishing will always be a risk, and just one slip up can have a huge effect on business (i.e. lost money).

Possible KPIs:

• % of staff passing the phishing test.
• % of employees reporting phishing attempts within time window.

Strategize your presentation based on metric type (5 of 5)

Metric Type: Organizational Trends

Scenario: You've reached a mature state and now how several KPIs being tracked. You begin to look at several KPIs together (i.e. a KPX) to assess the organization's exposure for certain broad risk trends.

Decisions: Which KPIs can be used together to look at broader risks?

Strategy: Focus on the overall likelihood of a certain risk and why you've chosen to assess it with your chosen KPIs. Spend some time discussing what factors affect the movement of these KPIs, demonstrating how smaller behaviors create a ripple effect that affects the organization’s exposure to large-scale risks.

Possible KPX: Insider Threat Risk

• % of phishing test failures.
• % of critical patches missed.
• % of accounts with unnecessary privileges.

Change your strategy to address security challenges

Even challenges can elicit useful metrics.

Not every security program is capable of progressing smoothly through the various metric types. In some cases, it is impossible to move towards goals and metrics for implementation, improvement, or organizational trends because the security program lacks resources.

Info-Tech Insight

When your business is suffering from a lack of resources, acquiring these resources automatically becomes the goal that your metrics should be addressing. To do this, focus on what risks are being created because something is missing.

When your security program is lacking a critical resource, such as staff or technology, your metrics should focus on what security processes are suffering due to this lack. In other words, what critical activities are not getting done?

KPI Examples:

• % of critical patches not deployed due to lack of staff.
• % of budget shortfall to acquire vulnerability scanner.
• % of systems with unknown risk due to lack of vulnerability scanner.

2.3 Tailor presentation to your audience

Metrics come in three forms...

1. Raw Data

• Taken from logs or reports, provides values but not context.
• Useful for those with technical understanding of the organization’s security program.

2. Management-Level

• Raw data that has been contextualized and indicates performance of something (i.e. a KPI).
• Useful for those with familiarity with the overall state of the security program but do not have a hands-on role.

3. Board-Level

• KPI with additional context indicating overall effect on the organization.
• Useful for those removed from the security program but who need to understand the relationship between security, business goals, and cyber risk.

For a metric to be useful it must...

1. Be understood by the audience it’s being presented to.
   • Using the criteria on the left, choose which metric form is most appropriate.
2. Indicate whether or not a certain target or goal is being met.
   • Don’t expect metrics to speak for themselves; explain what the indications and implications are.
3. Drive some kind of behavioral or strategic change if that target or goal is not being met.
   • Metrics should either affirm that things are where you want them to be or compel you to take action to make an improvement. If not, it is not a worthwhile metric.

As a general rule, security metrics should become decreasingly technical and increasingly behavior-based as they are presented up the organizational hierarchy.

"The higher you travel up the corporate chain, the more challenging it becomes to create meaningful security metrics. Security metrics are intimately tied to their underlying technologies, but the last thing the CEO cares about is technical details." – Ben Rothke, Senior Information Security Specialist, Tapad.

Plan for reporting success

The future of your security program may depend on this presentation; make it count.

Reporting metrics is not just another presentation. Rather, it is an opportunity to demonstrate and explain the value of security.

It is also a chance to correct any misconceptions about what security does or how it works.

Use the tips on the right to help make your presentation as relatable as possible.

Info-Tech Insight

There is a difference between data manipulation and strategic presentation: the goal is not to bend the truth, but to present it in a way that allows you to show the board what they need to see and to explain it in terms familiar to them.

General Tips for a Successful Presentation

Avoid jargon; speak in practical terms

• The board won’t receive your message if they can’t understand you.
• Explain things as simply as you can; they only need to know enough to make decisions about addressing cyber risk.

Address compliance

• Boards are often interested in compliance, so be prepared to talk about it, but clarify that it doesn't equal security.
• Instead, use compliance as a bridge to discussing areas of the security program that need attention.

Have solid answers

• Try to avoid answering questions with the answer, “It depends.”
  • Depends on what?
  • Why?
  • What do you recommend?
• The board is relying on you for guidance, so be prepared to clarify what the board is asking (you may have to read between the lines to do this).
• Also address the pain points of board members and have answers to their questions about how to resolve them.

2.4 Use your metrics to create a story about risk

Become the narrator of your organization’s security program.

Security is about managing risk. This is also its primary value to the organization. As such, risk should be the theme of the story you tell.

"Build a cohesive story that people can understand . . . Raw metrics are valuable from an operations standpoint, but at the executive level, it's about a cohesive story that helps executives understand the value of the security program and keeps the company moving forward. "– Adam Ely, CSO and Co-Founder, Bluebox Security, qtd. by Tenable, 2016

How to Develop Your Own Story...

1. Review your security program goals and the metrics you’re using to track progress towards them. Then, decide which metrics best tell this story (i.e. what you’re doing and why).
   • Less is more when presenting metrics, so be realistic about how much your audience can digest in one sitting.
   • Three metrics is usually a safe number; choose the ones that are most representative of your goals.
2. Explain why you chose the goals you did (i.e. what risks were you addressing?). Then, make an honest assessment of how the security program is doing as far as meeting those goals:
   • What’s going well?
   • What still needs improvement?
   • What about your metrics suggests this?
3. Address how risks have changed and explain your new recommended course of action.
   • What risks were present when you started?
   • What risks remain despite your progress?
   • How do these risks affect the business operation and what can security do to help?

Story arc for security metrics

The following model encapsulates the basic trajectory of all story development.

Use this model to help you put together your story about risk.

Introduction: Overall assessment of security program.

Initial Incident: Determination of the problems and associated risks.

Rising Action: Creation of goals and metrics to measure progress.

Climax: Major development indicated by metrics.

Falling Action: New insights gained about organization’s risks.

Resolution: Recommendations based on observations.

Info-Tech Best Practice

Follow this model to ensure that your metrics presentation follows a coherent storyline that explains how you assessed the problem, why you chose to address it the way you did, what you learned in doing so, and finally what should be done next to boost the security program’s maturity.

Use a nesting-doll approach when presenting metrics

Move from high-level to low-level to support your claims

1. Avoid the temptation to emphasize technical details when presenting metrics. The importance of a metric should be clear from just its name.
2. This does not mean that technical details should be disregarded entirely. Your digestible, high-level metrics should be a snapshot of what’s taking place on the security ground floor.
3. With this in mind, we should think of our metrics like a nesting doll, with each metrics level being supported by the one beneath it.

...How do you know that?

Board-Level KPI

Mgmt.-Level KPI

Raw Data

Think of your lower-level metrics as evidence to back up the story you are telling.

When you’re asked how you arrived at a given conclusion, you know it’s time to go down a level and to explain those results.

Think of this like showing your work.

Info-Tech Insight

This approach is built into the KPX reporting format, but can be used for all metric types by drawing from your associated metrics and goals already achieved.

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics Determination and Tracking Tool

Choose the dashboard tool that makes the most sense for you.

Info-Tech provides two options for metric dashboards to meet the varying needs of our members.

If you’re just starting out, you’ll likely be inclined towards the dashboard within the Security Metrics Determination and Tracking Tool (seen here).

But if you’ve already got several KPIs to report on, you may prefer the Security Metrics KPX Dashboard Tool, featured on the following slides.

Info-Tech Best Practice

Not all graphs will be needed in all cases. When presenting, consider taking screenshots of the most relevant data and displaying them in Info-Tech’s Board-Level Security Metrics Presentation Template.

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics KPX Dashboard

Use Info-Tech’s Security Metrics KPX Dashboard to track and show your work.

1. Start by customizing the definitions on tab 1 to match your organization’s understanding of high, medium, and low risk across the three impact areas (functional, informational, and recoverability).
2. Next, enter up to 5 business goals that your security program supports.

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics KPX Dashboard

Use Info-Tech’s Security Metrics KPX Dashboard to track and show your work.

3. On tab 2, enter the large-scale risk you are tracking
4. Proceed by naming each of your KPXs after three broad risks that – to you – contribute to the large-scale risk.

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics KPX Dashboard

Use Info-Tech’s Security Metrics KPX Dashboard to track and show your work.

5. Then, add up to five KPIs aimed at managing more granular risks that contribute to the broad risk.
6. Assess the frequency and impact associated with these more granular risks to determine how likely it is to contribute to the broad risk the KPX is tracking.

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics KPX Dashboard

Use Info-Tech’s Security Metrics KPX Dashboard to track and show your work.

7. Repeat as necessary for the other KPXs on tab 2.
8. Repeat steps 3-7 for up to two more large-scale risks and associated KPXs on tabs 3 and 4.

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics KPX Dashboard

Use Info-Tech’s Security Metrics KPX Dashboard to track and show your work.

9. If desired, complete the Business Alignment evaluation (located to the right of KPX 2 on tabs 2-4) to demonstrate how well security is supporting business goals.

"An important key to remember is to be consistent and stick to one framework once you've chosen it. As you meet with the same audiences repeatedly, having the same framework for reference will ensure that your communications become smoother over time." – Caroline Wong, Chief Strategy Officer, Cobalt.io

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics KPX Dashboard

Use Info-Tech’s Security Metrics KPX Dashboard to track and show your work.

10. Use the dashboard on tab 5 to help you present your security metrics to senior leadership.

Use one of Info-Tech’s dashboards to present your metrics

2.4 Security Metrics KPX Dashboard

Use Info-Tech’s Security Metrics KPX Dashboard to track and show your work.

Best Practice:

This tool helps you convert your KPIs into the language of risk by assessing frequency and severity, which helps to make the risk relatable for senior leadership. However, it is still useful to track fluctuations in terms of percentage. To do this, track changes in the frequency, severity, and trend scores from quarter to quarter.

Customize Info-Tech’s Security Metrics Presentation Template

2.4 Board-Level Security Metrics Presentation Template

Use the Board-Level Security Metrics Presentation Template deck to help structure and deliver your metrics presentation to the board.

To make the dashboard slide, simply copy and paste the charts from the dashboard tool and arrange the images as needed.

Adapt the status report and business alignment slides to reflect the story about risk that you are telling.

2.5 Revise your metrics

What's next?

Now that you’ve made it through your metrics presentation, it’s important to reassess your goals with feedback from your audience in mind. Use the following workflow.

If you want additional support, have our analysts guide you through this phase as part of an Info-Tech workshop

Book a workshop with our Info-Tech analysts:

Workshops offer an easy way to accelerate your project. While onsite, our analysts will work with you and your team to facilitate the activities outlined in the blueprint.

Getting key stakeholders together to formalize the program, while getting started on data discovery and classification, allows you to kickstart the overall program.

In addition, leverage over-the-phone support through Guided Implementations included in advisory memberships to ensure the continuous improvement of the classification program even after the workshop.

Logan Rohde

Research Analyst – Security, Risk & Compliance Info-Tech Research Group

Ian Mulholland

Senior Research Analyst – Security, Risk & Compliance Info-Tech Research Group

Call 1-888-670-8889 for more information.

Insight breakdown

Metrics lead to maturity, not vice versa.

• Tracking metrics helps you assess progress and regress in your security program, which helps you quantify the maturity gains you’ve made.

Don't lose hope if you lack resources to move beyond baseline testing.

• Even if you are struggling to pull data, you can still draw meaningful metrics. The percent or ratio of processes or systems you lack insight into can be very valuable, as it provides a basis to initiate a risk-based discussion with management about the organization's security blind spots.

The best metrics are tied to goals.

• Tying your metrics to goals ensures that you are collecting metrics for a specific purpose rather than just to watch the numbers change.

Summary of accomplishment

Knowledge Gained

• Current maturity assessment of security areas
• Setting SMART goals
• Metric types
• KPI development
• Goals prioritization
• Reporting and revision strategies

Processes Optimized

• Metrics development
• Metrics collection
• Metrics reporting

Deliverables Completed

• KPI Development Worksheet
• Security Metrics Determination and Tracking Tool
• Security Metrics KPX Dashboard Tool
• Board-Level Security Metrics Presentation Template

Research contributors and experts

Mike Creaney, Senior Security Engineer at Federal Home Loan Bank of Chicago

Peter Chestna, Director, Enterprise Head of Application Security at BMO Financial Group

Zane Lackey, Co-Founder / Chief Security Officer at Signal Sciences

Ben Rothke, Senior Information Security Specialist at Tapad

Caroline Wong, Chief Strategy Officer at Cobalt.io

2 anonymous contributors

Related Info-Tech research

Build an Information Security Strategy

Tailor best practices to effectively manage information security.

Implement a Security Governance and Management Program

Align security and business objectives to get the greatest benefit from both.

Bibliography

Capability Maturity Model Integration (CMMI). ISACA. Carnegie Mellon University.

Ely, Adam. “Choose Security Metrics That Tell a Story.” Using Security Metrics to Drive Action: 33 Experts Share How to Communicate Security Program Effectiveness to Business Executives and the Board Eds. 2016. Web.

https://www.ciosummits.com/Online_Assets_Tenable_eBook-_Using_Security_Metrics_to_Drive_Action.pdf

ISACA. “Board Director Concerns about Cyber and Technology Risk.” CSX. 11 Sep. 2018. Web.

Rothke, Ben. “CEOs Require Security Metrics with a High-Level Focus.” Using Security Metrics to Drive Action: 33 Experts Share How to Communicate Security Program Effectiveness to Business Executives and the Board Eds. 2016. Web.

https://www.ciosummits.com/Online_Assets_Tenable_eBook-_Using_Security_Metrics_to_Drive_Action.pdf

Wong, Caroline. Security Metrics: A Beginner’s Guide. McGraw Hill: New York, 2012.

Agile Readiness Assessment

Understand how ready your organization is for an Agile transformation.

Info-Tech Research Group Inc. is a global leader in providing IT research and advice. Info-Tech’s products and services combine actionable insight and relevant advice with ready-to-use tools and templates that cover the full spectrum of IT concerns.

Analyst Perspective

Use the wisdom of crowds to understand how ready you are for Agile transformation.

Agile transformations can be difficult and complex to implement. That’s because they require fundamental changes in the way an organization thinks and behaves (and many organizations are not ready for these changes).

Use Info-Tech’s Agile Readiness Assessment to broadly survey the organization’s readiness for Agile along six dimensions:

• Culture
• Learning
• Automation
• Integrated teams
• Metrics
• Governance

The survey results will help you to examine and address those areas that are most likely to hinder your move to Agile.

Alex Ciraco
Principal Research Director, Application Delivery and Management
Info-Tech Research Group

Executive Summary

Your Challenge

• Your organization wants to shorten delivery time and improve quality by adopting Agile practices.
• Your organization has not yet used Agile successfully.
• You know that Agile transformations are complex and difficult to implement.
• You want to maximize your Agile transformation’s chances of success.

Common Obstacles

• Risks to your Agile transformation can come from a variety of sources, including:
  • Organizational culture
  • Learning practices
  • Use of automation
  • Ability to create integrated teams
  • Use of metrics
  • Governance practices

Info-Tech’s Approach

• Use Info-Tech’s Agile Readiness Assessment to broadly survey your organization’s readiness for Agile.
• Examine the consolidated results of this survey to identify challenges that are most likely to hinder Agile success.
• Discuss and address these challenges to increase your chances of success.

Info-Tech Insight

By first understanding the numerous challenges to Agile transformations and then broadly surveying your organization to identify and address the challenges that are at play, you are more likely to have a successful Agile transformation.

Info-Tech’s methodology

Blueprint deliverables

Each step of this blueprint is accompanied by supporting deliverables to help you accomplish your goals.

Info-Tech offers various levels of support to best suit your needs

Diagnostics and consistent frameworks used throughout all four options

Guided Implementation

A Guided Implementation (GI) is a series of calls with an Info-Tech analyst to help implement our best practices in your organization.

A typical GI is between 6 to 8 calls over the course of 1 to 2 months.

What does a typical GI on this topic look like?

Phase 1: Distribute Survey

• Call #1: Scope requirements, objectives, and your specific challenges (identify potential participants).
• Call #2: First call with participants (introduce Phase 1 and assign survey for completion).
• Call #3: Gather survey responses (prep for Phase 2 calls).

Phase 2: Consolidate Survey Results

• Call #4: Consolidate all survey responses using the template.
• Call #5: Conduct initial review of consolidated results (prep for Phase 3 calls).

Phase 3: Examine Results and Problem Solve

• Call #6: Present consolidated results to participants and agree on most pressing challenges.
• Call #7: Brainstorm, identify, and refine potential solutions to most pressing challenges.
• Call #8: Conduct closing and communication call.

Phase 1 —

Customize and distribute the survey

Decide which teams/roles will participate in the survey.

Decide which format and language(s) you will use for your Agile Readiness Assessment Survey.

Configure the survey templates to reflect your selected teams/roles.

Distribute the survey for participants to complete.

• 1.1 The Agile Readiness Assessment Survey will help you to identify both team-specific and organization-wide challenges to your Agile transformation. It is best to distribute the survey broadly across the organization and include several teams and roles. Identify and make note of the teams/roles that will be participating in the survey.
• 1.2 Select which format of survey you will be using (Excel or online), along with the language(s) you will use (links to the survey templates can be found in the table below). Then configure the survey templates to reflect your list of teams/roles from Step 1.1.

Format	Language	Download Survey Template
Excel	English	Agile Readiness Assessment Excel Survey Template – EN and FR
Excel	French
Online	English	Agile Readiness Assessment Online Survey Template – EN
Online	French	Agile Readiness Assessment Online Survey Template – FR



• 1.3 Distribute your Agile Readiness Assessment Survey broadly in the organization. Give all participants a deadline date for completion of the survey.

Phase 2 —

Consolidate Survey Results

Collect and consolidate all survey responses using the template provided.

Review the OVERALL and DETAILED Heatmaps generated by the template.

• 2.1 Collect the survey responses from all participants. All responses completed using the online form will be anonymous (for responses returned using the Excel form, assign each a unique identifier so that anonymity of responses is maintained).
• 2.2 Consolidate the survey responses using the template below. Follow the instructions in the template to incorporate all survey responses.

Download the Agile Readiness Assessment Consolidated Results Tool

Phase 3 —

Examine Survey Results and Problem Solve

Review the consolidated survey results as a team.

Identify the challenges that need the most attention.

Brainstorm potential solutions. Decide which are most promising and create a plan to implement them.

• 3.1 Examine the consolidated results (both OVERALL and DETAILED Heatmaps) and look at both team-specific and organization-wide challenge areas.
• 3.2 Identify which challenge areas need the most attention (typically those that are most red in the heatmap) and discuss these challenges with survey participants.
• 3.3 As a team, brainstorm potential solutions to these challenges. Select from and refine the solutions that are most promising, then create a plan to implement them.

3.1 Exercise: Collaborative Problem Solving —

Input: Consolidated survey results

Output: List of actions to address your most pressing challenges along with a timeline to implement them

Materials: Agile Readiness Assessment Consolidated Results Tool, Whiteboard and markers

Participants: Survey participants, Other interested parties

This exercise will create a plan for addressing your most pressing Agile-related challenges.

• As a team, agree on which survey challenges are most important to address (typically the most red in the heatmap).
• Brainstorm potential solutions/actions to address these challenges.
• Assign solutions/actions to individuals and set a timeline for completion.

Related Info-Tech Research

	Modernize Your SDLC Strategically adopt today’s SDLC good practices to streamline value delivery.
	Implement Agile Practices That Work Guide your organization through its Agile transformation journey.
	Implement DevOps Practices That Work Streamline business value delivery through the strategic adoption of DevOps practices.
	Mentoring for Agile Teams Leverage an experience Agile Mentor to give your in-flight Agile project a helping hand.

Research Contributors and Experts

• Columbus Brown, Senior Principal – Practice Lead – Business Alignment, Daugherty Business Solutions
• Saeed Khan, Founder, Transformation Labs
• Brenda Peshak, Product Owner/Scrum Master/Program Manager, John Deere/Source Allies/Widget Industries LLC
• Vincent Mirabelli, Principal, Global Project Synergy Group
• Len O'Neill, Sr. Vice President and Chief Information Officer, The Suddath Companies
• Shameka A. Jones, MPM, CSM, Lead Business Management Consultant, Mainspring Business Group, LLC
• Ryland Leyton, Lead Business Analyst, Aptos Retail
• Ashish Nangia, Lead Business System Analyst, Ashley Furniture Industries
• Barbara Carkenord, CBAP, IIBA-AAC, PMI-PBA, PMP, SAFe POPM, President, Carkenord Consulting
• Danelkis Serra, CBAP, Chapter Operations Manager, Regions & Chapters, IIBA (International Institute of Business Analysis)
• Lorrie Staples-Ellis, CyberSecurity Integration Strategist, Wealth Management, Truist Bank
• Ginger Sundberg, Independent Consultant
• Kham Raven, Project Manager, Fraud Strategy & Execution, Truist Bank
• Sarah Vollett, PMP, Business Analyst, Operations, College of Physicians and Surgeons of British Columbia
• Nicole J Coyle, ICP-ACC, CEAC, SPC4, SASM, POPM, CSM, ECM, CCMP, CAPM, Team Agile Coach and Team Facilitator, HCQIS Foundational Components
• Joe Glower, IT Director, Jet Support Services, Inc. (JSSI)
• Harsh Daharwal, Senior Director, Application Delivery, J.R. Simplot
• Hans Eckman, Principal Research Director, Info-Tech Research Group
• Valence Howden, Principal Research Director, Info-Tech Research Group

Build a Reporting and Analytics Strategy

Deliver actionable business insights by creating a business-aligned reporting and analytics strategy.

Terminology

As the reporting and analytics space matured over the last decade, software suppliers used different terminology to differentiate their products from others’. This caused a great deal of confusion within the business communities.

Following are two definitions of the term Business Intelligence:

Business intelligence (BI) leverages software and services to transform data into actionable insights that inform an organization’s strategic and tactical business decisions. BI tools access and analyze data sets and present analytical findings in reports, summaries, dashboards, graphs, charts, and maps to provide users with detailed intelligence about the state of the business.

The term business intelligence often also refers to a range of tools that provide quick, easy-to-digest access to insights about an organization's current state, based on available data.

CIO Magazine

Business intelligence (BI) comprises the strategies and technologies used by enterprises for the data analysis of business information. BI technologies provide historical, current, and predictive views of business operations.

Common functions of business intelligence technologies include reporting, online analytical processing, analytics, data mining, process mining, complex event processing, business performance management, benchmarking, text mining, predictive analytics, and prescriptive analytics.

Wikipedia

This blueprint will use the terms “BI,” “BI and Analytics,” and “Reporting and Analytics” interchangeably in different contexts, but always in compliance to the above definitions.

ANALYST PERSPECTIVE

A fresh analytics & reporting strategy enables new BI opportunities.

We need data to inform the business of past and current performance and to support strategic decisions. But we can also drown in a flood of data. Without a clear strategy for business intelligence, a promising new solution will produce only noise.

BI and Analytics teams must provide the right quantitative and qualitative insights for the business to base their decisions on.

Your Business Intelligence and Analytics strategy must support the organization’s strategy. Your strategy for BI & Analytics provides direction and requirements for data warehousing and data integration, and further paves the way for predictive analytics, big data analytics, market/industry intelligence, and social network analytics.

Dirk Coetsee,

Director, Data and Analytics Info-Tech Research Group

Our understanding of the problem

This Research is Designed For:

• A CIO or Business Unit (BU) Leader looking to improve reporting and analytics, reduce time to information, and embrace fact-based decision making with analytics, reporting, and business intelligence (BI).
• Application Directors experiencing poor results from an initial BI tool deployment who are looking to improve the outcome.

This Research Will Also Assist:

• Project Managers and Business Analysts assigned to a BI project team to collect and analyze requirements.
• Business units that have their own BI platforms and would like to partner with IT to take their BI to an enterprise level.

This Research Will Help You:

• Align your reporting and analytics strategy with the business’ strategic objectives before you rebuild or buy your Business Intelligence platform.
• Identify reporting and analytics objectives to inform the data warehouse and integration requirements gathering process.
• Avoid common pitfalls that derail BI and analytic deployments and lower their adoption.
• Identify Business Intelligence gaps prior to deployment and incorporate remedies within your plans.

This Research Will Help Them:

• Recruit the right resources for the program.
• Align BI with corporate vision, mission, goals, and strategic direction.
• Understand the needs of business partners.
• Assess BI maturity and plan for target state.
• Develop a BI strategy and roadmap.
• Track the success of the BI initiative.

Executive summary

Situation:

BI drives a new reality. Uber is the world’s largest taxi company and they own no vehicles; Alibaba is the world’s most valuable retailer and they have no inventory; Airbnb is the world’s largest accommodation provider and they own no real estate. How did they disrupt their markets and get past business entry barriers? A deep understanding of their market through impeccable business intelligence!

Complication:

• In respect to BI matureness, you can’t expect the whole organization to be at the same place at the same time. Your BI strategy needs to recognize this and should strive to align rather than dictate.
• Technology is just one aspect of your BI and Analytics strategy and is not a quick solution or a guarantee for long term success.

Resolution:

• Drive strategy development by establishing the business context upfront in order to align business intelligence providers with the most important needs of their BI consumers and the strategic priorities of the organization.
• Revamp or create a BI strategy to update your BI program to make it fit for purpose.
• Understand your existing BI baggage – e.g. your existing BI program, the artifacts generated from the program, and the users it supports. Those will inform the creation of the strategy and roadmap.
• Assess current BI maturity and determine your future state BI maturity.
• BI needs governance to ensure consistent planning, communication, and execution of the BI strategy.
• Create a network of BI ambassadors across the organization to promote BI.
• Plan for the future to ensure that required data will be available when the organization needs it.

Info-Tech Insight

1. Put the “B” back in BI. Don’t have IT doing BI for IT’s sake; ensure the voice and needs of the business are the primary drivers of your strategy.
2. The BI strategy drives data warehouse and integration strategies and the data needs to support business decisions.
3. Go beyond the platform. The solution to better BI often lies in improving the BI practice, not acquiring the latest and greatest tool.

Metrics to track BI & Analytical program progress

Goals for BI:

• Understand business context and needs. Identify business processes that can leverage BI.
• Define the Reporting & Analytics Roadmap. Develop data initiatives, and create a strategy and roadmap for Business Intelligence.
• Continuous improvements. Your BI program is evolving and improving over time. The program should allow you to have faster, better, and more comprehensive information.

Info-Tech’s Suggested Metrics for Tracking the BI Program

Intangible Metrics:

Tap into the results of Info-Tech’s CIO Business Vision diagnostic to monitor the changes in business-user satisfaction as you implement the initiatives in your BI improvement roadmap.

Your Enterprise BI and Analytics Strategy is driven by your organization’s Vision and Corporate Strategy

Formulating an Enterprise Reporting and Analytics Strategy requires the business vision and strategies to first be substantiated. Any optimization to the Data Warehouse, Integration and Source layer is in turn driven by the Enterprise Reporting and Analytics Strategy

The current state of your Integration and Warehouse platforms determine what data can be utilized for BI and Analytics

Where we are, and how we got here

How we got here

• In the beginning was BI 1.0. Business intelligence began as an IT-driven centralized solution that was highly governed. Business users were typically the consumers of reports and dashboards created by IT, an analytics-trained minority, upon request.
• In the last five to ten years, we have seen a fundamental shift in the business intelligence and analytics market, moving away from such large-scale, centralized IT-driven solutions focused on basic reporting and administration, towards more advanced user-friendly data discovery and visualization platforms. This has come to be known as BI 2.0.
• Many incumbent market leaders were disrupted by the demand for more user-friendly business intelligence solutions, allowing “pure-play” BI software vendors to carve out a niche and rapidly expand into more enterprise environments.
• BI-on-the-cloud has established itself as a solid alternative to in-house implementation and operation.

Where we are now

• BI 3.0 has arrived. This involves the democratization of data and analytics and a predominantly app-centric approach to BI, identifiable by an anywhere, anytime, and device-or-platform-independent collaborative methodology. Social workgroups and self-guided content creation, delivery, analysis, and management is prominent.
• Where the need for reporting and dashboards remains, we’re seeing data discovery platforms fulfilling the needs of non-technical business users by providing easy-to-use interactive solutions to increase adoption across enterprises.
• With more end users demanding access to data and the tools to extract business insights, IT is looking to meet these needs while continuing to maintain governance and administration over a much larger base of users. The race for governed data discovery is heated and will be a market differentiator.
• The next kid on the block is Artificial Intelligence that put further demands on data quality and availability.

RICOH Canada used this methodology to develop their BI strategy in consultation with their business stakeholders

CASE STUDY

Industry: Manufacturing and Retail

Source: RICOH

Ricoh Canada transforms the way people work with breakthrough technologies that help businesses innovate and grow. Its focus has always been to envision what the future will look like so that it can help its customers prepare for success. Ricoh empowers digital workplaces with a broad portfolio of services, solutions, and technologies – helping customers remove obstacles to sustained growth by optimizing the flow of information and automating antiquated processes to increase workplace productivity. In their commitment towards a customer-centric approach, Ricoh Canada recognized that BI and analytics can be used to inform business leaders in making strategic decisions.

Enterprise BI and analytics Initiative

Ricoh Canada enrolled in the ITRG Reporting & Analytics strategy workshop with the aim to create a BI strategy that will allow the business to harvest it strengths and build for the future. The workshop acted as a forum for the different business units to communicate, share ideas, and hear from each other what their pains are and what should be done to provide a full customer 360 view.

Results

“This workshop allowed us to collectively identify the various stakeholders and their unique requirements. This is a key factor in the development of an effective BI Analytics tool.” David Farrar

The Customer 360 Initiative included the following components

Improve BI Adoption Rates

Sisense

Reasons for low BI adoption

• Employees that never used BI tools are slow to adopt new technology.
• Lack of trust in data leads to lack of trust in the insights.
• Complex data structures deter usage due to long learning curves and contained nuances.
• Difficult to translate business requirements into tool linguistics due to lack of training or technical ineptness.
• Business has not taken ownership of data, which affects access to data.

How to foster BI adoption

• Senior management proclaim data as a strategic asset and involved in the promotion of BI
• Role Requirement that any business decision should be backed up by analytics
• Communication of internal BI use case studies and successes
• Exceptional data lineage to act as proof for the numbers
• A Business Data glossary with clearly defined business terms. Use the Business Data Glossary in conjunction with data lineage and semantic layers to ensure that businesses are clearly defined and traced to sources.
• Training in business to take ownership of data from inception to analytics.

Why bother with analytics?

In today’s ever-changing and global environment, organizations of every size need to effectively leverage their data assets to facilitate three key business drivers: customer intimacy, product/service innovation, and operational excellence. Plus, they need to manage their operational risk efficiently.

Investing in a comprehensive business intelligence strategy allows for a multidimensional view of your organization’s data assets that can be operationalized to create a competitive edge:

Historical Data

Without a BI strategy, creating meaningful reports for business users that highlight trends in past performance and draw relationships between different data sources becomes a more complex task. Also, the ever growing need to identify and assess risks in new ways is driving many companies to BI.

Data Democracy

The core purpose of BI is to provide the right data, to the right users, at the right time, and in a format that is easily consumable and actionable. In developing a BI strategy, remember the driver for managed cross-functional access to data assets and features such as interactive dashboards, mobile BI, and self-service BI.

Predictive and Big Data Analytics

As the volume, variety, and velocity of data increases rapidly, businesses will need a strategy to outline how they plan to consume the new data in a manner that does not overwhelm their current capabilities and aligns with their desired future state. This same strategy further provides a foundation upon which organizations can transition from ad hoc reporting to using data assets in a codified BI platform for decision support.

Business intelligence serves as the layer that translates data, information, and organizational knowledge into insights

As executive decision making shifts to more fact-based, data-driven thinking, there is an urgent need for data assets to be organized and presented in a manner that enables immediate action.

Typically, business decisions are based on a mix of intuition, opinion, emotion, organizational culture, and data. Though business users may be aware of its potential value in driving operational change, data is often viewed as inaccessible.

Business intelligence bridges the gap between an organization’s data assets and consumable information that facilitates insight generation and informed decision making.

Most organizations realize that they need a BI strategy; it’s no longer a nice-to-have, it’s a must-have.

– Albert Hui, Principal, Data Economist

Business intelligence and business analytics: what is the difference and should you care

Ask 100 people and you will get 100 answers. We like the prevailing view that BI looks at today and backward for improving who we are, while BA is forward-looking to support change decisions.

• Business intelligence is concerned with looking at present and historical data.
• Use this data to create reports/dashboards to inform a wide variety of information consumers of the past and current state of affairs.
• Almost all organizations, regardless of size and maturity, use some level of BI even if it’s just very basic reporting.
• Business analytics, on the other hand, is a forward-facing use of data, concerned with the present to the future.
• Analytics uses data to both describe the present, and more importantly, predict the future, enabling strategic business decisions.
• Although adoption is rapidly increasing, many organizations still do not utilize any advanced analytics in their environment.

However, establishing a strong business intelligence program is a necessary precursor to an organization’s development of its business analytics capabilities.

Organizations that successfully grow their BI capabilities are reaping the rewards

Evidence is piling up: if planned well, BI contributes to the organization’s bottom line.

It’s expected that there will be nearly 45 billion connected devices and a 42% increase in data volume each year posing a high business opportunity for the BI market (BERoE, 2020).

The global business intelligence market size to grow from US$23.1 billion in 2020 to US$33.3 billion by 2025, at a compound annual growth rate (CAGR) of 7.6% (Global News Wire, 2020)

In the coming years, 69% of companies plan on increasing their cloud business intelligence usage (BARC Research and Eckerson Group Study, 2017).

Call to Action

Small organizations of up to 100 employees had the highest rate of business intelligence penetration last year (Forbes, 2018).

Source: IBM Business Value, 2015

For the New England Patriots, establishing a greater level of customer intimacy was driven by a tactical analytics initiative

CASE STUDY

Industry: Professional Sports

Source Target Marketing

Problem

Despite continued success as a franchise with a loyal fan base, the New England Patriots experienced one of their lowest season ticket renewal rates in over a decade for the 2009 season. Given the numerous email addresses that potential and current season-ticket holders used to engage with the organization, it was difficult for Kraft Sports Group to define how to effectively reach customers.

Turning to a Tactical Analytics Approach

Kraft Sports Group turned to the customer data that it had been collecting since 2007 and chose to leverage analytics in order to glean insight into season ticket holder behavior. By monitoring and reporting on customer activity online and in attendance at games, Kraft Sports Group was able to establish that customer engagement improved when communication from the organization was specifically tailored to customer preferences and historical behavior.

Results

By operationalizing their data assets with the help of analytics, the Patriots were able to achieve a record 97% renewal rate for the 2010 season. KSG was able to take their customer engagement to the next level and proactively look for signs of attrition in season-ticket renewals.

We're very analytically focused and I consider us to be the voice of the customer within the organization… Ultimately, we should know when renewal might not happen and be able to market and communicate to change that behavior.

– Jessica Gelman,

VP Customer Marketing and Strategy, Kraft Sports Group

A large percentage of all BI projects fail to meet the organization’s needs; avoid falling victim to common pitfalls

Tool Usage Pitfalls

• Business units are overwhelmed with the amount and type of data presented.
• Poor data quality erodes trust, resulting in a decline in usage.
• Analysis performed for the sake of analysis and doesn’t focus on obtaining relevant business-driven insights.

Selection Pitfalls

• Inadequate requirements gathering.
• No business involvement in the selection process.
• User experience is not considered.
• Focus is on license fees and not total cost.

Implementation Pitfalls

• Absence of upfront planning
• Lack of change management to facilitate adoption of the new platform
• No quick wins that establish the value of the project early on
• Inadequate initial or ongoing training

Strategic Pitfalls

• Poor alignment of BI goals with organization goals
• Absence of CSFs/KPIs that can measure the qualitative and quantitative success of the project
• No executive support during or after the project

BI pitfalls are lurking around every corner, but a comprehensive strategy drafted upfront can help your organization overcome these obstacles. Info-Tech’s approach to BI has involvement from the business units built right into the process from the start and it equips IT to interact with key stakeholders early and often.

Only 62% of Big Data and AI projects in 2019 provided measurable results.

Source: NewVantage Partners LLC

Business and IT have different priorities for a BI tool

Business executives look for:

• Ease of use
• Speed and agility
• Clear and concise information
• Sustainability

IT professionals are concerned about:

• Solid security
• Access controls on data
• Compliance with regulations
• Ease of integration

Info-Tech Insight

Combining these priorities will lead to better tool selection and more synergy.

Elizabeth Mazenko

The top-down BI Opportunity Analysis is a tool for senior executives to discover where Business Intelligence can provide value

Example: Uncover BI opportunities with an opportunity analysis

Williams (2016)

Bridge the gap between business drivers and business intelligence features with a three-tiered framework

Info-Tech’s approach to formulating a fit-for-purpose BI strategy is focused on making the link between factors that are the most important to the business users and the ways that BI providers can enable those consumers.

Drivers to Establish Competitive Advantage

• Operational Excellence
• Client Intimacy
• Innovation

BI and Analytics Spectrum

• Strategic Analytics
• Tactical Analytics
• Operational Analytics

Info-Tech’s BI Patterns

• Delivery
• User Experience
• Deep Analytics
• Supporting

This is the content for Layout H3 Tag

Though business intelligence is primarily thought of as enabling executives, a comprehensive BI strategy involves a spectrum of analytics that can provide data-driven insight to all levels of an organization.

Recommended

Strategic Analytics

• Typically focused on predictive modeling
• Leverages data integrated from multiple sources (structured through unstructured)
• Assists in identifying trends that may shift organizational focus and direction
• Sample objectives:
• Drive market share growth
• Identify new markets, products, services, locations, and acquisitions
• Build wider and deeper customer relationships earning more wallet share and keeping more customers

Tactical Analytics

• Often considered Response Analytics and used to react to situations that arise, or opportunities at a department level.
• Sample objectives:
• Staff productivity or cost analysis
• Heuristics/algorithms for better risk management
• Product bundling and packaging
• Customer satisfaction response techniques

Operational Analytics

• Analytics that drive business process improvement whether internal, with external partners, or customers.
• Sample objectives:
• Process step elimination
• Best opportunities for automation

Business Intelligence Terminology

Williams (2016)

"BI can be confusing and overwhelming…"

– Dirk Coetsee,

Research Director,

Info-Tech Research Group

Business intelligence lies in the Information Dimensions layer of Info-Tech’s Data Management Framework

The interactions between the information dimensions and overlying data management enablers such as data governance, data architecture, and data quality underscore the importance of building a robust process surrounding the other data practices in order to fully leverage your BI platform.

Within this framework BI and analytics are grouped as one lens through which data assets at the business information level can be viewed.

Use Info-Tech’s three-phase approach to a Reporting & Analytics strategy and roadmap development

Project Insight

A BI program is not a static project that is created once and remains unchanged. Your strategy must be treated as a living platform to be revisited and revitalized in order to effectively enable business decision making. Develop a reporting and analytics strategy that propels your organization by building it on business goals and objectives, as well as comprehensive assessments that quantitatively and qualitatively evaluate your current reporting and analytical capabilities.

Stand on the shoulders of Information Management giants

As part of our research process, we leveraged the frameworks of COBIT5, Mike 2.0, and DAMA DMBOK2. Contextualizing business intelligence within these frameworks clarifies its importance and role and ensures that our assessment tool is focused on key priority areas.

The DMBOK2 Data Management framework by the Data Asset Management Association (DAMA) provided a starting point for our classification of the components in our IM framework.

Mike 2.0 is a data management framework that helped guide the development of our framework through its core solutions and composite solutions.

The Cobit 5 framework and its business enablers were used as a starting point for assessing the performance capabilities of the different components of information management, including business intelligence.

Info-Tech has a series of deliverables to facilitate the evolution of your BI strategy

BI Strategy Roadmap Template

BI Practice Assessment Tool

BI Initiatives and Roadmap Tool

BI Strategy and Roadmap Executive Presentation Template

Info-Tech offers various levels of support to best suit your needs

Diagnostics and consistent frameworks used throughout all four options

Build a Reporting and Analytics Strategy – Project Overview

Workshop overview

Contact your account representative or email Workshops@InfoTech.com for more information.

Phase 2

Understand the Business Context and BI Landscape

Build a Reporting and Analytics Strategy

Phase 1 overview

Detailed Overview

Step 1: Establish the business context in terms of business vision, mission, objectives, industry drivers, and business processes that can leverage Business Intelligence

Step 2: Understand your BI Landscape

Step 3: Understand business needs

Outcomes

• Clearly articulated high-level mission, vision, and key drivers from the business, as well as objectives related to business intelligence.
• In-depth documentation regarding your organization’s BI usage, user perception, and outputs.
• Consolidated list of requirements, existing and desired, that will direct the deployment of your BI solution.

Benefits

• Align business context and drivers with IT plans for BI and Analytics improvement.
• Understand your current BI ecosystem’s performance.

Understand your business context and BI landscape

Phase 1 Overarching Insight

The closer you align your new BI platform to real business interests, the stronger the buy-in, realized value, and groundswell of enthusiastic adoption will be. Get this phase right to realize a high ROI on your investment in the people, processes, and technology that will be your next generation BI platform.

Track these metrics to measure your progress through Phase 1

Goals for Phase 1:

• Understand the business context. Determine if BI can be used to improve business outcomes by identifying benefits, costs, opportunities, and gaps.
• Understand your existing BI. Plan your next generation BI based on a solid understanding of your existing BI.
• Identify business needs. Determine the business processes that can leverage BI and Analytics.

Info-Tech’s Suggested Metrics for Tracking Phase 1 Goals

Intangible Metrics:

1. Executives’ understanding of the BI program and what BI can do for the organization.
2. Improved trust between IT and the business by re-opening the dialogue.
3. Closer alignment with the organization strategy and business plan leading to higher value delivered.
4. Increased business engagement and input into the Analytics strategy.

Use advisory support to accelerate your completion of Phase 1 activities

Call 1-888-670-8889 or email GuidedImplementations@InfoTech.com for more information.

Complete these steps on your own, or call us to complete a guided implementation. A guided implementation is a series of two to three advisory calls that help you execute each phase of a project. They are included in most advisory memberships.

Guided Implementation 1: Understand the Business Context and BI Landscape

Proposed Time to Completion: 2-4 weeks

Step 1.0: Assemble Your Project Team

Start with an analyst kick-off call:

• Discuss Info-Tech’s viewpoint and definitions of business intelligence.
• Discuss the project sponsorship, ideal team members and compositions.

Then complete these activities…

• Identify a project sponsor and the project team members.

Step 1.1: Understand Your Business Context

Start with an analyst kick-off call:

• Discuss Info-Tech’s approach to BI strategy development around using business information as the key driver.

Then complete these activities…

• Detail the business context (vision, mission, goals, objectives, etc.).
• Establish business–IT alignment for your BI strategy by detailing the business context.

Step 1.2: Establish the Current BI Landscape

Review findings with analyst:

• Review the business context outputs from Step 1.1 activities.
• Review Info-Tech’s approach for documenting your current BI landscape.
• Review the findings of your BI landscape.

Then complete these activities…

• Gather information on current BI usage and perform a BI artifact inventory.
• Construct and conduct a user perception survey.

With these tools & templates:

BI Strategy and Roadmap Template

Step 1.0

Assemble the Project Team

Select a BI project sponsor

Info-Tech recommends you select a senior executive with close ties to BI be the sponsor for this project (e.g. CDO, CFO or CMO). To maximize the chance of success, Info-Tech recommends you start with the CDO, CMO, CFO, or a business unit (BU) leader who represents strategic enterprise portfolios.

Initial Sponsor

CFO or Chief Risk Officer (CRO)

• The CFO is responsible for key business metrics and cost control. BI is on the CFO’s radar as it can be used for both cost optimization and elimination of low-value activity costs.
• The CRO is tasked with the need to identify, address, and when possible, exploit risk for business security and benefit.
• Both of these roles are good initial sponsors but aren’t ideal for the long term.

CDO or a Business Unit (BU) Leader

• The CDO (Chief Data Officer) is responsible for enterprise-wide governance and utilization of information as an asset via data processing, analysis, data mining, information trading, and other means, and is the ideal sponsor.
• BU leaders who represent a growth engine for a company look for ways to mine BI to help set direction.

Ultimate Sponsor

CEO

• As a the primary driver of enterprise-wide strategy, the CEO is the ideal evangelist and project sponsor for your BI strategy.
• Establishing a CEO–CIO partnership helps elevate IT to the level of a strategic partner, as opposed to the traditional view that IT’s only job is to “keep the lights on.”
• An endorsement from the CEO may make other C-level executives more inclined to work with IT and have their business unit be the starting point for growing a BI program organically.

"In the energy sector, achieving production KPIs are the key to financial success. The CFO is motivated to work with IT to create BI applications that drive higher revenue, identify operational bottlenecks, and maintain gross margin."

– Yogi Schulz, Partner, Corvelle Consulting

Select a BI project team

Create a project team with the right skills, experience, and perspectives to develop a comprehensive strategy aligned to business needs.

You may need to involve external experts as well as individuals within the organization who have the needed skills.

A detailed understanding of what to look for in potential candidates is essential before moving forward with your BI project.

Leverage several of Info-Tech’s Job Description Templates to aid in the process of selecting the right people to involve in constructing your BI strategy.

Roles to Consider

Business Stakeholders

Business Intelligence Specialist

Business Analyst

Data Mining Specialist

Data Warehouse Architect

Enterprise Data Architect

Data Steward

"In developing the ideal BI team, your key person to have is a strong data architect, but you also need buy-in from the highest levels of the organization. Buy-in from different levels of the organization are indicators of success more than anything else."

– Rob Anderson, Database Administrator and BI Manager, IT Research and Advisory Firm

Create a RACI matrix to clearly define the roles and responsibilities for the parties involved

A common project management pitfall for any endeavour is unclear definition of responsibilities amongst the individuals involved.

As a business intelligence project requires a significant amount of back and forth between business and IT – bridged by the BI Steering Committee – clear guidelines at the project outset with a RACI chart provide a basic framework for assigning tasks and lines of communication for the later stages.

Responsible Accountable Consulted Informed

Note: This RACI is an example of how role expectations would be broken down across the different steps of the project. Develop your own RACI based on project scope and participants.

STEP 1.1

Understand Your Business Context and Structure the Project

Establish business–IT alignment for your BI strategy by detailing the business context

Step Objectives

• Engage the business units to find out where users need BI enablement.
• Ideate preliminary points for improvement that will further business goals and calculate their value.

Step Activities

1.1.1 Craft the vision and mission statements for the Analytics program using the vision, mission, and strategies of your organization as basis.

1.1.2 Articulate program goals and objectives

1.1.3 Determine business differentiators and key drivers

1.1.4 Brainstorm BI-specific constraints and improvement objectives

Outcomes

• Clearly articulated business context that will provide a starting point for formulating a BI strategy
• High-level improvement objectives and ROI for the overall project
• Vision, mission, and objectives of the analytics program

Research Support

• Info-Tech’s BI Strategy and Roadmap Template

Proposed Participants in this Step

• Project Manager
• Project Team
• Relevant Business Stakeholders and Subject Matter Experts

Transform the way the business makes decisions

Your BI strategy should enable the business to make fast, effective, and comprehensive decisions.

"We can improve BI environments in several ways. First, we can improve the speed with which we create BI objects by insisting that the environments are designed with flexibility and adaptability in mind. Second, we can produce higher quality deliverables by ensuring that IT collaborate with the business on every deliverable. Finally, we can reduce the costs of BI by giving access to the environment to knowledgeable business users and encouraging a self-service function."

– Claudia Imhoff, Founder, Boulder BI Brain Trust, Intelligent Solutions Inc.

Assess needs of various stakeholders using personas

User groups/user personas

Different users have different consumption and usage patterns. Categorize users into user groups and visualize the usage patterns. The user groups are the connection between the BI capabilities and the users.

The pains of inadequate BI are felt across the entire organization – and land squarely on the shoulders of the CIO

Organization:

• Insufficient information to make decisions.
• Unable to measure internal performance.
• Losses incurred from bad decisions or delayed decisions.
• Canned reports fail to uncover key insights.
• Multiple versions of information exist in silos.

IT Department

• End users are completely dependent on IT for reports.
• Ad hoc BI requests take time away from core duties.
• Spreadsheet-driven BI is overly manual.
• Business losing trust in IT.

CIO

• Under great pressure and has a strong desire to improve BI.
• Ad hoc BI requests are consuming IT resources and funds.
• My organization finds value in using data and having decision support to make informed decisions.

The overarching question that needs to be continually asked to create an effective BI strategy is:

How do I create an environment that makes information accessible and consumable to users, and facilitates a collaborative dialogue between the business and IT?

Pre-requisites for success

Prerequisite #1: Secure Executive Sponsorship

Sponsorship of BI that is outside of IT and at the highest levels of the organization is essential to the success of your BI strategy. Without it, there is a high chance that your BI program will fail. Note that it may not be an epic fail, but it is a subtle drying out in many cases.

Prerequisite #2: Understand Business Context

Providing the right tools for business decision making doesn’t need to be a guessing game if the business context is laid as the project foundation and the most pressing decisions serve as starting points. And business is engaged in formulating and executing the strategy.

Prerequisite #3: Deliver insights that lead to action

Mico Yuk, 2019

Make it clear to the business that IT is committed to building and supporting a BI platform that is intimately tied to enabling changing business objectives.

Leverage Info-Tech’s BI Strategy and Roadmap Template to accelerate BI planning

How to accelerate BI planning using the template

1. Prepopulated text that you can use for your strategy formulation:

2. Sample bullet points that you can pick and choose from:

Document the BI program planning in Info-Tech’s

BI Strategy and Roadmap Template.

Activity: Describe your organization’s vision and mission

1.1.1

30-40 minutes

Compelling vision and mission statements will help guide your internal members toward your company’s target state. These will drive your business intelligence strategy.

1. Your vision clearly represents where your organization aspires to be in the future and aligns the entire organization. Write down a future-looking, inspirational, and realizable vision in one concise statement. Consider:

• “Five years from now, our business will be _______.”
• What do we want to do tomorrow? For whom? What is the benefit?

• Why does the business exist? What problems does it solve? Who are its customers?
• How does the business accomplish strategic tasks or reach its target?

Input

• Business vision and mission statements

Output

• Alignment and understanding on business vision

Materials

• Corporate Vision, Mission, and Goals section in the BI Strategy and Roadmap Template

Participants

• BI project lead
• Executive business stakeholders

Info-Tech Insight

Adjust your statements until you feel that you can elicit a firm understanding of both your vision and mission in three minutes or less.

Formulating an Enterprise BI and Analytics Strategy: Top-down BI Opportunity analysis

Example of deriving BI opportunities using BI Opportunity Analysis

Williams 2016

Get your organization buzzing about BI – leverage Info-Tech’s Executive Brief as an internal marketing tool

Two key tasks of a project sponsor are to:

1. Evangelize the realizable benefits of investing in a business intelligence strategy.
2. Help to shift the corporate culture to one that places emphasis on data-driven insight.

Arm your project sponsor with our Executive Brief for this blueprint as a quick way to convey the value of this project to potential stakeholders.

Bolster this presentation by adding use cases and metrics that are most relevant to your organization.

Develop a business framework

Identifying organizational goals and how data can support those goals is key to creating a successful BI & Analytical strategy. Rounding out the business model with technology drivers, environmental factors (as described in previous steps), and internal barriers and enablers creates a holistic view of Business Intelligence within the context of the organization as a whole.

Through business engagement and contribution, the following holistic model can be created to understand the needs of the business.

Activity: Describe the Industry Drivers and Organization strategy to mitigate the risk

1.1.2

30-45 minutes

Industry drivers are external influencers that has an effect on a business such as economic conditions, competitor actions, trade relations, climate etc. These drivers can differ significantly by industry and even organizations within the same industry.

1. List the industry drivers that influences your organization:

• Public sentiment in regards to energy source
• Rising cost of raw materials due to increase demand

• Change production process to become more energy efficient
• Win at customer service

• Strategic cost management
• Operational Excellence

• Purchasing
• Sales

• Cost and financial analysis
• Customer service analysis

Input

• Industry drivers

Output

• BI Opportunities that business can leverage

Materials

• Industry driver section in the BI Strategy and Roadmap Template

Participants

• BI project lead
• Executive business stakeholders

Understand BI and analytics drivers and organizational objectives

Activity: Discuss BI/Analytics drivers and organizational objectives

1.1.3

30-45 minutes

1. Use the industry drivers and business goals identified in activity 1.1.2 as a starting point.
2. Understand how the company runs today and what the organization’s future will look like. Try to identify the purpose for becoming an integrated organization. Use a whiteboard and markers to capture key findings.
3. Take into account External Considerations, Organizational Drivers, Technology Drivers, and Key Functional Requirements.

Identify challenges and barriers to the BI project

There are several factors that may stifle the success of a BI implementation. Scan the current environment to identify internal barriers and challenges to identify potential challenges so you can meet them head-on.

Common Internal Barriers

Management Support

Organizational Culture

Organizational Structure

IT Readiness

Activity: Discuss BI/Analytics challenges and pain points

1.1.4

30-45 minutes

1. Identify challenges with the process identified in step 1.1.2.
2. Brainstorm potential barriers to successful BI implementation and adoption. Use a whiteboard and marker to capture key findings.
3. Consider Functional Gaps, Technical Gaps, Process Gaps, and Barriers to BI Success.

Activity: Discuss opportunities and benefits

1.1.5

30-45 minutes

1. Identify opportunities and benefits from an integrated system.
2. Brainstorm potential enablers for successful BI implementation and adoption. Use a whiteboard and markers to capture key findings.
3. Consider Business Benefits, IT Benefits, Organizational Benefits, and Enablers of BI success.

Your organization’s framework for Business Intelligence Strategy

Example: Business Framework for Data & Analytics Strategy

The following diagram represents [Client]’s business model for BI and data. This holistic view of [Client]’s current environment serves as the basis for the generation of the business-aligned Data & Analytics Strategy.

Info-Tech recommends balancing a top-down approach with bottom up for building your BI strateg

Taking a top-down approach will ensure senior management’s involvement and support throughout the project. This ensures that the most critical decisions are supported by the right data/information, aligning the entire organization with the BI strategy. Furthermore, the gains from BI will be much more significant and visible to the rest of the organization.

Far too often, organizations taking a bottom-up approach to BI will fail to generate sufficient buy-in and awareness from senior management. Not only does a lack of senior involvement result in lower adoption from the tactical and operational levels, but more importantly, it also means that the strategic decision makers aren’t taking advantage of BI.

Estimate the ROI of your BI and analytics strategy to secure executive support

The value of creating a new strategy – or revamping an existing one – needs to be conveyed effectively to a high-level stakeholder, ideally a C-level executive. That executive buy-in is more likely to be acquired when effort has been made to determine the return on investment for the overall initiative.

1. Business Impacts
   New revenue
   Cost savings
   Time to market

   Internal Benefits
   Productivity gain
   Process optimization

   Investment
   People – employees’ time, external resources
   Data – cost for new datasets
   Technology – cost for new technologies

2. QuantifyCan you put a number or a percentage to the impacts and benefits? QuantifyCan you estimate the investments you need to put in?
3. TranslateTranslate the quantities into dollar value
4. 

Example

Activity: Establish a high-level ROI as part of an overall use case for developing a fit-for-purpose BI strategy

1.1.6

1.5 hours

Communicating an ROI that is impactful and reasonable is essential for locking in executive-level support for any initiative. Use this activity as an initial touchpoint to bring business and IT perspectives as part of building a robust business case for developing your BI strategy.

1. Revisit the business context detailed in the previous sections of this phase. Use priority objectives to identify use case(s), ideally where there are easily defined revenue generators/cost reductions (e.g. streamlining the process of mailing physical marketing materials to customers).
2. Assign research tasks around establishing concrete numbers and dollar values.

• Have a subject matter expert weigh in to validate your figures.
• When calculating ROI, consider how you might leverage BI to create opportunities for upsell, cross-sell, or increased customer retention.

• This is the point where expectation management is important. Separate the need-to-haves from the nice-to-haves.

Emphasize that ROI is not fully realized after the first implementation, but comes as the platform is built upon iteratively and in an integrated fashion to mature capabilities over time.

Input

• Vision statement
• Mission statement

Output

• Business differentiators and key drivers

Materials

• Benefit Cost Analysis section of the BI Strategy and Roadmap Template

Participants

• BI project lead
• Executive IT & business stakeholders

An effective BI strategy positions business intelligence in the larger data lifecycle

In an effort to keep users satisfied, many organizations rush into implementing a BI platform and generating reports for their business users. BI is, first and foremost, a presentation layer; there are several stages in the data lifecycle where the data that BI visualizes can be compromised.

Without paying the appropriate amount of attention to the underlying data architecture and application integration, even the most sophisticated BI platforms will fall short of providing business users with a holistic view of company information.

Example

In moving away from single application-level reporting, a strategy around data integration practices and technology is necessary before the resultant data can be passed to the BI platform for additional analyses and visualization.

BI doesn’t exist in a vacuum – develop an awareness of other key data management practices

As business intelligence is primarily a presentation layer that allows business users to visualize data and turn information into actionable decisions, there are a number of data management practices that precede BI in the flow of data.

Data Warehousing

The data warehouse structures source data in a manner that is more operationally focused. The Reporting & Analytics Strategy must inform the warehouse strategy on data needs and building a data warehouse to meet those needs.

Data Integration, MDM & RDM

The data warehouse is built from different sources that must be integrated and normalized to enable Business Intelligence. The Info-Tech integration and MDM blueprints will guide with their implementation.

Data Quality

A major roadblock to building an effective BI solution is a lack of accurate, timely, consistent, and relevant data. Use Info-Tech’s blueprint to refine your approach to data quality management.

Data quality, poor integration/P2P integration, poor data architecture are the primary barriers to truly leveraging BI, and a lot of companies haven’t gotten better in these areas.

– Shari Lava, Associate Vice-President, IT Research and Advisory Firm

Building consensus around data definitions across business units is a critical step in carrying out a BI strategy

Business intelligence is heavily reliant on the ability of an organization to mesh data from different sources together and create a holistic and accurate source of truth for users.

Useful analytics cannot be conducted if your business units define key business terms differently.

Example

Finance may label customers as those who have transactional records with the organization, but Marketing includes leads who have not yet had any transactions as customers. Neglecting to note these seemingly small discrepancies in data definition will undermine efforts to combine data assets from traditionally siloed functional units.

In the stages prior to implementing any kind of BI platform, a top priority should be establishing common definitions for key business terms (customers, products, accounts, prospects, contacts, product groups, etc.).

As a preliminary step, document different definitions for the same business terms so that business users are aware of these differences before attempting to combine data to create custom reports.

Self-Assessment

Do you have common definitions of business terms?

• If not, identify common business terms.
• At the very least, document different definitions of the same business terms so the corporate can compare and contrast them.

STEP 1.2

Assess the Current BI Landscape

Establish an in-depth understanding of your current BI landscape

Step Objectives

• Inventory and assess the state of your current BI landscape
• Document the artifacts of your BI environment

Step Activities

1.2.1 Analyze the usage levels of your current BI programs/platform

1.2.2 Perform a survey to gather user perception of your current BI environment

1.2.3 Take an inventory of your current BI artifacts

Outcomes

• Summarize the qualitative and quantitative performance of your existing BI environment
• Understand the outputs coming from your BI sources

Research Support

• Info-Tech’s BI Strategy and Roadmap Template

Project Manager

Data Architect(s) or Enterprise Architect

Project Team

Understand your current BI landscape before you rationalize

Relying too heavily on technology as the sole way to solve BI problems results in a more complex environment that will ultimately frustrate business users. Take the time to thoroughly assess the current state of your business intelligence landscape using a qualitative (user perception) and quantitative (usage statistics) approach. The insights and gaps identified in this step will serve as building blocks for strategy and roadmap development in later phases.

Phase 1

Current State Summary of BI Landscape

PHASE 2

Strategy and Roadmap Formulation

Gather usage insights to pinpoint the hot spots for BI usage amongst your users

Usage data reflects the consumption patterns of end users. By reviewing usage data, you can identify aspects of your BI program that are popular and those that are underutilized. It may present some opportunities for trimming some of the underutilized content.

Benefits of analyzing usage data:

• Usage is a proxy for popularity and usability of the BI artifacts. The popular content should be kept and improved in your next generation BI.
• Usage information provides insight on what, when, where, and how much users are consuming BI artifacts.
• Unlike methods such as user interviews and focus groups, usage information is fact based and is not subject to peer pressure or “toning down.”

Sample Sources of Usage Data:

1. Usage reports from your BI platform Many BI platforms have out-of-the-box usage reports that log and summarize usage data. This is your ideal source for usage data.
2. Administrator console in your BI platformBI platforms usually have an administrator console that allows BI administrators to configure settings and to monitor activities that include usage. You may obtain some usage data in the console. Note that the usage data is usually real-time in nature, and you may not have access to a historical view of the BI usage.

Info-Tech Insight

Don’t forget some of the power users. They may perform analytics by accessing datasets directly or with the help of a query tool (even straight SQL statements). Their usage information is important. The next generation BI should provide consumption options for them.

Accelerate the process of gathering user feedback with Info-Tech’s Application Portfolio Assessment (APA)

In an environment where multiple BI tools are being used, discovering what works for users and what doesn’t is an important first step to rationalizing the BI landscape.

Info-Tech’s Application Portfolio Assessment allows you to create a custom survey based on your current applications, generate a custom report that will help you visualize user satisfaction levels, and pinpoint areas for improvement.

Activity: Review and analyze usage data

1.2.1

2 hours

This activity helps you to locate usage data in your existing environment. It also helps you to review and analyze usage data to come up with a few findings.

1. Get to the usage source. You may obtain usage data from one of the below options. Usage reports are your ideal choice, followed by some alternative options:

a. Administrator console – limited to real-time or daily usage data. You may need to track usage data over for several days to identify patterns.

b. Info-Tech’s Application Portfolio Assessment (APA).

c. Other – be creative. Some may use an IT usage monitoring system or web analytics to track time users spent on the BI portal.

2. Develop categories for classifying the different sources of usage data in your current BI environment. Use the following table as starting point for creating these groups:

This is the content for Layout H4 Tag

INPUT

• Usage reports
• Usage statistics

OUTPUT

• Insights pertaining to usage patterns

Materials

• Usage Insights of the BI Strategy and Roadmap Template

Participants

• BA
• BI Administrator
• PM

Activity: Review and analyze usage (cont.)

1.2.1

2 hours

3. Sort your collection of BI artifacts by usage. Discuss some of the reasons why some content is popular whereas some has no usage at all.

Popular BI Artifacts – Discuss improvements, opportunities and new artifacts

Unpopular BI Artifacts – Discuss retirement, improvements, and realigning information needs

4. Summarize your findings in the Usage Insights section of the BI Strategy and Roadmap Template.

INPUT

• Usage reports
• Usage statistics

OUTPUT

• Insights pertaining to usage patterns

Materials

• Usage Insights section of the BI Strategy and Roadmap Template

Participants

• BA
• BI Administrator
• PM

Gather perception to understand the existing BI users

In 1.2.1, we gathered the statistics for BI usage; it’s the hard data telling who uses what. However, it does not tell you the rationale, or the why, behind the usage. Gathering user perception and having conversations with your BI consumers is the key to bridging the gap.

User Perception Survey

Helps you to:

1. Get general insights on user perception
2. Narrow down to selected areas

User Interviews

Perception can be gathered by user interviews and surveys. Conducting user interviews takes time so it is a good practice to get some primary insights via survey before doing in-depth interviews in selected areas.

– Shari Lava, Associate Vice-President, IT Research and Advisory Firm

Define problem statements to create proof-of-concept initiatives

Info-Tech’s Four Column Model of Data Flow

Find a data-related problem or opportunity

Ask open-ended discovery questions about stakeholder fears, hopes, and frustrations to identify a data-related problem that is clear, contained, and fixable. This is then to be written as a problem/opportunity statement.

1. Fear: What is the number one risk you need to alleviate?
2. Hope: What is the number one opportunity you wish to realize?
3. Frustration: What is the number one annoying pet peeve you wish to scratch?

Next, gather information to support a problem/opportunity statement:

4. What are your challenges in performing the activity or process today?
5. What does amazing look like if we solve this perfectly?
6. What other business activities/processes will be impacted/improved if we solve this?
7. What compliance/regulatory/policy concerns do we need to consider in any solution?
8. What measures of success/change should we use to prove value of the effort (KPIs/ROI)?
9. What are the steps in the process/activity?
10. What are the applications/systems used at each step and from step to step?
11. What data elements are created, used, and/or transformed at each step?

Leverage Info-Tech’s BI survey framework to initiate a 360° perception survey

Info-Tech has developed a BI survey framework to help existing BI practices gather user perception via survey. The framework is built upon best practices developed by McLean & Company.

1. Communicate the survey
2. Create a survey
3. Conduct the survey
4. Collect and clean survey data
5. Analyze survey data
6. Conduct follow-up interviews
7. Identify and prioritize improvement initiatives

The survey takes a comprehensive approach by examining your existing BI practices through the following lenses:

360° Perception

Use Info-Tech’s BI End-User Satisfaction Survey Framework to develop a comprehensive BI survey tailored to your organization.

Activity: Develop a plan to gather user perception of your current BI program

1.2.2

2 hours

This activity helps you to plan for a BI perception survey and subsequent interviews.

1. Proper communication while conducting surveys helps to boost response rate. The project team should have a meeting with business executives to decide:

• The survey goals
• Which areas to cover
• Which trends and hypotheses you want to confirm
• Which pre-, during, and post-survey communications should be sent out

1. Organize the data in a useful format
2. Clean up the survey data when there are gaps
3. Summarize the data into a presentable/distributable format

Collectively, the project team and the BI consuming departments should review the presentation and discuss these items:

Misalignment

Opportunities

Inefficiencies

Trends

Need detailed interviews?

INPUT

• Usage information and analyses

OUTPUT

• User-perception survey

Materials

• Perception Insights section of the BI Strategy and Roadmap Template

Participants

• BA
• BI Administrator
• PM
• Business SMEs

Create a comprehensive inventory of your BI artifacts

Taking an inventory of your BI artifacts allows you to understand what deliverables have been developed over the years. Inventory taking should go beyond the BI content. You may want to include additional information products such as Excel spreadsheets, reports that are coming out of an Access database, and reports that are generated from front-end applications (e.g. Salesforce).

1. Existing Reports from BI platform

If you are currently using a BI platform, you have some BI artifacts (reports, scorecards, dashboards) that are developed within the platform itself.

• BI Usage Reports (refer to step 2.1) – if you are getting a comprehensive BI usage reports for all your BI artifacts, there is your inventory report too.
• BI Inventory Reports – Your BI platform may provide out-of-the-box inventory reports. You can use them as your inventory.
• If the above options are not feasible, you may need to manually create the BI inventory. You may build that from some of your existing BI documentations to save time.

• Data applications such as Salesforce, CRM, and ERP often provide reports as an out-of-the-box feature.
• Those reports only include data within their respective applications. However, this may present opportunities for integrating application data with additional data sources.

Activity: Inventory your BI artifacts

1.2.3

2+ hours

This activity helps you to inventory your BI information artifacts and other related information artifacts.

1. Define the scope of your inventory. Work with the project sponsor and CIO to define which sources should be captured in the inventory process. Consider: BI inventory, Excel spreadsheets, Access reports, and application reporting.
2. Define the depth of your inventory. Work with the project sponsor and CIO to define the level of granularity. In some settings, the artifact name and a short description may be sufficient. In other cases, you may need to document users and business logic of the artifacts.
3. Review the inventory results. Discuss findings and opportunities around the following areas:

Interpret your Inventory

INPUT

• Current BI artifacts and documents
• BI Type classification

OUTPUT

• Summary of BI artifacts

Materials

• BI Inventory Insights section of the BI Strategy and Roadmap Template

Participants

• BA
• Data analyst
• PM
• Project sponsor

Project sponsor

1.2.4

2+ hours

This activity helps you to inventory your BI by report type.

1. Classify BI artifacts by type. Use the BI Type tool to classify Work with the project sponsor and CIO to define which sources should be captured in the inventory process. Consider: BI inventory, Excel spreadsheets, Access reports, and application reporting.
2. Define the depth of your inventory. Work with the project sponsor and CIO to define the level of granularity. In some settings, the artifact name and a short description may be sufficient. In other cases, you may need to document users and business logic of the artifacts.
3. Review the inventory results. Discuss findings and opportunities around the following areas:

Interpretation of your Inventory

INPUT

• The BI Type as used by different business units
• Business BI requirements

OUTPUT

• Summary of BI type usage across the organization

Materials

• BI Inventory Insights section of the BI Strategy and Roadmap Template

Participants

• BA
• Data analyst
• PM
• Project sponsor

STEP 1.3

Undergo BI Requirements Gathering

Perform requirements gathering for revamping your BI environment

Step Objectives

• Create principles that will direct effective requirements gathering
• Create a list of existing and desired BI requirements

Step Activities

1.3.1 Create requirements gathering principles

1.3.2 Gather appropriate requirements

1.3.3 Organize and consolidate the outputs of requirements gathering activities

Outcomes

• Requirements gathering principles that are flexible and repeatable
• List of BI requirements

Research Support

• Info-Tech’s BI Strategy and Roadmap Template

Proposed Participants in this Step

Project Manager

Data Architect(s) or Enterprise Architect

Project Team

Business Users

Don’t let your new BI platform become a victim of poor requirements gathering

The challenges in requirements management often have underlying causes; find and eliminate the root causes rather than focusing on the symptoms.

Root Causes of Poor Requirements Gathering:

• Requirements gathering procedures exist but aren’t followed.
• There isn't enough time allocated to the requirements gathering phase.
• There isn't enough involvement or investment secured from business partners.
• There is no senior leadership involvement or mandate to fix requirements gathering.
• There are inadequate efforts put towards obtaining and enforcing sign off.

Outcomes of Poor Requirements Gathering:

• Rework due to poor requirements leads to costly overruns.
• Final deliverables are of poor quality and are implemented late.
• Predicted gains from deployed applications are not realized.
• There are low feature utilization rates by end users.
• Teams are frustrated within IT and the business.

Info-Tech Insight

Requirements gathering is the number one failure point for most development or procurement projects that don’t deliver value. This has been, and continues to be, the case as most organizations still don't get requirements gathering right. Overcoming organizational cynicism can be a major obstacle to clear when it is time to optimize the requirements gathering process.

Define the attributes of a good requirement to help shape your requirements gathering principles

A good requirement has the following attributes:

Other Considerations

Organizations can also track a requirement owner, rationale, priority level (must have vs. nice to have), and current status (approved, tested, etc.).

Info-Tech Insight

Requirements must be solution agnostic – they should focus on the underlying need rather than the technology required to satisfy the need.

Activity: Define requirements gathering principles

1.3.1

1 hour

1. Invite representatives from the project management office, project management team, and BA team, as well as some key business stakeholders.
2. Use the sample categories and principles in the table below as starting points for creating your own requirements gathering principles.
3. Document the requirements gathering principles in the BI Strategy and Roadmap Template.
4. Communicate the requirements gathering principles to the affected BI stakeholders.

Sample Principles to Start With

INPUT

• Existing requirement principles (if any)

OUTPUT

• Requirements gathering principles that can be revisited and reused

Materials

• Requirements Insights section of the BI Strategy and Roadmap Template

Participants

• BA Team
• PM
• Business stakeholders
• PMO

Info-Tech Insight

Turn requirements gathering principles into house rules. The house rules should be available in every single requirements gathering session and the participants should revisit them when there are disagreements, confusion, or silence.

Right-size your approach to BI requirements management

Info-Tech suggests four requirements management approaches based on project complexity and business significance. BI projects usually require the Strategic Approach in requirements management.

Requirements Management Process Explanations

Vary the modes used in eliciting requirements from your user base

Requirements Gathering Modes

Info-Tech has identified four effective requirements gathering modes. During the requirements gathering process, you may need to switch between the four gathering modes to establish a thorough understanding of the information needs.

Dream Mode

• Mentality: Let users’ imaginations go wild. The sky’s the limit.
• How it works: Ask users to dream up the ideal future state and ask how analytics can support those dreams.
• Limitations: Not all dreams can be fulfilled. A variety of constraints (budget, personnel, technical skills) may prevent the dreams from becoming reality.

Pain Mode

• Mentality: Users are currently experiencing pains related to information needs.
• How it works: Vent the pains. Allow end users to share their information pains, ask them how their pains can be relieved, then convert those pains to requirements.
• Limitations: Users are limited by the current situation and aren’t looking to innovate.

Decode Mode

• Mentality: Read the hidden messages from users. Speculate as to what the users really want.
• How it works: Decode the underlying messages. Be innovative to develop hypotheses and then validate with the users.
• Limitations: Speculations and hypothesis could be invalid. They may direct the users into some pre-determined directions.

Profile Mode

• Mentality: “I think you may want XYZ because you fall into that profile.”
• How it works: The information user may fall into some existing user group profile or their information needs may be similar to some existing users.
• Limitations: This mode doesn’t address very specific needs.

Supplement BI requirements with user stories and prototyping to ensure BI is fit for purpose

BI is a continually evolving program. BI artifacts that were developed in the past may not be relevant to the business anymore due to changes in the business and information usage. Revamping your BI program entails revisiting some of the BI requirements and/or gathering new BI requirements.

Three-Step Process for Gathering Requirements

Pain Mode: “I can’t access and manipulate data on my own...”

Decode Mode: Dig deeper: could this hint at a self-service use case?

Dream Mode: E.g. a sandbox area where I can play around with clean, integrated, well-represented data.

Profile Mode: E.g. another marketing analyst is currently using something similar.

ExampleMary has a spreadmart that keeps track of all campaigns. Maintaining and executing that spreadmart is time consuming.

Mary is asking for a mash-up data set that she can pivot on her own…

Upon reviewing the data and the prototype, Mary decided to use a heat map and included two more data points – tenure and lifetime value.

Identify which BI styles best meet user requirements

A spectrum of Business Intelligence solutions styles are available. Use Info-Tech’s BI Styles Tool to assess which business stakeholder will be best served by which style.

Activity: Gather BI requirements

1.3.2

2-6 hours

Using the approaches discussed on previous slides, start a dialogue with business users to confirm existing requirements and develop new ones.

1. Invite business stakeholders to a requirements gathering session.

For existing BI artifacts – Invite existing users of those artifacts.

For new BI development – Invite stakeholders at the executive level to understand the business operation and their needs and wants. This is especially important if their department is new to BI.

2. Discuss the business requirements. Systematically switch between the four requirements gathering modes to get a holistic view of the requirements.
3. Once requirements are gathered, organize them to tell a story. A story usually has these components:

INPUT

• Existing documentations on BI artifacts

OUTPUT

• Preliminary, uncategorized list of BI requirements

Materials

• Requirements Insights section of the BI Strategy and Roadmap Template

Participants

• BA team
• Business stakeholders
• Business SMEs
• BI developers

Clarify consumer needs by categorizing BI requirements

Requirements are too broad in some situations and too detailed in others. In the previous step we developed user stories to provide context. Now you need to define requirement categories and gather detailed requirements.

Considerations for Requirement Categories

Prioritize requirements to assist with solution modeling

Prioritization ensures that the development team focuses on the right requirements.

The MoSCoW Model of Prioritization

The MoSCoW model was introduced by Dai Clegg of Oracle UK in 1994.

Prioritization is the process of ranking each requirement based on its importance to project success. Hold a separate meeting for the domain SMEs, implementation SMEs, project managers, and project sponsors to prioritize the requirements list. At the conclusion of the meeting, each requirement should be assigned a priority level. The implementation SMEs will use these priority levels to ensure that efforts are targeted towards the proper requirements and the plan features available on each release. Use the MoSCoW Model of Prioritization to effectively order requirements.

Activity: Finalize the list of BI requirements

1.3.3

1-4 hours

Requirement Category Framework

Create requirement buckets and classify requirements.

1. Define requirement categories according to the framework.
2. Review the user story and requirements you collected in Step 1.3.2. Classify the requirements within requirement categories.
3. Review the preliminary list of categorized requirements and look for gaps in this detailed view. You may need to gather additional requirements to fill the gaps.
4. Prioritize the requirements according to the MoSCoW framework.
5. Document your final list of requirements in the BI Strategy and Roadmap Template.

INPUT

• Existing requirements and new requirements from step 1.3.2

OUTPUT

• Prioritized and categorized requirements

Materials

• Requirements Insights section of the BI Strategy and Roadmap Template

Participants

• BA
• Business stakeholders
• PMO

Translate your findings and ideas into actions that will be integrated into the BI Strategy and Roadmap Template

As you progress through each phase, document findings and ideas as they arise. At phase end, hold a brainstorming session with the project team focused on documenting findings and ideas and substantiating them into improvement actions.

Ask yourself how BI or analytics can be used to address the gaps and explore opportunities uncovered in each phase. For example, in Phase 1, how do current BI capabilities impede the realization of the business vision?

Document and prioritize Phase 1 findings, ideas, and action items

1.3.4

1-2 hours

1. Reconvene as a group to review findings, ideas, and actions harvested in Phase 1. Write the findings, ideas, and actions on sticky notes.
2. Prioritize the sticky notes to yield those with high business value and low implementation effort. View some sample findings below:

High Business Value, Low Effort	High Business Value, High Effort
Low Business Value, High Effort	Low Business Value, High Effort

Phase 1

Sample Phase 1 Findings	Found two business objectives that are not supported by BI/analytics
	Some executives still think BI is reporting
	Some confusion around operational reporting and BI
	Data quality plays a big role in BI
	Many executives are not sure about the BI ROI or asking for one

3. Select the top findings and document them in the “Other Phase 1 Findings” section of the BI Strategy and Roadmap Template. The findings will be used again in Phase 3.

INPUT

• Phase 1 activities
• Business context (vision, mission, goals, etc.

OUTPUT

• Other Phase 1 Findings section of the BI Strategy and Roadmap Template

Materials

• Whiteboard
• Sticky notes

Participants

• Project manger
• Project team
• Business stakeholders

If you want additional support, have our analysts guide you through this phase as part of an Info-Tech workshop

Book a workshop with our Info-Tech analysts:

• To accelerate this project, engage your IT team in an Info-Tech workshop with an Info-Tech analyst team.
• Info-Tech analysts will join you and your team onsite at your location or welcome you to Info-Tech’s historic Toronto office to participate in an innovative onsite workshop.
• Contact your account manager (www.infotech.com/account), or email Workshops@InfoTech.com for more information.

The following are sample activities that will be conducted by Info-Tech analysts with your team:

1.1.1-1.1.5

Establish the business context

To begin the workshop, your project team will be taken through a series of activities to establish the overall business vision, mission, objectives, goals, and key drivers. This information will serve as the foundation for discerning how the revamped BI strategy needs to enable business users.

1.2.1- 1.2.3

Create a comprehensive documentation of your current BI environment

Our analysts will take your project team through a series of activities that will facilitate an assessment of current BI usage and artifacts, and help you design an end-user interview survey to elicit context around BI usage patterns.

If you want additional support, have our analysts guide you through this phase as part of an Info-Tech workshop

Book a workshop with our Info-tech analysts

1.3.1-1.3.3

Establish new BI requirements

Our analysts will guide your project team through frameworks for eliciting and organizing requirements from business users, and then use those frameworks in exercises to gather some actual requirements from business stakeholders.

Phase 2

Evaluate Your Current BI Practice

Build a Reporting and Analytics Strategy

Revisit project metrics to track phase progress

Goals for Phase 2:

• Assess your current BI practice. Determine the maturity of your current BI practice from different viewpoints.
• Develop your BI target state. Plan your next generation BI with Info-Tech’s BI patterns and best practices.
• Safeguard your target state. Avoid BI pitfalls by proactively monitoring BI risks.

Info-Tech’s Suggested Metrics for Tracking Phase 2 Goals

Intangible Metrics:

• Prototyping approach allows the BI group to understand more about business requirements, and in the meantime, allows the business to understand how to partner with the BI group.
• The BI group and the business have more confidence in the BI program as risks are monitored and mitigated on an ad hoc basis.

Evaluate your current BI practice

Phase 2 Overarching Insight

BI success is not based solely on the technology it runs on; technology cannot mask gaps in capabilities. You must be capable in your environment, and data management, data quality, and related data practices must be strong. Otherwise, the usefulness of the intelligence suffers. The best BI solution does not only provide a technology platform, but also addresses the elements that surround the platform. Look beyond tools and holistically assess the maturity of your BI practice with input from both the BI consumer and provider perspectives.

Phase 2 overview

Detailed Overview

Step 1: Assess Your Current BI Practice

Step 2: Envision a Future State for Your BI Practice

Outcomes

• A comprehensive assessment of current BI practice maturity and capabilities.
• Articulation of your future BI practice.
• Improvement objectives and activities for developing your current BI program.

Benefits

• Identification of clear gaps in BI practice maturity.
• A current state assessment that includes the perspectives of both BI providers and consumers to highlight alignment and/or discrepancies.
• A future state is defined to provide a benchmark for your BI program.
• Gaps between the future and current states are identified; recommendations for the gaps are defined.

Phase 2 outline

Call 1-888-670-8889 or email GuidedImplementations@InfoTech.com for more information.

Complete these steps on your own, or call us to complete a guided implementation. A guided implementation is a series of 2-3 advisory calls that help you execute each phase of a project. They are included in most advisory memberships.

Guided Implementation 2: Evaluate Your Current BI Practice

Proposed Time to Completion: 1-2 weeks

Step 2.1: Assess Your Current BI Practice

Start with an analyst kick-off call:

• Detail the benefits of conducting multidimensional assessments that involve BI providers as well as consumers.
• Review Info-Tech’s BI Maturity Model.

Then complete these activities…

• SWOT analyses
• Identification of BI maturity level through a current state assessment

With these tools & templates:

BI Practice Assessment Tool

BI Strategy and Roadmap Template

Step 2.2: Envision a Future State for Your BI Practice

Review findings with an analyst:

• Discuss overall maturity gaps and patterns in BI perception amongst different units of your organization.
• Discuss how to translate activity findings into robust initiatives, defining critical success factors for BI development and risk mitigation.

Then complete these activities…

• Identify your desired BI patterns and functionalities.
• Complete a target state assessment for your BI practice.
• Review capability practice gaps and phase-level metrics.

With these tools & templates:

BI Practice Assessment Tool

BI Strategy and Roadmap Template

Phase 2 Results & Insights:

• A comprehensive assessment of the organization’s current BI practice capabilities and gaps
• Visualization of BI perception from a variety of business users as well as IT
• A list of tasks and initiatives for constructing a strategic BI improvement roadmap

STEP 2.1

Assess the Current State of Your BI Practice

Assess your organization’s current BI capabilities

Step Objectives

• Understand the definitions and roles of each component of BI.
• Contextualize BI components to your organization’s environment and current practices.

Step Activities

2.1.1 Perform multidimensional SWOT analyses

2.1.2 Assess current BI and analytical capabilities, Document challenges, constraints, opportunities

2.1.3 Review the results of your current state assessment

Outcomes

• Holistic perspective of current BI strengths and weaknesses according to BI users and providers
• Current maturity in BI and related data management practices

Research Support

• Info-Tech’s Data Management Framework
• Info-Tech’s BI Practice Assessment Tool
• Info-Tech’s BI Strategy and Roadmap Template

Proposed Participants in this Step

Project Manager

Data Architect(s) or Enterprise Architect

Project Team

Gather multiple BI perspectives with comprehensive SWOT analyses

SWOT analysis is an effective tool that helps establish a high-level context for where your practice stands, where it can improve, and the factors that will influence development.

While SWOT is not a new concept, you can add value to SWOT by:

• Conducting a multi-dimensional SWOT to diversify perspectives – involve the existing BI team, BI management, business executives and other business users.
• SWOT analyses traditionally provide a retrospective view of your environment. Add a future-looking element by creating improvement tasks/activities at the same time as you detail historical and current performance.

Info-Tech Insight

Consider a SWOT with two formats: a private SWOT worksheet and a public SWOT session. Participants will be providing suggestions anonymously while solicited suggestions will be discussed in the public SWOT session to further the discussion.

Activity: Perform a SWOT analysis in groups to get a holistic view

2.1.1

1-2 hours

This activity will take your project team through a holistic SWOT analysis to gather a variety of stakeholder perception of the current BI practice.

1. Identify individuals to involve in the SWOT activity. Aim for a diverse pool of participants that are part of the BI practice in different capacities and roles. Solution architects, application managers, business analysts, and business functional unit leaders are a good starting point.
2. Review the findings summary from Phase 1. You may opt to facilitate this activity with insights from the business context. Each group will be performing the SWOT individually.
3. The group results will be collected and consolidated to pinpoint common ideas and opinions. Individual group results should be represented by a different color. The core program team will be reviewing the consolidated result as a group.
4. Document the results of these SWOT activities in the appropriate section of the BI Strategy and Roadmap Template.

SWOT

Group 1 Provider Group E.g. The BI Team

Group 2 Consumer Group E.g. Business End Users

INPUT

• IT and business stakeholder perception

OUTPUT

• Multi-faceted SWOT analyses
• Potential BI improvement activities/objectives

Materials

• SWOT Analysis section of the BI Strategy and Roadmap Template

Participants

• Selected individuals in the enterprise (variable)

Your organization’s BI maturity is determined by several factors and the degree of immersion into your enterprise

BI Maturity Level

A way to categorize your analytics maturity to understand where you are currently and what next steps would be best to increase your BI maturity.

There are several factors used to determine BI maturity:

Buy-in and Data Culture

Determines if there is enterprise-wide buy-in for developing business intelligence and if a data-driven culture exists.

Business–IT Alignment

Examines if current BI and analytics operations are appropriately enabling the business objectives.

Governance Structure

Focuses on whether or not there is adequate governance in place to provide guidance and structure for BI activities.

Organization Structure and Talent

Pertains to how BI operations are distributed across the overall organizational structure and the capabilities of the individuals involved.

Process

Reviews analytics-related processes and policies and how they are created and enforced throughout the organization.

Data

Deals with analytical data in terms of the level of integration, data quality, and usability.

Technology

Explores the opportunities in building a fit-for-purpose analytics platform and consolidation opportunities.

Evaluate Your Current BI Practice with the CMMI model

To assess BI, Info-Tech uses the CMMI model for rating capabilities in each of the function areas on a scale of 1-5. (“0” and “0.5” values are used for non-existent or emerging capabilities.)

Use Info-Tech’s BI Maturity Model as a guide for identifying your current analytics competence

Leverage a BI strategy to revamp your BI program to strive for a high analytics maturity level. In the future you should be doing more than just traditional BI. You will perform self-service BI, predictive analytics, and data science.

Leverage Info-Tech’s BI Practice Assessment Tool to define your BI current state

BI Practice Assessment Tool

1. Assess Current State

• Eight BI practice areas to assess maturity.
• Based on CMMI maturity scale.

• Determine your BI maturity level.
• Identify areas with outstanding maturity.
• Uncover areas with low maturity.
• Visualize the presence of misalignments.

• Tackle target state from two views: business and IT.
• Calculate gaps between target and current state.

• A heat map diagram to compare the target state and the current state.
• Show both current and target maturity levels.
• Detailed charts to show results for each area.
• Detailed list of recommendations.

Purposes:

• Assess your BI maturity.
• Visualize maturity assessment to quickly spot misalignments, gaps, and opportunities.
• Provide right-sized recommendations.

Info-Tech Insight

Assessing current and target states is only the beginning. The real value comes from the interpretation and analysis of the results. Use visualizations of multiple viewpoints and discuss the results in groups to come up with the most effective ideas for your strategy and roadmap.

Activity: Conduct a current state assessment of your BI practice maturity

2.1.2

2-3 hours

Use the BI Practice Assessment Tool to establish a baseline for your current BI capabilities and maturity.

1. Navigate to Tab 2. Current State Assessment in the BI Practice Assessment Tool and complete the current state assessment together or in small groups. If running a series of assessments, do not star or scratch every time. Use the previous group’s results to start the conversation with the users.

Info-Tech suggests the following groups participate in the completion of the assessment to holistically assess BI and to uncover misalignment:

Providers		Consumers
CIO & BI Management	BI Work Groups (developers, analysts, modelers)	Business Unit #1	Business Unit #2	Business Unit #3

2. For each assessment question, answer the current level of maturity in terms of:
1. Initial/Ad hoc – the starting point for use of a new or undocumented repeat process
2. Developing – the process is documented such that it is repeatable
3. Defined – the process is defined/confirmed as a standard business process
4. Managed and Measurable – the process is quantitatively managed in accordance with agreed-upon metrics.
5. Optimized – the process includes process optimization/improvement.

INPUT

• Observations of current maturity

OUTPUT

• Comprehensive current state assessment

Materials

• BI Practice Assessment Tool
• Current State Assessment section of the BI Strategy and Roadmap Template

Participants

• Selected individuals as suggested by the assessment tool

Info-Tech Insight

Discuss the rationale for your answers as a group. Document the comments and observations as they may be helpful in formulating the final strategy and roadmap.

Activity: Review and analyze the results of the current state assessment

2.1.3

2-3 hours

1. Navigate to Tab 3. Current State Results in the BI Practice Assessment Tool and review the findings:

The tool provides a brief synopsis of your current BI state. Review the details of your maturity level and see where this description fits your organization and where there may be some discrepancies. Add additional comments to your current state summary in the BI Strategy and Roadmap Document.

In addition to reviewing the attributes of your maturity level, consider the following:

1. What are the knowns – The knowns confirm your understanding on the current landscape.

Activity: Review and analyze the results of the current state assessment (cont.)

2.1.3

2-3 hours

2. Tab 3 will also visualize a breakdown of your maturity by BI practice dimension. Use this graphic as a preliminary method to identify where your organization is excelling and where it may need improvement.

Better Practices

Consider: What have you done in the areas where you perform well?

Candidates for Improvement

Consider: What can you do to improve these areas? What are potential barriers to improvement?

STEP 2.2

Envision a Future State for Your Organization’s BI Practice

Detail the capabilities of your next generation BI practice

Step Objectives

• Create guiding principles that will shape your organization’s ideal BI program.
• Pinpoint where your organization needs to improve across several BI practice dimensions.
• Develop approaches to remedy current impediments to BI evolution.

Step Activities

2.2.1 Define guiding principles for the future state

2.2.2 Define the target state of your BI practice

2.2.3 Confirm requirements for BI Styles by management group

2.2.4 Analyze gaps in your BI practice and generate improvement activities and objectives

2.2.5 Define the critical success factors for future BI

2.2.6 Identify potential risks for your future state and create a mitigation plan

Outcomes

• Defined landscape for future BI capabilities, including desired BI functionalities.
• Identification of crucial gaps and improvement points to include in a BI roadmap.
• Updated BI Styles Usage sheet.

Research Support

• Info-Tech’s Data Management Framework
• Info-Tech’s BI Practice Assessment Tool
• Info-Tech’s BI Strategy and Roadmap Template

Proposed Participants in this Step

Project Manager

Data Architect(s) or Enterprise Architect

Project Team

Define guiding principles to drive your future state envisioning

Envisioning a BI future state is essentially architecting the future for your BI program. It is very similar to enterprise architecture (EA). Guiding principles are widely used in enterprise architecture. This best practice should also be used in BI envisioning.

Benefits of Guiding Principles in a BI Context

• BI planning involves a number of business units. Defining high-level future state principles helps to establish a common ground for those different business units.
• Ensure the next generation BI aligns with the corporate enterprise architecture and data architecture principles.
• Provide high-level guidance without depicting detailed solutioning by leaving room for innovation.

Sample Principles for BI Future State

1. BI should be fit for purpose. BI is a business technology that helps business users.
2. Business–IT collaboration should be encouraged to ensure deliverables are relevant to the business.
3. Focus on continuous improvement on data quality.
4. Explore opportunities to onboard and integrate new datasets to create a holistic view of your data.
5. Organize and present data in an easy-to-consume, easy-to-digest fashion.
6. BI should be accessible to everything, as soon as they have a business case.
7. Do not train just on using the platform. Train on the underlying data and business model as well.
8. Develop a training platform where trainees can play around with the data without worrying about messing it up.