Make Sense of Strategic Portfolio Management

Separate what's new and valuable from bloated claims on the hype cycle.

Analyst Perspective

Do you need strategic portfolio management, or do you need to do portfolio management more strategically?

Travis Duncan
Research Director, PPM and CIO Strategy
Info-Tech Research Group

While the market is eager to get users into what they're calling "strategic portfolio management," there's a lot of uncertainty out there about what this market is and how it's different from other, more established portfolio disciplines – most significantly, project portfolio management.

Indeed, if you look at how the space is covered within the industry, you'll encounter a dog's breakfast of players, a comparison of apples and oranges: Jira in the same quadrants as Planisware, Smartsheets in the same profiles as Planview and ServiceNow. While each of the individual players is impressive, their areas of focus are unique and the extent to which they should be compared together under the category of strategic portfolio management is questionable.

It speaks to some of the grey area within the SPM space more generally, which is at a bit of a crossroads: Will it formally shed the guardrails of its antecedents to become its own space, or will it devolve into a bait and switch through which capabilities that struggled to gain much traction beyond IT settings seek to infiltrate the business and grow their market share under a different name?

Part of it is up to the rest of us as users and potential customers. Clarifying what we need before we jump into something simply because our prior attempts failed will help determine whether we need a unique space for strategic portfolio management or whether we simply need to do portfolio management more strategically.

Executive Summary

Info-Tech Insight
In the same way that it took many years for PPM to stabilize as a concept distinct from traditional enterprise project management, strategic portfolio management is experiencing a similar period of formational uncertainty. In a space that can be all things to all users, clarify your actual needs before jumping onto a bandwagon and ending up with something that you don't need, and that the organization can't adopt.

Strategic portfolio management is enterprise portfolio management

Evolved from various other capabilities and vendor solutions, strategic portfolio management (SPM) seeks to connect strategy to execution.

While the concept of 'strategic portfolio management' has been written about within project portfolio management circles for nearly 20 years, SPM, as a distinct organizational competence and software category, is a relatively new and largely vendor-driven capability.

First emerging in the discourse during the mid-to-late 2010s, SPM has evolved from its roots in traditional enterprise project portfolio management. Though, as we will discuss, it has other antecedents not limited to PPM.

In this publication, we'll unpack what SPM is, how it is distinct (and, in turn, how it is not distinct) from PPM and other capabilities, and we will consider the extent to which your organization can and should leverage an SPM application to help drive strategic outcomes.

–The increasing need to deliver value from digital initiatives is giving rise to strategic portfolio management, a digital investment management discipline that enables strategy realization in complex dynamic environments."
– OnePlan, "Is Strategic Portfolio Management the Future of PPM?"

Only 2% of business leaders are confident that they will achieve 80% to 100% of their strategic objectives.
Source: Smith, 2022

Put strategic portfolio management in context

SPM is a new stage in the history of project portfolio management more generally. While it's emerging as a distinct capability, and it borrows from capabilities beyond PPM, unpacking its distinctiveness is best done by first understanding its source.

Understand the recent triggers for strategic portfolio management

Triggers for the emergence of strategic portfolio management in the discourse include the pace of technology-introduced change, the waning of enterprise project management, and challenges around enterprise PPM tool adoption.

Spot the difference?

Scope, focus, and audience are just a few of the factors distinguishing what the market calls "SPM" from traditional PPM.

Info-Tech Insight
The distinction between the two capabilities is not necessarily as black and white as the table above would have it (some "PPM" tools offer what we're identifying above as "SPM" capabilities), but it can be helpful to think in these binaries when trying to distinguish the two capabilities. At the very least, SPM broadens its scope to target more executive and business users, and functions best when it's speaking at a higher level, to a business audience.

Strategic portfolio management offers a more holistic view of the enterprise

At its best, strategic portfolio management can accommodate various paradigms of work management and incorporate different types of portfolio management.

Perhaps the biggest evolution from traditional PPM that strategic portfolio management promises is that it casts a wider net in terms of the types of work it tracks (and how it tracks that work) and the types of portfolios it accommodates.

Not bound to the concepts of "projects" and a "project portfolio" specifically, SPM broadens its scope to encompass capabilities like product and product portfolio management, enterprise architecture management, security and risk management, and more.

• Where a PPM solution only shows one piece of the puzzle, SPM looks at the entire investment ecosystem, tracking strategic goals, the ideas generated to help achieve those goals, and all the various kinds of investments made in the service of those goals.
• what's more, where traditional PPM tools required users to adhere to a certain way of working and managing tasks, SPM is more flexible, relying on integrations across various ways of working to provide higher-level insight on the progress of work and the achievement of goals.

Deliver business strategy and change effectively

"An SPM tool will capture business strategy, business capabilities, operating models, the enterprise architecture and the project portfolio with unmatched visibility into how they all relate. This will give...a robust understanding of the impact of a proposed IT change " and enable IT and business to act like cocreators driving innovation."
– Paula Ziehr

You might need a strategic portfolio management tool if–

If you find yourself facing any of these situations, it might be time to step away from your PPM tool and into an SPM approach:

• Your organization is facing a large implementation that will cross multiple departmental units and requires alignment across senior leadership (e.g. a digital transformation initiative).
• You currently have disparate systems tracking different portfolios (project, product, applications, etc.) and types of investments, but lack insight into the whole in terms of how work efforts and investments tie back to strategy realization.
• You are an ePMO or a strategy realization office that doesn't manage work necessarily, but that rather ensures that the work, assets, and capabilities that are funded connect to strategy and drive the realization of strategy.

Sixty one percent of leaders acknowledge their companies struggle to bridge the gap between creating a strategy and executing on that strategy.
Source: StrategyBlocks, 2020

Get to know your strategic portfolio management stakeholders

In terms of users, SPM's focus is further up the org chart than most applications, relying on high-level but usable outputs to help drive decision making.

What should you look for in a strategic portfolio management tool? (1 of 2)

Standard features for SPM include:

What should you look for in a strategic portfolio management tool? (2 of 2)

Advanced features for SPM can include:

What's promising within the space?

As this space continues to stabilize, the following are some promising associations for business and IT enablement.

What's of concern within the space?

As a progeny from other capabilities, SPM has some risks and connotations potential users should be wary of.

Does your organization need a strategic portfolio management tool?

Use Info-Tech's Strategic Portfolio Management Needs Assessment to gauge your readiness for SPM.

• As noted in previous places in this deck, there is often a grey area in the market between project portfolio management tools and strategic portfolio management tools.
• Some PPM tools offer SPM functionality, while some SPM tools avoid traditional PPM outcomes and stay at a higher, strategic level.
• Depending on the scope of your PMO or portfolio optimization needs, you may need a tool that has just one, or both, of these capabilities.
• Use Info-Tech's Strategic Portfolio Management Needs Assessment to help you assess whether you require a high-level strategy management tool, a more low-level project portfolio management tool, or a mix of both.

Download Info-Tech's Strategic Portfolio Management Needs Assessment

1.1 Assess your needs

10 to 20 minutes

1. The Strategic Portfolio Management Needs Assessment is a 41-question survey broken up into three parts: (1) PMO Type, (2) Features and Functionality, (3) Roles.
2. Go through each section using the provided dropdowns to help identify the orientation of your PMO, the feature and functionality needs of your office, as well as the roles whose needs will need to be serviced through the potential tool implementation.

This screenshot shows a sample output from the assessment. Based upon your inputs, you'll be grouped within three ranges:

1. Green: Based upon your inputs, you will benefit from an SPM tool.
2. Yellow: You may benefit from an SPM tool, but you may also require something more traditional. Clarify your requirements before proceeding.
3. Red: you're unlikely to leverage many of the benefits of an SPM tool at this time. Look for a more tactical solution.

Explore the SPM vendor landscape

Use Info-Tech's application selection resources to help find the right solution for your organization.

If the analysis in the previous slides suggested you can benefit from an SPM tool, you can quick-start your vendor evaluation process with SoftwareReviews.

SoftwareReviews has extensive coverage of not just the SPM space, but of the project portfolio management (pictured to the top right) and project management spaces as well. So, from the tactical to the strategic, SoftwareReviews can help you find the right tools.

Further, as you settle in on a shortlist, you can begin your vendor analysis using our rapid application selection methodology (see framework on bottom right). For more information see our The Rapid Application Selection Framework blueprint.

Info-Tech's Rapid Application Selection Framework (RASF)

Related Info-Tech Research

Develop a Project Portfolio Management Strategy
Drive IT project throughput by throttling resource capacity.

Prepare an Actionable Roadmap for your PMO
Turn planning into action with a realistic PMO timeline.

Maintain an Organized Portfolio
Align portfolio management practices with COBIT (APO05: Manage Portfolio)

Bibliography

Angliss, Katy, and Pete Harpum. Strategic Portfolio Management: In the Multi-Project and Program Organization. Book. Routledge. 30 Dec. 2022.

Anthony, James. "95 Essential Project Management Statistics: 2022 Market Share & Data Analysis." Finance Online. 2022. Web. Accessed 21 March 2022

Banham, Craig. "Integrating strategic planning with portfolio management." Sopheon. Webinar. Accessed 6 Feb. 2023.

Garfein, Stephen J. "Executive Guide to Strategic Portfolio Management: roadmap for closing the gap between strategy and results." PMI. Conference Paper. Oct. 2007. Accessed 6 Feb. 2023.

Garfein, Stephen J. "Strategic Portfolio Management: A smart, realistic and relatively fast way to gain sustainable competitive advantage." PMI. Conference Paper. 2 March 2005. Accessed 6 Feb. 2023.

Hontar, Yulia. "Strategic Portfolio Management." PPM Express. Blog 16 June 2022. Accessed 6 Feb. 2023.

Milsom, James. "6 Strategic Portfolio Management Trends for 2023." i-nexus. Blog. 25 Jan. 2022. Accessed 6 Feb. 2023.

Milsom, James. "Strategic Portfolio Management 101." i-nexus. 8 Dec. 2021. Blog . Accessed 6 Feb. 2023.

OnePlan, "Is Strategic Portfolio Management the Future of PPM?" YouTube. 17 Nov. 2022. Accessed 6 Feb. 2023.

OnePlan. "Strategic Portfolio Management for Enterprise Agile." YouTube. 27 May 2022. Accessed 6 Feb. 2023.

Piechota, Frank. "Strategic Portfolio Management: Enabling Successful Business Outcomes." Shibumi. Blog . 31 May 2022. Accessed 6 Feb. 2023.

ServiceNow. "Strategic Portfolio Management—The Thing You've Been Missing." ServiceNow. Whitepaper. 2021. Accessed 6 Feb. 2023.

Smith, Shepherd, "50+ Eye-Opening Strategic Planning Statistics" ClearPoint Strategy. Blog. 13 Sept. 2022. Accessed 6 Feb. 2023.

SoftwareAG. "What is Strategic Portfolio Management (SPM)?" SoftwareAG. Blog. Accessed 6 Feb. 2023.

Stickel, Robert. "What It Means to be Adaptive." OnePlan. Blog. 24 May 2021. Accessed 6 Feb. 2023.

UMT360. "What is Strategic Portfolio Management?" YouTube. Webinar. 22 Oct. 2020. Accessed 6 Feb. 2023.

Wall, Caroline. "Elevating Strategy Planning through Strategic Portfolio Management." StrategyBlocks. Blog. 26 Feb. 2020. Accessed 6 Feb. 2023.

Westmoreland, Heather. "What is Strategic Portfolio Management." Planview. Blog. 19 Oct 2002. Accessed 6 Feb. 2023.

Wiltshire, Andrew. "Shibumi Included in Gartner Magic Quadrant for Strategic Portfolio Management for the 2nd Straight Year." Shibumi. Blog. 20 Apr. 2022. Accessed 6 Feb. 2023.

Ziehr, Paula. "Keep your eye on the prize: Align your IT investments with business strategy." SoftwareAG. Blog. 5 Jul. 2022. Accessed 6 Feb. 2023.

Manage Your Chromebooks and MacBooks

Financial constraints, strategy, and your user base dictate the need for Chromebooks and MacBooks – now you have to manage them in your environment.

Analyst Perspective

Managing MacBooks and Chromebooks is similar to managing Windows devices in many ways and different in others. The tools have many common features, yet they struggle to achieve the same goals.

Until recently, Windows devices dominated the workplace globally. Computing devices were also rare in many industries such as education. Administrators and administrative staff may have used Windows-based devices, but Chromebooks were not yet in use. Most universities and colleges were Windows-based in offices with some flavor of Unix in other areas, and Apple devices were gaining some popularity in certain circles.

That is a stark contrast compared to today, where Chromebooks dominate the classrooms and MacBooks and Chromebooks are making significant inroads into the enterprise environment. MacBooks are also a common sight on many university campuses. There is no doubt that while Windows may still be the dominant player, it is far from the only one in town.

Now that Chromebooks and MacBooks are a notable, if not significant, part of the education and enterprise environments, they must be afforded the same considerations as Windows devices in those environments when it comes to management. The good news is that there is no lack of available solutions for managing these devices, and the endpoint management landscape is continually evolving and improving.

P.J. Ryan
Research Director, Infrastructure & Operations
Info-Tech Research Group

Executive Summary

Your Challenge

• You modernized your end-user computing strategy and now have Windows 10 devices as well as MacBooks.
• Virtual desktop infrastructure (VDI) and desktop as a service (DaaS) are becoming popular. Chromebooks would be ideal as a low-cost interface into DaaS for your employees.
• You are responsible for the management of all the new Chromebooks in your educational district.
• Windows is no longer the only option. MacBooks and Chromebooks are justified, but now you have to manage them.

Common Obstacles

• Endpoint management solutions typically do a great job at managing one category of devices, like Windows or MacBooks, but they struggle to fully manage alternative endpoints.
• Multiple solutions to manage multiple devices will result in multiple dashboards. A single view would be better.
• One solution may not fit all, but multiple solutions is not desirable either, especially if you have Windows devices, MacBooks, and Chromebooks.

Info-Tech's Approach

• Use the tools at your disposal first – don't needlessly spend money if you don't have to. Many solutions can already manage other types of devices to some degree.
• Use the integration capabilities of endpoint management tools. Many of them can integrate with each other to give you a single interface to manage multiple types of devices while taking advantage of additional functionality.
• Don't purchase capabilities you will never use. Using 80% of a less expensive tool is economically smarter than using 10% of a more expensive tool.

Info-Tech Insight

Managing end-user devices may be accomplished with a variety of solutions, but many of those solutions advocate integration with a Microsoft-friendly solution to take advantage of features such as conditional access, security functionality, and data governance.

Insight Summary

Insight 1

Google Admin Console is necessary to manage Chromebooks, but it can be paired with other tools. Implementation partnerships provide solutions to track the device lifecycle, track the repair lifecycle, sync with Google Admin Console as well as PowerSchool to provide a more complete picture of the user and device, and facilitate reminders to return the device, pay fees if necessary, pick up a device when a repair is complete, and more.

Insight 2

The Google Admin Console allows admins to follow an organizational unit (OU) structure very similar to what they may have used in Microsoft's Active Directory environment. This familiarity makes the task of administering Chromebooks easier for admins.

Insight 3

Chromebook management goes beyond securing and manipulating the device. Controls to protect the students while online, such as Safe Search and Safe Browsing, should also be implemented.

Insight 4

Most companies choose to use a dedicated MacBook management tool. Many unified endpoint management (UEM) tools can manage MacBooks to some extent, but admins tend to agree that a MacBook-focused endpoint management tool is best for MacBooks while a Windows-based endpoint management tool is best for Windows devices.

Insight 5

Some MacBook management solutions advocate integration with Windows UEM solutions to take advantage of Microsoft features such as conditional access, security functionality, and data governance. This approach can also be applied to Chromebooks.

Chromebooks

Chromebooks had a respectable share of the education market before 2020, but the COVID-19 pandemic turbocharged the penetration of Chromebooks in the education industry.

Chromebooks are also catching the attention of some decision makers in the enterprise environment.

"In 2018, Chromebooks represented an incredible 60 percent of all laptop or tablet devices in K-12 -- up from zero percent when the first Chromebook launched during the summer break in 2011."
– "Will Chromebooks Rule the Enterprise?" Computerworld

"Chromebooks were the best performing PC products in Q3 2020, with shipment volume increasing to a record-high 9.4 million units, up a whopping 122% year-on-year."
– Android Police

"Until the pandemic, Chrome OS' success was largely limited to U.S. schools. Demand in 2020 appears to have expanded beyond that small but critical part of the U.S. PC market."
– Geekwire

"In addition to running a huge number of Chrome Extensions and Apps at once, Chromebooks also run Android, Linux and Windows apps."
– "Will Chromebooks Rule the Enterprise?" Computerworld

Managing Chromebooks

Start with the Google Admin Console (GAC)

GAC is necessary to initially manage Chrome OS devices.

GAC gives you a centralized console that will allow you to:

• Create organizational units
• Add your Chromebook devices
• Add users
• Assign users to devices
• Create groups
• Create and assign policies
• Plus more

GAC can facilitate device management with features such as:

• Control admin permissions
• Encryption and update settings
• App deployment, screen timeout settings
• Perform a device wipe if required
• Audit user activity on a device
• Plus more

Device and user addition, group and organizational unit creation and administration, applying policies to devices and users – does all this remind you of your Active Directory environment?

GAC lets you administer users and devices with a similar approach.

Managing Chromebooks

Use Active Directory to manage Chromebooks.

• Enable Active Directory (AD) management from within GAC and you will be able to integrate your Chromebook devices with your AD environment.
• Devices will be visible in both the GAC and AD environment.
• Use Windows Group Policy to manage devices and to push policies to users and devices.
• Users can use their AD username and password to sign into Chromebook devices.
• GAC can still be used for devices that are not synced with AD.

Chromebooks can also be managed through these approved partners:

• Cisco Meraki
• Citrix XenMobile
• IBM MaaS360
• ManageEngine Mobile Device Manager Plus
• VMware Workspace ONE

Source: Google

You must be running the Chrome Enterprise Upgrade and have any licenses required by the approved partner to take advantage of this management option. The partner admin policies supersede GAC.

If you stop using the approved partner admin console to manage your devices, the polices and settings in GAC will immediately take over the devices.

Microsoft still has the market share when it comes to device sales, and many administrators are already familiar with Microsoft's Active Directory. Google took advantage of that familiarity when it designed the Google Admin Console structure for users, groups, and organizational units.

Chromebook Deployment

Chromebook deployment becomes a challenge when device quantities grow. The enrollment process can be time consuming, and every device must be enrolled before it can be used by an employee or a student. Many admins enlist their full IT teams to assist in the short term. Some vendor partners may assist with distribution options if staffing levels permit. Recent developments from Google have opened additional options for device enrollment beyond the manual enrollment approach.

Enrolling Chromebooks comes down to one of two approaches:

1. Manually enrolling one device at a time
   • Users can assist by entering some identifying details during the enrollment if permitted.
   • Some third-party solutions exist, such as USB drives to reduce repetitive keystrokes or hubs to facilitate manually enrolling multiple Chromebooks simultaneously.
2. Google's Chrome Enterprise Upgrade or the Chrome Education Upgrade
   • This allows you to let your users enroll devices after they accept the end-user license agreement.
   • You can take advantage of Google's vendor partner program and use a zero-touch deployment method where the Chromebook devices automatically receive the assigned policies, apps, and settings as soon as the device is powered on and an authorized user signs in.
   • The Enterprise Upgrade and the Education Upgrade do come with an annual cost per device, which is currently less than US$50.
   • The Enterprise and Education Upgrades come with other features as well, such as enhanced security.

Chromebooks are automatically assigned to the top-level organizational unit (OU) when enrolled. Devices can be manually moved to another OU, but admins can also create enrollment policies to place newly enrolled devices in a specific OU or have the device locate itself in the same OU as the user.

Chromebooks in Education

GAC is also used with Education-licensed devices

Most of the settings and features previously mentioned are also available for Education-licensed devices and users. Enterprise-specific features will not be available to Education licenses. (Active Directory integration with Education licenses, for example, is accomplished using a different approach)

• Groups, policies, administrative controls, app deployment and management, adding devices and users, creating organizational units, and more features are all available to Education Admins to use.

Education device policies and settings tend to focus more on protecting the students with controls such as:

• Disable incognito mode
• Disable location tracking
• Disable external storage devices
• Browser based protections such as Safe Search or Safe Browsing
• URL blocking
• Video input disable for websites
• App installation prevention, auto re-install, and app blocking
• Forced re-enrollment to your domain after a device is wiped
• Disable Guest Mode
• Restrict who can sign in
• Audit user activity on a device

When a student takes home a Chromebook assigned to them, that Chromebook may be the only computer in the household. Administrative polices and settings must take into account the fact that the device may have multiple users accessing many different sites and applications when the device is outside of the school environment.

Chromebook Management Extended

An online search for Chromebook management solutions will reveal several software solutions that augment the capabilities of the Google Admin Console. Many of these solutions are focused on the education sector and classroom and student options, although the features would be beneficial to enterprises and educational organizations alike.

These solutions assist or augment Chromebook management with features such as:

• Ability to sync with Google Admin Console
• Ability to sync with student information systems, such as PowerSchool
• Financial management, purchase details, and chargeback
• Asset lifecycle management
• 1:1 Chromebook distribution management
• Repair programs and repair process management
• Check-out/loan program management
• Device distribution/allocation management, including barcode reader integration
• Simple learning material distribution to the classroom for teachers
• Facilitate GAC bulk operations
• Manage inventory of non-IT assets such as projectors, TVs, and other educational assets
• Plus more

"There are many components to managing Chromebooks. Schools need to know which student has which device, which school has which device, and costs relating to repairs. Chromebook Management Software … facilitates these processes."
– VIZOR

MacBooks

• MacBooks are gaining popularity in the Enterprise world.
• Some admins claim MacBooks are less expensive in the long run over Windows-based PCs.
• Users claim less issues when using a MacBook, and overall, companies report increased retention rates when users are using MacBooks.

"Macs now make up 23% of endpoints in enterprises."
– ComputerWeekly.com

"When given the choice, no less than 72% of employees choose Macs over PCs."
– "5 Reasons Mac is a must," Jamf

"IBM says it is 3X more expensive to manage PCs than Macs."
– Computerworld

"74% of those who previously used a PC for work experienced fewer issues now that they use a Mac"
– "Global Survey: Mac in the Enterprise," Jamf

"When enterprise moves to Mac, staff retention rates improve by 20%. That's quite a boost! "
– "5 Reasons Mac is a must," Jamf

Managing MacBooks

Can your existing UEM keep up?

Many Windows unified endpoint management (UEM) tools can manage MacBooks, but most companies choose to use a dedicated MacBook management tool.

• UEM tools that are primarily Windows focused do not typically go deep enough into the management capabilities of non-Windows devices.
• Admins have noted limitations when it comes to using Windows UEM tools, and reasons they prefer a dedicated MacBook management solution include:
  • Easier to use
• Faster response times when deploying settings and policies
• Better control over notification settings and lock screen settings.
• Easier Apple Business Manager (ABM) integration and provisioning.
• Note that not every UEM will have the same limitations or advantages. Functionality is different between vendor products.

Info-Tech Insight

Most Windows UEM tools are constantly improving, and it is only a matter of time before they rival many of the dedicated MacBook management tools out there.

Admins tend to agree that a Windows UEM is best for Windows while an Apple-based UEM is best for Apple devices.

Managing MacBooks

The market for "MacBook-first" management solutions includes a variety of players of varying ages such as:

• Jamf
• Kandji
• Mosyle
• SimpleMDM
• Others

MacBook-focused management tools can provide features such as:

• Encryption and update settings
• App deployment and lifecycle management
• Remote device wipe, scan, shutdown, restart, and lock
• Zero touch deployment and support
• Location tracking
• Browser content filtering
• Enable, hide/block, or disable built-in features
• Configure Wi-Fi, VPN, and certificate-based settings
• Centralized dashboard with device and app listings as well as individual details
• Data restrictions
• Plus more

Unified endpoint management (UEM) solutions that can provide MacBook management to some degree include (but are not limited to):

• Intune
• Ivanti
• Endpoint Central
• WorkspaceOne

Dedicated solutions advocate integration with UEM solutions to take advantage of conditional access, security functionality, and data governance features.

Jamf and Microsoft entered into a collaboration several years ago with the intention of making the MacBook management process easier and more secure.

Microsoft Intune and Jamf Pro: Better together to manage and secure Macs
Microsoft Conditional Access with Jamf Pro ensures that company data is only accessed by trusted users, on trusted devices, using trusted apps. Jamf extends this Enterprise Mobile + Security (EMS) functionality to Mac, iPhone and iPad.
– "Microsoft Intune and Jamf Pro," Jamf

Endpoint Management Selection Tool
Activity

There are many solutions available to manage end-user devices, and they come with a long list of options and features. Clarify your needs and define your requirements before you purchase another endpoint management tool. Don't purchase capabilities that you may never use.

Use the Endpoint Management Selection Tool to identify your desired endpoint solution features and compare vendor solution functionality based on your desired features.

1. List out the desired features you want in an endpoint solution for your devices and record those features in the first column. Use the features provided, or add your own and edit or delete the existing ones if necessary.
2. List your selected endpoint management solution vendors in each of the columns in place of "Vendor 1," "Vendor 2," etc.
3. Fill out the spreadsheet by changing the corresponding desired feature cell under each vendor to a "yes" or "no" based on your findings while investigating each vendor solution.
4. When you have finished your investigation, review your spreadsheet to compare the various offerings and pros and cons of each vendor.
5. Select your endpoint management solution.

Related Info-Tech Research

Modernize and Transform Your End-User Computing Strategy
This project helps support the workforce of the future by answering the following questions: What types of computing devices, provisioning models, and operating systems should be offered to end users? How will IT support devices? What are the policies and governance surrounding how devices are used? What actions are we taking and when? How do end-user devices support larger corporate priorities and strategies?

Best Unified Endpoint Management (UEM) Software 2022 | SoftwareReviews
Compare and evaluate unified endpoint management vendors using the most in-depth and unbiased buyer reports available. Download free comprehensive 40+ page reports to select the best unified endpoint management software for your organization.

Best Enterprise Mobile Management (EMM) Software 2022 | (softwarereviews.com)
Compare and evaluate enterprise mobile management vendors using the most in-depth and unbiased buyer reports available. Download free comprehensive 40+ page reports to select the best enterprise mobile management software for your organization.

Bibliography

Bridge, Tom. "Macs in the enterprise – what you need to know". Computerweekly.com, TechTarget. 27 May 2022. Accessed 12 Aug. 2022.
Copley-Woods, Haddayr. "5 reasons Mac is a must in the enterprise". Jamf.com, Jamf. 28 June 2022. Accessed 16 Aug. 2022.
Duke, Kent. "Chromebook sales skyrocketed in Q3 2020 with online education fueling demand." androidpolice.com, Android Police. 16 Nov 2020. Accessed 10 Aug. 2022.
Elgin, Mike. "Will Chromebooks Rule the Enterprise? (5 Reasons They May)". Computerworld.com, Computerworld. 30 Aug 2019. Accessed 10 Aug. 2022.
Evans, Jonny. "IBM says it is 3X more expensive to manage PCs than Macs". Computerworld.com, Computerworld. 19 Oct 2016. Accessed 23 Aug. 2022.
"Global Survey: Mac in the Enterprise". Jamf.com, Jamf. Accessed 16 Aug. 2022.
"How to Manage Chromebooks Like a Pro." Vizor.cloud, VIZOR. Accessed 10 Aug. 2022.
"Manage Chrome OS Devices with EMM Console". support.google.com, Google. Accessed 16 Aug. 2022.
Protalinski, Emil. "Chromebooks outsold Macs worldwide in 2020, cutting into Windows market share". Geekwire.com, Geekwire. 16 Feb 2021. Accessed 22 Aug. 2022.
Smith, Sean. "Microsoft Intune and Jamf Pro: Better together to manage and secure Macs". Jamf.com, Jamf. 20 April 2022. Accessed 16 Aug. 2022.

Build an IT Risk Management Program

Mitigate the IT risks that could negatively impact your organization.

Table of Contents

3 Executive Brief

4 Analyst Perspective

5 Executive Summary

19 Phase 1: Review IT Risk Fundamentals & Governance

43 Phase 2: Identify and Assess IT Risk

74 Phase 3: Monitor, Communicate, and Respond to IT Risk

102 Appendix

108 Bibliography

Build an IT Risk Management Program

Mitigate the IT risks that could negatively impact your organization.

EXECUTIVE BRIEF

Analyst Perspective

Siloed risks are risky business for any enterprise.

Valence Howden Principal Research Director, CIO Practice

Brittany Lutes Senior Research Analyst, CIO Practice

Risk is an inherent part of life but not very well understood or executed within organizations. This has led to risk being avoided or, when it’s implemented, being performed in isolated siloes with inconsistencies in understanding of impact and terminology.

Looking at risk in an integrated way within an organization drives a truer sense of the thresholds and levels of risks an organization is facing – making it easier to manage and leverage risk while reducing risks associated with different mitigation responses to the same risk events.

This opens the door to using risk information – not only to prevent negative impacts but as a strategic differentiator in decision making. It helps you know which risks are worth taking, driving strong positive outcomes for your organization.

Executive Summary

Your Challenge

IT has several challenges when it comes to addressing risk management:

• Risk is unavoidable. Without a formal program to manage IT risk, you may be unaware of your severest IT risks.
• The business could be making decisions that are not informed by risk.
• Reacting to risks after they occur can be costly and crippling, yet it is one of the most common tactics used by IT departments.

Common Obstacles

Many IT organizations realize these obstacles:

• IT risks and business risks are often addressed separately, causing inconsistencies in the approach.
• Security risk receives such a high profile that it often eclipses other important IT risks, leaving the organization vulnerable.
• Failing to include the business in IT risk management leaves IT leaders too accountable; the business must have accountability as well.

Info-Tech’s Approach

• Transform your ad hoc IT risk management processes into a formalized, ongoing program and increase risk management success.
• Take a proactive stance against IT threats and vulnerabilities by identifying and assessing IT’s greatest risks before they occur.
• Involve key stakeholders, including the business senior management team, to gain buy-in and to focus on the IT risks most critical to the organization.

Info-Tech Insight

IT risk is business risk. Every IT risk has business implications. Create an IT risk management program that shares accountability with the business.

Ad hoc approaches to managing risk fail because…

If you are like the majority of IT departments, you do not have a consistent and comprehensive strategy for managing IT risk.

1. Ad hoc risk management is reactionary.
2. Ad hoc risk management is often focused only on IT security.
3. Ad hoc risk management lacks alignment with business objectives.

The results:

• Increased business risk exposure caused by a lack of understanding of the impact of IT risks on the business.
• Increased IT non-compliance, resulting in costly settlements and fines.
• IT audit failure.
• Ineffective management of risk caused by poor risk information and wrong risk response decisions.
• Increased unnecessary and avoidable IT failures and fixes.

58% of organizations still lack a systematic and robust method to actually report on risks (Source: AICPA, 2021)

Data is an invaluable asset – ensure it’s protected

Risk management is a business enabler

Formalize risk management to increase your likelihood of success.

By identifying areas of risk exposure and creating solutions proactively, obstacles can be removed or circumvented before they become a real problem.

A certain amount of risk is healthy and can stimulate innovation:

• A formal risk management strategy doesn’t mean trying to mitigate every possible risk; it means exposing the organization to the right amount of risk.
• Taking a formal risk management approach allows an organization to thoughtfully choose which risks it is willing to accept.
• Organizations with high risk management maturity will vault themselves ahead of the competition because they will be aware of which risks to prepare for, which risks to ignore, and which risks to take.

Only 12% of organizations are using risk as a strategic tool most or all of the time (Source: AICPA, 2021)

IT risk is enterprise risk

Accountability for IT risks and the decisions made to address them should be shared between IT and the business.

IT risks have a direct and often aggregated impact on enterprise risks and opportunities in the same way other business risks can. This relationship must be understood and addressed through integrated risk management to ensure a consistent approach to risk.

Follow the steps of this blueprint to build or optimize your IT risk management program

Start Here	PHASE 1 Review IT Risk Fundamentals and Governance		PHASE 2 Identify and Assess IT Risk		PHASE 3 Monitor, Report, and Respond to IT Risk
	1.1 Review IT Risk Management Fundamentals	1.2 Establish a Risk Governance Framework	2.1 Identify IT Risks	2.2 Assess and Prioritize IT Risks	3.1 Monitor IT Risks and Develop Risk Responses	3.2 Report IT Risk Priorities

Integrate Risk and Use It to Your Advantage

Accelerate and optimize your organization by leveraging meaningful risk data to make intelligent enterprise risk decisions.

Risk management is more than checking an audit box or demonstrating project due diligence.

Risk Drivers Audit & compliance Preserve value & avoid loss Previous risk impact driver Major transformation Strategic opportunities		Only 7% of organizations are in a “leading” or “aspirational” level of risk maturity. (OECD, 2021)	63% of organizations struggle when it comes to defining their appetite toward strategy related risks. (“Global Risk Management Survey,” Deloitte, 2021)	Late adopters of risk management were 70% more likely to use instinct over data or facts to inform an efficient process. (Clear Risk, 2020)	55% of organizations have little to no training on ERM to properly implement such practices. (AICPA, NC State Poole College of Management, 2021)
		1. Assess Enterprise Risk Maturity	3. Build a Risk Management Program Plan	4. Establish Risk Management Processes	5. Implement a Risk Management Program
		2. Determine Authority with Governance Unfortunately, less than 50% of those in risk focused roles are also in a governance role where they have the authority to provide risk oversight. (Governance Institute of Australia, 2020)
IT can improve the maturity of the organization’s risk governance and help identify risk owners who have authority and accountability. Governance and related decision making is optimized with integrated and aligned risk data.			ERM incorporates the different types of risk, including IT, security, digital, vendor, and other risk types. The program plan is meant to consider all the major risk types in a unified approach.		Implementation of an integrated risk management program requires ongoing access to risk data by those with decision making authority who can take action.

Blueprint deliverables

Each step of this blueprint is accompanied by supporting deliverables to help you accomplish your goals:

Key deliverable: Risk Management Program Manual Use the tools and activities in each phase of the blueprint to create a comprehensive, customized program manual for the ongoing management of IT risk.	Integrated Risk Maturity Assessment Assess the organization's current maturity and readiness for integrated risk management (IRM).		Centralized Risk Register The repository for all the risks that have been identified within your environment.
	Risk Costing Tool A potential cost-benefit analysis of possible risk responses to determine a good method to move forward.		Risk Report & Risk Event Action Plan A method to report risk severity and hold risk owners accountable for chosen method of responding.

Benefit from industry-leading best practices

As a part of our research process, we used the COSO, ISO 31000, and COBIT 2019 frameworks. Contextualizing IT risk management within these frameworks ensured that our project-focused approach is grounded in industry-leading best practices for managing IT risk.

Abandon ad hoc risk management

Blueprint benefits

Info-Tech offers various levels of support to best suit your needs

Diagnostics and consistent frameworks used throughout all four options

Guided Implementation

A Guided Implementation (GI) is a series of calls with an Info-Tech analyst to help implement our best practices in your organization.

A typical GI is 6 to 8 calls over the course of 3 to 6 months.

What does a typical GI on this topic look like?

Phase 1

• Call #1: Assess current risk maturity and organizational buy-in.
• Call #2: Establish an IT risk council and determine IT risk management program goals.

Phase 2

• Call #3: Identify the risk categories used to organize risk events.
• Call #4: Identify the threshold for risk the organization can withstand.

Phase 3

• Call #5: Create a method to assess risk event severity.
• Call #6: Establish a method to monitor priority risks and consider possible risk responses.
• Call #7: Communicate risk priorities to the business and implement risk management plan.

Workshop Overview

Contact your account representative for more information.
workshops@infotech.com 1-888-670-8889

Build an IT Risk Management Program

Phase 1

Review IT Risk Fundamentals and Governance

This phase will walk you through the following activities:

• Gain buy-in from senior leadership
• Assess current program maturity
• Identify obstacles and pain points
• Determine the risk culture of the organization
• Develop risk management goals
• Develop SMART project metrics
• Create the IT risk council
• Complete a RACI chart

This phase involves the following participants:

• IT executive leadership
• Business executive leadership

Step 1.1

Review IT Risk Management Fundamentals

Activities

• 1.1.1 Gain buy-in from senior leadership
• 1.1.2 Assess current program maturity

This step involves the following participants:

• IT executive leadership
• Business executive leadership

Outcomes of this step

• Reviewed key IT principles and terminology
• Gained understanding of the relationship between IT risk management and ERM
• Introduced to Info-Tech’s IT Risk Management Framework
• Obtained the support of senior leadership

Effective IT risk management is possible with or without ERM

Whether or not your organization has ERM, integrating your IT risk management program with the business is possible.

Most IT departments find themselves in one of these two organizational frameworks for managing IT risk:

Info-Tech’s IT risk management framework walks you through each step to achieve risk readiness

IT Risk Management Framework

Risk Governance Optimize Risk Management Processes Assess Risk Maturity Measure the Success of the Program		Risk Identification Engage Stakeholder Participation Use Risk Identification Frameworks Compile IT-Related Risks
Risk Response Establish Monitoring Responsibilities Perform Cost-Benefit Analysis Report Risk Response Actions		Risk Assessment Establish Thresholds for Unacceptable Risk Calculate Expected Cost Determine Risk Severity & Prioritize IT Risks

Effective IT risk management benefits

Obtain the support of the senior leadership team or IT steering committee by communicating how IT risk impacts their priorities.

1.1.1 Gain buy-in from senior leadership

Input: List of IT personnel and business stakeholders

Output: Buy-in from senior leadership for an IT risk management program

Materials: Risk Management Program Manual

Participants: IT executive leadership, Business executive leadership

The resource demands of IT risk management will vary from organization to organization. Here are typical requirements:

• Occasional participation of key IT personnel and select business stakeholders in IT risk council meetings (e.g. once every two weeks).
• Periodic risk assessments (e.g. 4 days, twice a year).
• IT personnel must take on risk monitoring responsibilities (e.g. 1-4 hours per week).
• Record the results in the Program Manual sections 3.3, 3.4 and 3.5.

Record the results in the Risk Management Program Manual.

Integrated Risk Maturity Assessment

The purpose of the Integrated Risk Maturity Assessment is to assess the organization's current maturity and readiness for integrated risk management (IRM)

Frequently and continually assessing your organization’s maturity toward integrated risk ensures the right risk management program can be adopted by your organization.

Integrated Risk Maturity Assessment A simple tool to understand if your organization is ready to embrace integrated risk management by measuring maturity across four key categories: Context & Strategic Direction, Risk Culture & Authority, Risk Management Process, and Risk Program Optimization.

Use the results from this integrated risk maturity assessment to determine the type of risk management program that can and should be adopted by your organizations.

Some organizations will need to remain siloed and focused on IT risk management only, while others will be able to integrate risk-related information to start enabling automatic controls that respond to this data.

1.1.2 Assess current program maturity

1-4 hours

Input: List of IT personnel and business stakeholders

Output: Maturity scores across four key risk categories

Materials: Integrated Risk Maturity Assessment Tool

Participants: IT executive leadership, Business executive leadership

This assessment is intended for frequent use; process completeness should be re-evaluated on a regular basis.

How to Use This Assessment:

1. Download the Integrated Risk Management Maturity Assessment Tool.
2. Tab 2, "Data Entry:" This is a qualitative assessment of your integrated risk management process and is organized by the categories of integrated risk maturity. You will be asked to rate the extent to which you are executing the activities required to successfully complete each phase of the assessment. Use the drop-down menus provided to select the appropriate level of execution for each activity listed.
3. Tab 3, "Results:" This tab will display your rate of IRM completeness/maturity. You will receive a score for each category as well as an overall score. The results will be displayed numerically, by percentage, and graphically.

Record the results in the Integrated Risk Maturity Assessment.

Step 1.2

Establish a Risk Governance Framework

Activities

• 1.2.1 Identify pain points/obstacles and opportunities
• 1.2.2 Determine the risk culture of the organization
• 1.2.3 Develop risk management goals
• 1.2.4 Develop SMART project metrics
• 1.2.5 Create the IT risk council
• 1.2.6 Complete a RACI chart

This step involves the following participants:

• IT executive leadership
• Business executive leadership

Outcomes of this step

• Developed goals for the risk management program
• Established the IT risk council
• Assigned accountability and responsibility for risk management processes

Review IT Risk Fundamentals and Governance

Create an IT risk governance framework that integrates with the business

Follow these best practices to make sure your requirements are solid:

1. Self-assess your current approach to IT risk management.
2. Identify organizational obstacles and set attainable risk management goals.
3. Track the effectiveness and success of the program using SMART risk management metrics.
4. Establish an IT risk council tasked with managing IT risk.
5. Set clear risk management accountabilities and responsibilities for IT and business stakeholders.

Key metrics for your IT risk governance framework

Info-Tech Insight

Metrics provide the foundation for determining the success of your IT risk management program and ensure ongoing funding to support appropriate risk responses.

IT risk management success factors

1.2.1 Identify obstacles and pain points

1-4 hours

Input: Integrated Risk Maturity Assessment

Output: Obstacles and pain points identified

Materials: IT Risk Management Success Factors

Participants: IT executive leadership, Business executive leadership

Anticipate potential challenges and “blind spots” by determining which success factors are missing from your current situation.

Instructions:

1. List the potential obstacles and missing success factors that you must overcome to effectively manage IT risk and build a risk management program.
2. Consider some opportunities that could be leveraged to increase the success of this program.
3. Use this list in Activity 1.2.3 to develop program goals.

Risk Management

Replace the example pain points and opportunities with real scenarios in your organization.

1.2.2 Determine the risk culture of your organization

Determine how your organization fits the criteria listed below. Descriptions and examples do not have to match your organization perfectly.

Be aware of the organization’s attitude towards risk

Risk culture is an organization’s attitude towards taking risks. This attitude manifests itself in two ways:

Info-Tech Insight

Organizations typically fall in the middle of these spectrums. While risk culture will vary depending on the industry and maturity of the organization, a culture with a balanced risk appetite that is extremely risk conscious is able to make creative, dynamic decisions with reasonable limits placed on risk-related decision making.

1.2.3 Develop goals for the IT risk management program

1-4 hours

Input: Integrated Risk Maturity Assessment, Risk Culture, Pain Points and Opportunities

Output: Goals for the IT risk management program

Materials: Risk Management Program Manual

Participants: IT executive leadership, Business executive leadership

Translate your maturity assessment and knowledge about organizational risk culture, potential obstacles, and success factors to develop goals for your IT risk management program.

Instructions:

1. In the Risk Management Program Manual, revise, replace, or add to the high-level goals provided in section 2.4.
2. Make sure that you have three to five high-level goals that reflect the current and targeted maturity of IT risk management processes.
3. Integrate potential obstacles, pain points, and insights from the organization’s risk culture.

Record the results in the Risk Management Program Manual.

1.2.4 Develop SMART project metrics

Create metrics for measuring the success of the IT risk management program.

1.2.4 Develop SMART project metrics (continued)

Attach metrics to your goals to gauge the success of the IT risk management program.

Replace the example metrics with accurate KPIs or metrics for your organization.

Create the IT risk committee (ITRC)

1.2.5 Create the IT risk council

1-4 hours

Input: List of IT personnel and business stakeholders

Output: Goals for the IT risk management program

Materials: Risk Management Program Manual

Participants: CIO, CRO (if applicable), Senior Directors, Head of Operations

Identify the essential individuals from both the IT department and the business to create a permanent committee that meets regularly and carries out IT risk management activities.

Instructions:

1. Review sections 3.1 (Mandate) and 3.2 (Agenda and Responsibilities) of the IT Risk Committee Charter, located in the Risk Management Program Manual. Make any necessary revisions.
2. In section 3.3, document how frequently the council is scheduled to meet.
3. In section 3.4, document members of the IT risk council.
4. Obtain sign-off for the IT risk council from the CIO or another member of the senior leadership team in section 3.5 of the manual.

Record the results in the Risk Management Program Manual.

1.2.6 Complete RACI chart

A RACI diagram is a useful visualization that identifies redundancies and ensures that every role, project, or task has an accountable party.

1.2.6 Complete RACI chart (continued)

Assign risk management accountabilities and responsibilities to key stakeholders:

Build an IT Risk Management Program

Phase 2

Identify and Assess IT Risk

This phase will walk you through the following activities:

• Add organization-specific risk scenarios
• Identify risk events
• Augment risk event list using COBIT 2019 processes
• Conduct a PESTLE analysis
• Determine the threshold for (un)acceptable risk
• Create a financial impact assessment scale
• Select a technique to measure reputational cost
• Create a likelihood scale
• Assess risk severity level
• Assess expected cost

This phase involves the following participants:

• IT risk council
• Relevant business stakeholders
• Representation from senior management team
• Business Risk Owners

Step 2.1

Identify IT Risks

Activities

• 2.1.1 Add organization-specific risk scenarios
• 2.1.2 Identify risk events
• 2.1.3 Augment risk event list using COBIT 19 processes
• 2.1.4 Conduct a PESTLE analysis

This step involves the following participants:

• IT executive leadership
• IT Risk Council
• Business executive leadership
• Business risk owners

Outcomes of this step

• Participation of key stakeholders
• Comprehensive list of IT risk events

Get to know what you don’t know

Info-Tech Insight

What you don’t know CAN hurt you. How do you identify IT-related threats and vulnerabilities that you are not already aware of? Now that you have created a strong risk governance framework that formalizes risk management within IT and connects it to the enterprise, follow the steps outlined in this section to reveal all of IT’s risks.

Engage key stakeholders

Ensure that all key risks are identified by engaging key business stakeholders.

Enable IT to target risk holistically

Take a top-down approach to risk identification to guide brainstorming

Info-Tech’s risk categories are consistent with a risk identification method called Risk Prompting.

A risk prompt list is a list that categorizes risks into types or areas. The n10 risk categories encapsulate the services, activities, responsibilities, and functions of most IT departments. Use these categories and the example risk scenarios provided as prompts to guide brainstorming and organize risks.

2.1.1 Add organization-specific risk scenarios

Review Info-Tech’s ten IT risk categories and add risk scenarios to the examples provided.

2.1.2 Identify risk events

Input: IT risk categories

Output: Risk events identified and categorized

Materials: Risk Register Tool

Participants: IT risk council, Relevant business stakeholders, Representation from senior management team, Business risk owners, CRO (if applicable)

Use Info-Tech’s IT risk categories and scenarios to brainstorm a comprehensive list of IT-related threats and vulnerabilities impacting your organization.

Instructions:

1. Document risk events in the Risk Register Tool.
2. List risk scenarios (organized by risk category) in the Risk Events/Threats column.
3. Disseminate the list to key stakeholders who were unable to participate and solicit their feedback.
   • Consult the RACI chart located in section 4.1 of the Risk Management Program Manual.
4. Attack one scenario at a time, exhausting all realistic risk events for that grouping before moving onto the next scenario. Each scenario should take approximately 45-60 minutes.

Tip: If disagreement arises regarding whether a specific risk event is relevant to the organization or not and it cannot be resolved quickly, include it in the list. The applicability of these risks will become apparent during the assessment process.

Record the results in the Risk Register Tool.

2.1.3 Augment the risk event list using COBIT 2019 processes (Optional)

Other industry-leading frameworks provide alternative ways of conceptualizing the functions and responsibilities of IT and may help you uncover additional risk events.

Instructions:

1. Review COBIT 2019’s 40 IT processes and identify additional risk events.
2. Match risk events to the corresponding risk category and scenario and add them to the Risk Register Tool.

2.1.4 Finalize your risk register by conducting a PESTLE analysis (Optional)

Explore alternative identification techniques to incorporate external factors and avoid “groupthink.”

Consider the External Environment – PESTLE Analysis Despite efforts to encourage equal participation in the risk identification process, key risks may not have been shared in previous exercises. Conduct a PESTLE analysis as a final safety net to ensure that all key risk events have been identified.		Avoid “Groupthink” – Nominal Group Technique The Nominal Group Technique uses the silent generation of ideas and an enforced “safe” period of time where ideas are shared but not discussed to encourage judgement-free idea generation. Ideas are generated silently and independently. Ideas are then shared and documented; however, discussion is delayed until all of the group’s ideas have been recorded. Idea generation can occur before the meeting and be kept anonymous. Note: Employing either of these techniques will lengthen an already time-consuming process. Only consider these techniques if you have concerns regarding the homogeneity of the ideas being generated or if select individuals are dominating the exercise.
List the following factors influencing the risk event: Political factors Economic factors Social factors Technological factors Legal factors Environmental factors

Step 2.2

Assess and Prioritize IT Risks

Activities

• 2.2.1 Determine the threshold for (un)acceptable risk
• 2.2.2 Create a financial impact assessment scale
• 2.2.3 Select a technique to measure reputational cost
• 2.2.4 Create a likelihood scale
• 2.2.5 Risk severity level assessment
• 2.2.6 Expected cost assessment

This step involves the following participants:

• IT risk council
• Relevant business stakeholders
• Representation from senior management team
• Business risk owners

Outcomes of this step

• Business-approved thresholds for unacceptable risk
• Completed Risk Register Tool with risks prioritized according to severity
• Expected cost calculations for high-priority risks

Identify and Assess IT Risk

Reveal the organization’s greatest IT threats and vulnerabilities

Info-Tech Insight

Risk is money. It’s impossible to make intelligent decisions about risks without knowing what their financial impact will be.

Review risk assessment fundamentals

Risk assessment provides you with the raw materials to conduct an informed cost-benefit analysis and make robust risk response decisions.

In this section, you will be prioritizing your IT risks according to their risk severity, which is a reflection of their expected cost.

Calculating risk severity

Maintain the engagement of key stakeholders in the risk assessment process

Info-Tech Insight

If business executives still won’t provide the necessary information to update your initial risk assessments, IT should approach business unit leaders and lower-level management. Lean on strong relationships forged over time between IT and business managers or supervisors to obtain any additional information.

Info-Tech recommends a two-level approach to risk assessment

Review the two levels of risk assessment offered in this blueprint.

Risk severity level assessment (mandatory)

Assess all of your identified risk events with a risk severity-level assessment.

• By creating a likelihood and impact assessment scale divided into three to nine “levels” (sometimes referred to as “buckets”), you can evaluate every risk event quickly while being confident that risks are being assessed accurately.
• In the following activities, you will create likelihood and impact scales that align with your organizational risk appetite and tolerance.
• Severity-level assessment is a “first pass” of your risk list, revealing your organization’s most severe IT risks, which can be assessed in greater detail by incorporating expected cost into your evaluation.

Info-Tech recommends a two-level approach to risk assessment (continued)

Expected cost assessment (optional)

Conduct expected cost assessments for IT’s greatest risks.

For risk events warranting further analysis, translate risk severity levels into hard expected-cost numbers.

Implement and leverage a centralized risk register

The purpose of the risk register is to act as the repository for all the risks that have been identified within your environment.

Use this tool to:

1. Collect and maintain a repository for all IT risk events impacting the organization and relevant information for each risk.
   • Capture all relevant IT risk information in one location.
   • Organize risk identification and assessment information for transparent risk management, stakeholder review, and/or internal audit.
2. Calculate risk severity scores to prioritize risk events and determine which risks require a risk response.
   • Separate acceptable and unacceptable risks (as determined by the business).
   • Rank risks based on severity levels.
3. Assess risk responses and calculate residual risk.
   • Evaluate the effect that proposed risk response actions will have on top risk events and quantify residual risk magnitude.
   • This step will be completed in section 3.1

2.2.1 Determine the threshold for (un)acceptable risk

Input: Risk events, Risk appetite

Output: Threshold for risk identified

Materials: Risk Register Tool, Risk Management Program Manual

Participants: IT risk council, Relevant business stakeholders, Representation from senior management team, Business risk owner

Instructions:

There are times when the business needs to know about IT risks with high expected costs.

1. Create an expected cost threshold that defines what constitutes an acceptable and unacceptable risk for the organization. This figure should be a concrete dollar value. In the next exercises, you will build risk impact and likelihood scales with this value in mind, ensuring that “high” or “extreme” risks are immediately communicated to senior leadership.
2. Do not consider IT budget restrictions when developing this number. The acceptable risk threshold should reflect the business’ tolerance/appetite for risk.

This threshold is typically based on the organization’s ability to absorb financial losses, and its tolerance/appetite towards risk.

If your organization has ERM, adopt the existing acceptability threshold.

Record this threshold in section 5.3 of the Risk Management Program Manual

2.2.2 Create a financial impact assessment scale

1-4 hours

Input: Risk events, Risk threshold

Output: Financial impact scale created

Materials: Risk Register Tool, Risk Management Program Manual

Participants: IT risk council, Relevant business stakeholders, Representation from senior management team, Business risk owner

Instructions:

1. Create a scale to assess the financial impact of risk events.
   • Typically, risk impacts are assessed on a scale of 1-5; however, some organizations may prefer to assess risks using 3, 4, 7, or 9-point scales.
2. Ensure that the unacceptable risk threshold is reflected in the scale.
   • In the example provided, the unacceptable risk threshold ($100,000) is represented as “High” on the impact scale.
3. Attach labels to each point on the scale. Effective labels will easily distinguish between risks on either side of the unacceptable risk threshold.

Record the risk impact scale in section 5.3 of the Risk Management Program Manual

Convert project overruns and service outages into costs

Use the tables below to quickly convert impacts typically measured in units of time to financial cost. Replace the values in the table with those that reflect your own costs.

• While project overruns and service outages may have intangible impacts beyond the unexpected costs stemming from paying employees and lost revenue (such as adding complexity to project management and undermining the business’ confidence in IT), these measurements will provide adequate impact estimations for risk assessment.
• Remember, complex risk events can be analyzed further with an expected cost assessment.

Project Overruns
Project	Time (days) 20 days	Number of employees 8	Average cost per employee (per day) $300	Estimated cost $48,000
Service Outages
Service	Time (hours) 4 hours	Lost revenue (per hour) $10,000	Estimated cost $40,000	Impact scale Low

2.2.3 Select a technique to measure reputational cost (1 of 3)

Realized risk events may have profound reputational costs that do not immediately impact your bottom line.

2.2.3 Select a technique to measure reputational cost (2 of 3)

2.2.3 Select a technique to measure reputational cost (3 of 3)

If you feel that the other techniques have not reflected reputational impacts in the overall severity level of the risk, create a parallel scale that roughly matches your financial impact scale.

Technique #3 – Create a parallel scale for reputational impact: Visibility is a useful metric for measuring reputational impact. Visibility measures how widely knowledge of the risk event has spread and how negatively the organization is perceived. Visibility has two main dimensions: Internal vs. External Low Amplification vs. High Amplification Internal/External: The further outside of the organization that the risk event is visible, the higher the reputational impact. Low/High Amplification: The greater the ability of the actor to communicate and amplify the occurrence of a risk event, the higher the reputational impact. After establishing a scale for reputational impact, test whether it reflects the severity of the financial impact levels in the financial impact scale. For example, if the media learns about a recent data breach, does that feel like a $100,000 loss?

Example:

2.2.4 Create a likelihood scale

1-3 hours

Instructions: Create a scale to assess the likelihood that a risk event will occur over a given period of time. Info-Tech recommends assessing the likelihood that the risk event will occur over a period of one year (the IT risk council should be reassessing the risk event no less than once per year). Ensure that the likelihood scale contains the same number of levels as the financial impact scale (3, 4, 5, 7, or 9). The example provided is likely to satisfy most IT departments; however, you may customize the distribution of likelihood values to reflect the organization’s aversion towards uncertainty. For example, an extremely risk-averse organization may consider any risk event with a likelihood greater than 20% to have a “High” likelihood of occurrence. Attach the same labels used for the financial impact scale (Low, Moderate, High, etc.) Record the risk impact scale in section 5.3 of the Risk Management Program Manual

Info-Tech Insight

Note: Info-Tech endorses the use of likelihood values (1-99%) rather than frequency (3 times per year) as a measurement.
For an explanation of why likelihood values lead to more precise and robust risk assessment, see the Appendix.

2.2.5 Risk severity level assessment

6-10 hours

Input: Risk events identified

Output: Assessed the likelihood of occurrence and impact for all identified risk events

Materials: Risk Register Tool

Participants: IT risk council, Relevant business stakeholders, Representation from senior management team, Business risk owner

Instructions:

1. Document the “Risk Category” and “Existing Controls.” in the Risk Register Tool.
   • (See the slide following this activity for tips on identifying existing controls.)
2. Assign each risk event a likelihood and impact level.
   • Remember, you are assessing the impact that a risk event will have on the organization as a whole, not just on IT.
3. When assigning a financial impact level to a risk event, factor in the likely number of instances that the event will occur within the time frame for which you are assessing (usually one year).
   • For risk events like third-party service outages that typically occur a few times each year, assign them an impact level that reflects the likelihood of financial impact the risk event will have over the entire year.
   • E.g. If your organization is likely to experience two major service outages next year and each outage costs the organization approximately $15,000, the total financial impact is $30,000.

Record results in the Risk Register Tool

2.2.5 Risk severity level assessment (continued)

Identify current risk controls

Consider how IT is already addressing key risks.

Types of current risk control

Consider both tactical and strategic controls already in place when filling out risk event information in the Risk Register Tool.

Info-Tech Insight

Identifying existing risk controls (past risk responses) provides a clear picture of the measures already in place to avoid, mitigate, or transfer key risks. This reveals opportunities to improve existing risk controls, or where new strategies are needed, to reduce risk severity levels below business thresholds.

Assign a risk owner for each risk event

Designate a member of the IT risk council to be responsible for each risk event.

Use Info-Tech’s Risk Costing Tool to calculate the expected cost of IT’s high-priority risks (optional)

Use this tool to:

1. Conduct a deeper analysis of severe risks.
   • Determine specific likelihood and financial impact values to communicate the severity of the risk in the Expected Cost tab.
   • Identify the maximum financial impact that the risk event may inflict.
2. Assess the effectiveness of multiple risk responses for each risk event.
   • Determine how proposed risk events will change the likelihood of occurrence and financial impact of the risk event.
3. Incorporate risk proximity into your cost-benefit analysis of risk responses.
   • Illustrate how spending decisions will impact the expected cost of the risk event over time.

2.2.6 Expected cost assessment (optional)

Assign likelihood and financial impact values to high-priority risks.

2.2.6 Expected cost assessment (continued)

Assign likelihood and financial impact values to high-priority risks.

Evaluate likelihood and impact

Refine your risk assessment process by developing more accurate measurements of likelihood and impact.

Build an IT Risk Management Program

Phase 3

Monitor, Respond, and Report on IT Risk

This phase will walk you through the following activities:

• Develop key risk indicators (KRIs) and escalation protocols
• Establish the reporting schedule
• Identify and assess risk responses
• Analyze risk response cost-benefit
• Create multi-year cost projections
• Obtain executive approval for risk action plans
• Socialize the Risk Report
• Transfer ownership of risk responses to project managers
• Finalize the Risk Management Program Manual

This phase involves the following participants:

• IT risk council
• Relevant business stakeholders
• Representation from senior management team
• Risk business owner

Step 3.1

Monitor IT Risks and Develop Risk Responses

Activities

• 3.1.1 Develop key risk indicators (KRIs) and escalation protocols
• 3.1.2 Establish the reporting schedule
• 3.1.3 Identify and assess risk responses
• 3.1.4 Risk response cost-benefit analysis
• 3.1.5 Create multi-year cost projections

This step involves the following participants:

• IT risk council
• Relevant business stakeholders
• Representation from senior management team
• Business risk owner

Outcomes of this step

• Completed risk event action plans
• Risk responses identified and assessed for top risks
• Risk response selected for top risks

Monitor, Respond, and Report on IT Risk

Use Info-Tech’s Risk Event Action Plan to manage high-priority risks

Manage risks in between risk assessments and create a paper trail for key risks that exceed the unacceptable risk threshold. Use a new form for every high-priority risk that requires tracking.

Risk Event Action Plan

Obtaining sign-off from the senior leadership team or from the ERM office is an important step of the risk management process. The Risk Event Action Plan ensures that high-priority risks are closely monitored and that changes in risk severity are detected and reported.

Clear documentation is a way to ensure that critical information is shared with management so that they can make informed risk decisions. These reports should be succinct yet comprehensive; depending on time and resources, it is good practice to fill out this form and obtain sign-off for the majority of IT risks.

3.1.1 Develop key risk indicators (KRIs) and escalation protocols

Document KRIs, escalation thresholds, and escalation protocols for each risk in a Risk Event Action Plan.

Developing KRIs for success

Examples of KRIs

• Number of resources who quit or were fired who had access to critical data
• Number of risk mitigation initiatives unfunded
• Changes in time horizon of mitigation implementation
• Number of employees who did not report phishing attempts
• Amount of time required to get critical operations access to necessary data
• Number of days it takes to implement a new regulation or compliance control

3.1.2 Establish the reporting schedule

For each risk event, document how frequently the risk owner must report to the IT risk council in the Risk Event Action Plan.

• A clear reporting schedule enforces accountability for each risk event, ensuring that risk owners are fulfilling their monitoring responsibilities.
• The ongoing discussion of risks between assessment cycles also increases overall awareness of how IT risks are not static but constantly evolving.

Reporting	Risk Event
Weekly reports to ITRC
Bi-weekly reports to ITRC
Monthly reports to ITRC
Report to ITRC only if KRI thresholds triggered
No reports; reassessed bi-annually

Use Info-Tech’s tools to identify, analyze, and select risk responses

Identify factors that contribute to the severity of the risk

Environmental factors interact with the root cause to increase the likelihood or impact of the risk event.

3.1.3 Identify and assess risk responses

Record the results in the Risk Event Action Plan.

Take actions to avoid the risk entirely

Risk Avoidance Risk avoidance involves taking evasive maneuvers to avoid the risk event. Risk avoidance targets risk likelihood, decreasing the likelihood of the risk event occurring. Since risk avoidance measures are fairly drastic, the likelihood is often reduced to negligible levels. However, risk avoidance response actions often sacrifice potential benefits to eliminate the possibility of the risk entirely. Typically, risk avoidance measures should only be taken for risk events with extremely high severity and when the severity (expected cost) of the risk event exceeds the cost (benefits sacrificed) of avoiding the risk. Example Risk event: Information security vulnerability from third-party cloud services provider. Risk avoidance action: Store all data in-house. Benefits sacrificed: Cost savings, storage flexibility, etc.

Pursue projects that reduce the likelihood or impact of the risk event

Risk Mitigation

• Risk mitigation actions are risk responses that reduce the likelihood and impact of the risk event.
• Risk mitigation actions can be to either implement new controls or enhance existing ones.

Pursue projects that reduce the likelihood or impact of the risk event (continued)

Use the following IT functions to guide your selection of risk mitigation actions:

Transfer risks to a third party

Risk transfer: the exchange of uncertain future costs for fixed present costs.

Accept risks that fall below established thresholds

Risk Acceptance

Accepting a risk means tolerating the expected cost of a risk event. It is a conscious and deliberate decision to retain the threat.

You may choose to accept a risk event for one of the following three reasons:

1. The risk severity (expected cost) of the risk event falls below acceptability thresholds and does not justify an investment in a risk avoidance, mitigation, or transfer measure.
2. The risk severity (expected cost) exceeds acceptability thresholds but all effective risk avoidance, mitigation, and transfer measures are ineffective or prohibitively expensive.
3. The risk severity (expected cost) exceeds acceptability thresholds but there are no feasible risk avoidance, mitigation, and transfer measures to be implemented.

Info-Tech Insight

Constant monitoring and the assignment of responsibility and accountability for accepted risk events is crucial for effective management of these risks. No IT risk should be accepted without detailed documentation outlining the reasoning behind that decision and evidence of approval by senior management.

3.1.4 Risk response cost-benefit analysis (optional)

The purpose of a cost-benefit analysis (CBA) is to guide financial decision making.

This helps IT make risk-conscious investment decisions that fall within the IT budget and helps the organization make sound budgetary decisions for risk response projects that cannot be addressed by IT’s existing budget.

Instructions: Reopen the Risk Costing Tool. For each risk that you conducted an expected cost assessment in section 2.2 for, find the Excel sheet that corresponds to the risk number (e.g. R001). Identify between one and four risk response options for the risk event and document them in the Risk Costing Tool. The “Risk Response 1” field will be automatically populated with expected cost data for a scenario where no action was taken (risk acceptance). This will serve as a baseline for comparing alternative responses. For the following steps, go through the risk responses one by one. Estimate the first-year cost for the risk response. This cost should reflect initial capital expenditures and first-year operating expenditures.

Record the results in the Risk Costing Tool.

3.1.4 Risk response cost-benefit analysis (continued)

The purpose of a cost-benefit analysis (CBA) is to guide financial decision making.

Instructions:

4. Estimate residual risk likelihood and financial impact for Year 1 with the risk response in place.
   • Rather than estimating the likelihood level (low, medium, high), determine a precise likelihood value of the risk event occurring once the response has been implemented.
   • Estimate the dollar value of financial impacts if the risk event were to occur with the risk response in place.

   The tool will calculate the expected residual cost of the risk event: (Financial Impact x Likelihood) - Costs = Expected Residual Cost

5. Select the highest value risk response and document it in the Risk Register Tool.
6. Document your analysis and recommendations in the Risk Event Action Plan.

Note: See Activity 3.1.5 to build multi-year cost projections for risk responses.

3.1.5 Create multi-year cost projections (optional)

Select between risk response options by projecting their costs and benefits over multiple years.

• It can be difficult to choose between risk response options that require different payment schedules. A risk response project with costs spread out over more than one year (e.g. incremental upgrades to an IT system) may be more advantageous than a project with costs concentrated up front that may cost less in the long run (e.g. replacing the system).
• However, the impact that risk response projects have on reducing risk severity is not necessarily static. For example, an expensive project like replacing a system may drastically reduce the risk severity of a system failure. Whereas, incremental system upgrades may only marginally reduce risk severity in the short term but reach similar levels as a full system replacement in a few years.

Instructions: Calculate expected cost for multiple years using the Risk Costing Tool for: Risk events that are subject to change in severity over time. Risk responses that reduce the severity of the risk gradually. Risk responses that cannot be implemented immediately. Copy and paste the graphs into the Risk Report and the Risk Event Action Plan for the risk event.

Record the results in the Risk Costing Tool.

Step 3.2

Report IT Risk Priorities

Activities

• 3.2.1 Obtain executive approval for risk action plans
• 3.2.2 Socialize the Risk Report
• 3.2.3 Transfer ownership of risk responses to project managers
• 3.2.4 Finalize the Risk Management Program Manual

This step involves the following participants:

• IT risk council
• Relevant business stakeholders
• Representation from senior management team

Outcomes of this step

• Obtained approval for risk action plans
• Communicated IT’s risk recommendations to senior leadership
• Embedded risk management into day-to-day IT operations

Monitor, Respond, and Report on IT Risk

Effectively deliver IT risk expertise to the business

3.2.1 Obtain executive approval for risk action plans

Best Practices and Key Benefits

Best practice is for all acceptable risks to also be signed-off by senior leadership. However, for ITRCs that brainstorm 100+ risks, this may not be possible. If this is the case, prioritize accepted risks that were assessed to be closest to the organization’s thresholds. By receiving a stamp of approval for each key risk from senior management, you ensure that: The organization is aware of important IT risks that may impact business objectives. The organization supports the risk assessment conducted by the ITRC. The organization supports the plan of action and monitoring responsibilities proposed by the ITRC. If a risk event were to occur, the organization holds ultimate accountability.

Task:
All IT risks that were flagged for exceeding the organization’s severity thresholds must obtain sign-off by the CIO or another member of the senior leadership team.

• In the assessment phase, you evaluated risks using severity thresholds approved by the business and determined whether or not they justified a risk response.
• Whether your recommendation was to accept the risk or to analyze possible risk responses, the business should be made aware of most IT risks.

3.2.2 Socialize the risk report

Create a succinct, impactful document that summarizes the outcomes of risk assessment and highlights the IT risk council’s top recommendations to the senior leadership team.

The Risk Report contains: An executive summary page highlighting the main takeaways for senior management: A short summary of results from the most recent risk assessment Dashboard A list of top 10 risks ordered from most severe to least Subsequent individual risk analyses (1 to 10) Detailed risk assessment data Risk responses Risk response analysis Multi-year cost projection (see the following slide) Dashboard Recommendations

Risk Report

Pursue projects that reduce the likelihood or impact of the risk event

Encourage risk awareness to extend the benefits of risk management to every aspect of IT.

Benefits of risk awareness:

• More preventative and proactive approaches to IT projects are discussed and considered.
• Changes to the IT threat landscape are more likely to be detected, communicated, and acted upon.
• IT possesses a realistic perception of its ability to perform functions and provide services.
• Contingency plans are put in place to hedge against risk events.
• Fewer IT risks go unidentified.
• CIOs and business executives make better risk decisions.

Consequences of low risk awareness:

• False confidence about the number of IT risks impacting the organization and their severity.
• Risk-relevant information is not communicated to the ITRC, which may result in inaccurate risk assessments.
• Confusion surrounding whose responsibility it is to consider how risk impacts IT decision making.
• Uncertainty and panic when unanticipated risks impact the IT department and the organization.

Embedding risk management in the IT department is a full-time job

Take concrete steps to increase risk-aware decision making in IT.

The IT risk council plays an instrumental role in fostering a culture of risk awareness throughout the IT department. In addition to periodic risk assessments, fulfilling reporting requirements, and undertaking ongoing monitoring responsibilities, members of the ITRC can take a number of actions to encourage other IT employees to adopt a risk-focused approach, particularly at the project planning stage.

Embedding risk management in the IT department is a full-time job (continued)

Encourage risk awareness by adjusting performance metrics and job titles.

Performance metrics:

Depending on the size of your IT department and the amount of resources dedicated to ongoing risk management, you may consider embedding risk management responsibilities into the performance assessments of certain ITRC members or other IT personnel.

• Personalize the risk management program metrics you have documented in your Risk Management Program Manual.
• Evidence that KPIs are monitored and frequently reported is also a good indicator that risk owners are fulfilling their risk management responsibilities.

Info-Tech Insight

If risk management responsibilities are not built into performance assessments, it is less likely that they will invest time and energy into these tasks. Adding risk management metrics to performance assessments directly links good job performance with good risk management, making it more likely that ITRC activities and initiatives gain traction throughout the IT department.

Job descriptions:

Changing job titles to reflect the focus of an individual’s role on managing IT risk may be a good way to distinguish personnel tasked with developing KRIs and monitoring risks on a week-to-week basis.

• Some examples include IT Risk Officer, IT Risk Manager, and IT Risk Analyst.

3.2.3 Transfer ownership of risk responses to project managers

Once risk responses have obtained approval and funding, it is time to transform them into fully-fledged projects.

• Assign responsibility for executing the specific risk response action to a project manager.
• For advice on how to optimize project management, read Info-Tech’s blueprint Tailor IT Project Management Processes to Fit Your Projects.

3.2.4 Finalize the Risk Management Program Manual

Go back through the Risk Management Program Manual and ensure that the material will accurately reflect your approach to risk management going forward.

Remember, the program manual is a living document that should be evolving alongside your risk management program, reflecting best practices, knowledge, and experiences accrued from your own assessments and experienced risk events.

The best way to ensure that the program manual continues to guide and document your risk management program is to make it the focal point of every ITRC meeting and ensure that one participant is tasked with making necessary adjustments and additions.

Risk Management Program Manual

“Upon completing the Info-Tech workshop, the deliverables that we were left with were really outstanding. We put together a 3-year project plan from a high level, outlining projects that will touch upon our high risk areas.” (Director of Security & Risk, Water Management Company)

Don’t allow your risk management program to flatline

54% of small businesses haven’t implemented controls to respond to the threat of cyber attacks (Source: Insurance Bureau of Canada, 2021)

Don’t be lulled into a false sense of security. It might be your greatest risk.

So you’ve identified the most important IT risks and implemented projects to protect IT and the business.

Unfortunately, your risk assessment is already outdated.

Perform regular health checks to keep your finger on the pulse of the key risks threatening the business and your reputation.

To continue the momentum of your newly forged IT risk management program, read Info-Tech’s research on conducting periodic risk assessments and “health checks”:

Revive Your Risk Management Program With a Regular Health Check

• Complete Info-Tech’s Risk Management Health Check to seize the momentum you created by building a robust IT risk management program and create a process for conducting periodic health checks and embedding ongoing risk management into every aspect of IT.
• Our focus is on using data to make IT risk assessment less like an art and more like a science. Ongoing data-driven risk management is self-improving and grounded in historical data.

Appendix I: Familiarize yourself with key risk terminology

Review important risk management terms and definitions.

Appendix II: Likelihood vs. Frequency

Why we measure likelihood, not frequency:

The basic formula of Likelihood x Impact = Severity is a common methodology used across risk management frameworks. However, some frameworks measure likelihood using Frequency rather than Likelihood.

Frequency is typically measured as the number of instances an event occurs over a given period of time (e.g. once per month).

• For risk assessment, historical data regarding the frequency of a risk event is commonly used to indicate the likelihood that the event will happen in the future.

Likelihood is a numerical representation of the “degree of belief” that the risk event will occur in a given future timeframe (e.g. 25% likelihood that the event will occur within the next year).

False Objectivity

While some may argue that frequency provides an objective measurement of likelihood, it is well understood in the field of likelihood theory that historical data regarding the frequency of a risk event may have little bearing over the likelihood of that event happening in the future. Frequency is often an indication of future likelihood but should not be considered an objective measurement of it.

Likelihood scales that use frequency underestimate the magnitude of risks that lack historical precedent. For example, an IT department that has never experienced a high-impact data breach would adopt a very low likelihood score using the frequentist approach. However, if all of the organization’s major competitors have suffered a major breach within the last two years, they ought to possess a much higher degree of belief that the risk event will occur within the next year.

Likelihood is a more comprehensive measurement of future likelihood, as frequency can be used to inform the selection of a likelihood value. The process of selecting intersubjective likelihood values will naturally internalize historical data such as the frequency that the event occurred in the past. Further, the frequency that the event is expected to occur in the future can be captured by the expected impact value. For example, a risk event that has an expected impact per occurrence of $10,000 that is expected to occur three times over the next year has an expected impact of $30,000.

Appendix III: Should max impacts sway decision making?

Don’t just fixate on the most likely impact – be aware of high-impact outcomes. During assessment, risks are evaluated according to their most likely financial impact. For example, a service outage will likely last for two hours and may have an expected cost of $14,000. Naturally, focusing on the most likely financial impact will exclude higher impacts that – while theoretically possible – are so unlikely that they do not warrant any real consideration. For example, it is possible that a service outage could last for days; however, the likelihood for such an event may be well below 1%. While the risk severity level assessment allows you to present impacts as a range of values (e.g. $50,000 to $75,000), the expected cost assessment requires you to select specific values. However, this analysis may fail to consider much higher potential impacts that have non-negligible likelihood values (likelihood values that you cannot ignore). What you consider “non-negligible” will depend on your organizational risk tolerance/appetite. Sometimes called Black Swan events or Fat-Tailed outcomes, high-impact events may occur when the far right of the likelihood distribution – or the “tail” – is thicker than a normal distribution (see fig. 2). A good example is a data breach. While small to medium impacts are far more likely to occur than a devastating intrusion, the high-impact scenario cannot be ignored completely. For risk events that contain non-negligible likelihoods (too high to be ignored) consider elevating the risk severity level or expected cost.

Leverage Info-Tech’s research on security and compliance risk to identify additional risk events

Take Control of Compliance Improvement to Conquer Every Audit	Info-Tech Insight Don’t gamble recklessly with external compliance. Play a winning system and take calculated risks to stack the odds in your favor. Take an agile approach to analyze your gaps and prioritize your remediations. You don’t always have to be fully compliant as long as your organization understands and can live with the consequences.
Develop and Implement a Security Risk Management Program	Info-Tech Insight Security risk management equals cost effectiveness. Time spent upfront identifying and prioritizing risks can mean the difference between spending too much and staying on budget.

Research Contributors and Experts

Research Contributors and Experts

*Plus 10 additional interviewees who wish to remain anonymous.

Bibliography

“2021 State of the CIO.” IDG, 28 January 2021. Web.

“4 Reasons Why CIOs Lose Their Jobs.” Silverton Consulting, 2012. Web.

Beasley, Mark, Bruce Branson, and Bonnie Hancock. “The State of Risk Oversight,” AICPA, April 2021. Web.

COBIT 2019. ISACA, 2019. Web.

“Cognyte jeopardized its database exposing 5 billion records, including earlier data breaches.” SecureBlink, 21 June 2021. Web.

Culp, Steve. “Accenture 2019 Global Risk Management Study, Financial Services Report.” Accenture, 2019. Web.

Curtis, Patchin, and Mark Carey. “Risk Assessment in Practice.” COSO Committee of Sponsoring Organizations of the Treadway Commission, Deloitte & Touche LLP, 2012. Web.

“Cyber Risk Management.” Insurance Bureau of Canada (IBC), 2022. Web.

Eccles, Robert G., Scott C. Newquist, and Roland Schatz. “Reputation and Its Risks.” Harvard Business Review, February 2007. Web.

Eden, C. and F. Ackermann. Making Strategy: The Journey of Strategic Management. Sage Publications, 1998.

“Enterprise Risk Management Maturity Model.” OECD, 9 February 2021. Web.

Ganguly, Saptarshi, Holger Harreis, Ben Margolis, and Kayvaun Rowshankish. “Digital Risks: Transforming risk management for the 2020s.” McKinsey & Company, 10 February 2017. Web.

“Governance Institute of Australia Risk Management Survey 2020.” Governance Institute of Australia, 2020. Web.

“Guidance on Enterprise Risk Management.” COSO, 2022. Web.

Henriquez, Maria. “The Top 10 Data Breaches of 2021” Security Magazine, 9 December 2021. Web.

Holmes, Aaron. “533 million Facebook users’ phone numbers and personal data have been leaked online.” Business Insider, 3 April 2021. Web.

Bibliography

“Integrated Risk and Compliance Management for Banks and Financial Services Organizations: Benefits of a Holistic Approach.” MetricStream, 2022. Web.

“ISACA’s Risk IT Framework Offers a Structured Methodology for Enterprises to Manage Information and Technology Risk.” ISACA, 25 June 2020. Web.

ISO 31000 Risk Management. ISO, 2018. Web.

Lawton, George. “10 Enterprise Risk Management Trends in 2022.” TechTarget, 2 February 2022. Web.

Levenson, Michael. “MGM Resorts Says Data Breach Exposed Some Guests’ Personal Information.” The New York Times, 19 February 2020. Web.

Management of Risk (M_o_R): Guidance for Practitioners. Office of Government Commerce, 2007. Web.

“Many small businesses vulnerable to cyber attacks.” Insurance Bureau of Canada (IBC), 5 October 2021.

Maxwell, Phil. “Why risk-informed decision-making matters.” EY, 3 December 2019. Web.

“Measuring and Mitigating Reputational Risk.” Marsh, September 2014. Web.

Natarajan, Aarthi. “The Top 6 Business Risks you should Prepare for in 2022.” Diligent, 22 December 2021. Web.

“Operational Risk Management Excellence – Get to Strong Survey: Executive Report.” KMPG and RMA, 2014. Web.

“Third-party risk is becoming a first priority challenge.” Deloitte, 2022. Web.

Thomas, Adam, and Dan Kinsella. “Extended Enterprise Risk Management Survey, 2020.” Deloitte, 2021. Web.

Treasury Board Secretariat. “Guide to Integrated Risk Management.” Government of Canada, 12 May 2016. Web.

Webb, Rebecca. “6 Reasons Data is Key for Risk Management.” ClearRisk, 13 January 2021. Web.

“What is Enterprise Risk Management (ERM)?” RIMS, 2015. Web.

Wiggins, Perry. “Do you spend enough time assessing strategic risks?” CFO, 26 January 2022. Web.

Domino – Maintain, Commit to, or Vacate?

Lotus Domino still lives, and you have options for migrating away from or remaining with the platform.

Executive Summary

Info-Tech Insight

“HCL announced that they have somewhere in the region of 15,000 Domino customers worldwide, and also claimed that that number is growing. They also said that 42% of their customers are already on v11 of Domino, and that in the year or so since that version was released, it’s been downloaded 78,000 times. All of which suggests that the Domino platform is, in fact, alive and well.”
– Nigel Cheshire in Team Studio

Your Challenge

You have a Domino/Notes footprint embedded within your business units and business processes. This is taxing your support organization; you are meeting resistance from the business, and you are now asked to help the organization migrate away from the Lotus Notes platform. The Lotus Notes platform was long used by technology and businesses as a multipurpose solution that, over the years, became embedded within core business applications and processes.

Common Obstacles

For organizations that are struggling to understand their options for the Domino platform, the depth of business process usage is typically the biggest operational obstacle. Migrating off the Domino platform is a difficult option for most organizations due to business process and application complexity. In addition, migrating clients have to resolve the challenges with more than one replaceable solution.

Info-Tech Approach

The most common tactic is for the organization to better understand their Domino migration options and adopt an application rationalization strategy for the Domino applications entrenched within the business. Options include retiring, replatforming, migrating, or staying with your Domino platform.

Review

Is “Lotus” Domino still alive?

Problem statement

The number of member engagements with customers regarding the Domino platform has, as you might imagine, dwindled in the past couple of years. While many members have exited the platform, there are still many members and organizations that have entered a long exit program, but with how embedded Domino is in business processes, the migration has slowed and been met with resistance. Some organizations had replatformed the applications but found that the replacement target state was inadequate and introduced friction because the new solution was not a low-code/business-user-driven environment. This resulted in returning the Domino platform to production and working through a strategy to maintain the environment.

This research is designed for:

• IT strategic direction decision-makers
• IT managers responsible for an existing Domino platform
• Organizations evaluating migration options for mission-critical applications running on Domino

This research will help you:

1. Evaluate migration options.
2. Assess the fit and purpose.
3. Consider strategies for overcoming potential challenges.
4. Determine the future of this platform for your organization.

The “everything may work” scenario

Adopt and expand

Believe it or not, Domino and Notes are still options to consider when determining a migration strategy. With HCL still committed to the platform, there are options organizations should seek to better understand rather than assuming SharePoint will solve all. In our research, we consider:

Importance to current business processes

• Importance of use
• Complexity in migrations
• Choosing a new platform

Available tools to facilitate

• Talent/access to skills
• Economies of scale/lower cost at scale
• Access to technology

Info-Tech Insight

With multiple options to consider, take the time to clearly understand the application rationalization process within your decision making.

• Archive/retire
• Application migration
• Application replatform
• Stay right where you are

Eliminate your bias – consider the advantages

“There is a lot of bias toward Domino; decisions are being made by individuals who know very little about Domino and more importantly, they do not know how it impacts business environment.”

– Rob Salerno, Founder & CTO, Rivet Technology Partners

Domino advantages include:

Modern Cloud & Application

• No-code/low-code technology

Business-Managed Application

• Business written and supported
• Embrace the business support model
• Enterprise class application

Leverage the Application Taxonomy & Build

• A rapid application development platform
• Develop skill with HCL training

HCL Domino is a supported and developed platform

Why consider HCL?

• Consider scheduling a Roadmap Session with HCL. This is an opportunity to leverage any value in the mission and brand of your organization to gain insights or support from HCL.
• Existing Domino customers are not the only entities seeking certainty with the platform. Software solution providers that support enterprise IT infrastructure ecosystems (backup, for example) will also be seeking clarity for the future of the platform. HCL will be managing these relationships through the channel/partner management programs, but our observations indicate that Domino integrations are scarce.
• HCL Domino should be well positioned feature-wise to support low-code/NoSQL demands for enterprises and citizen developers.

Visualize Your Application Roadmap

1. Focus on the application portfolio and crafting a roadmap for rationalization.
   • The process is intended to help you determine each application’s functional and technical adequacy for the business process that it supports.
2. Document your findings on respective application capability heatmaps.
   • This drives your organization to a determination of application dispositions and provides a tool to output various dispositions for you as a roadmap.
3. Sort the application portfolio into a disposition status (keep, replatform, retire, consolidate, etc.)
   • This information will be an input into any cloud migration or modernization as well as consolidation of the infrastructure, licenses, and support for them.

Our external support perspective

by Darin Stahl

Member Feedback

• Some members who have remaining Domino applications in production – while the retire, replatform, consolidate, or stay strategy is playing out – have concerns about the challenges with ongoing support and resources required for the platform. In those cases, some have engaged external services providers to augment staff or take over as managed services.
• While there could be existing support resources (in house or on retainer), the member might consider approaching an external provider who could help backstop the single resource or even provide some help with the exit strategies. At this point, the conversation would be helpful in any case. One of our members engaged an external provider in a Statement of Work for IBM Domino Administration focused on one-time events, Tier 1/Tier 2 support, and custom ad hoc requests.
• The augmentation with the managed services enabled the member to shift key internal resources to a focus on executing the exit strategies (replatform, retire, consolidate), since the business knowledge was key to that success.
• The member also very aggressively governed the Domino environment support needs to truly technical issues/maintenance of known and supported functionality rather than coding new features (and increasing risk and cost in a migration down the road) – in short, freezing new features and functionality unless required for legal compliance or health and safety.
• There obviously are other providers, but at this point Info-Tech no longer maintains a market view or scan of those related to Domino due to low member demand.

Domino database assessments

Consider the database.

• Domino database assessments should be informed through the lens of a multi-value database, like jBase, or an object system.
• The assessment of the databases, often led by relational database subject matter experts grounded in normalized databases, can be a struggle since Notes databases must be denormalized.

Key/Value		Column
Use case: Heavily accessed, rarely updated, large amounts of data Data Model: Values are stored in a hash table of keys. Fast access to small data values, but querying is slow Processor friendly Based on amazon's Dynamo paper Example: Project Voldemort used by LinkedIn		Use case: High availability, multiple data centers Data Model: Storage blocks of data are contained in columns Handles size well Based on Google's BigTable Example: Hadoop/Hbase used by Facebook and Yahoo
Document		Graph
Use case: Rapid development, Web and programmer friendly Data Model: Stores documents made up of tagged elements. Uses Key/Value collections Better query abilities than Key/Value databases. Inspired by Lotus Notes. Example: CouchDB used by BBC		Use case: Best at dealing with complexity and relationships/networks Data model: Nodes and relationships. Data is processed quickly Inspired by Euler and graph theory Can easily evolve schemas Example: Neo4j

Understand your options

Archive/Retire

Store the application data in a long-term repository with the means to locate and read it for regulatory and compliance purposes.

Migrate

Migrate to a new version of the application, facilitating the process of moving software applications from one computing environment to another.

Replatform

Replatforming is an option for transitioning an existing Domino application to a new modern platform (i.e. cloud) to leverage the benefits of a modern deployment model.

Stay

Review the current Domino platform roadmap and understand HCL’s support model. Keep the application within the Domino platform.

Archive/retire

Retire the application, storing the application data in a long-term repository.

Abstract

The most common approach is to build the required functionality in whatever new application/solution is selected, then archive the old data in PDFs and documents.

Typically this involves archiving the data and leveraging Microsoft SharePoint and the new collaborative solutions, likely in conjunction with other software-as-a-service (SaaS) solutions.

Advantages

• Reduce support cost.
• Consolidate applications.
• Reduce risk.
• Reduce compliance and security concerns.
• Improve business processes.

Considerations

• Application transformation
• eDiscovery costs
• Legal implications
• Compliance implications
• Business process dependencies

Info-Tech Insights

Be aware of the costs associated with archiving. The more you archive, the more it will cost you.

Application migration

Migrate to a new version of the application

Abstract

An application migration is the managed process of migrating or moving applications (software) from one infrastructure environment to another.

This can include migrating applications from one data center to another data center, from a data center to a cloud provider, or from a company’s on-premises system to a cloud provider’s infrastructure.

Advantages

• Reduce hardware costs.
• Leverage cloud technologies.
• Improve scalability.
• Improve disaster recovery.
• Improve application security.

Considerations

• Data extraction, starting from the document databases in NSF format and including security settings about users and groups granted to read and write single documents, which is a powerful feature of Lotus Domino documents.
• File extraction, starting from the document databases in NSF format, which can contain attachments and RTF documents and embedded files.
• Design of the final relational database structure; this activity should be carried out without taking into account the original structure of the data in Domino files or the data conversion and loading, from the extracted format to the final model.
• Design and development of the target-state custom applications based on the new data model and the new selected development platform.

Application replatform

Transition an existing Domino application to a new modern platform

Abstract

This type of arrangement is typically part of an application migration or transformation. In this model, client can “replatform” the application into an off-premises hosted provider platform. This would yield many benefits of cloud but in a different scaling capacity as experienced with commodity workloads (e.g. Windows, Linux) and the associated application.

Two challenges are particularly significant when migrating or replatforming Domino applications:

• The application functionality/value must be reproduced/replaced with not one but many applications, either through custom coding or a commercial-off-the-shelf/SaaS solution.
• Notes “databases” are not relational databases and will not migrate simply to an SQL database while retaining the same business value. Notes databases are essentially NoSQL repositories and are difficult to normalize.

Advantages

• Leverage cloud technologies.
• Improve scalability.
• Align to a SharePoint platform.
• Improve disaster recovery.
• Improve application security.

Considerations

• Application replatform resource effort
• Network bandwidth
• New platform terms and conditions
• Secure connectivity and communication
• New platform security and compliance
• Degree of complexity

Info-Tech Insights

There is a difference between a migration and a replatform application strategy. Determine which solution aligns to the application requirements.

Stay with HCL

Stay with HCL, understanding its future commitment to the platform.

Abstract

Following the announced acquisition of IBM Domino and up until around December 2019, HCL had published no future roadmap for the platform. The public-facing information/website at the time stated that HCL acquired “the product family and key lab services to deliver professional services.” Again, there was no mention or emphasis on upcoming new features for the platform. The product offering on their website at the time stated that HCL would leverage its services expertise to advise clients and push applications into four buckets:

1. Replatform
2. Retire
3. Move to cloud
4. Modernize

That public-facing messaging changed with release 11.0, which had references to IBM rebranded to HCL for the Notes and Domino product – along with fixes already inflight. More information can be found on HCL’s FAQ page.

Advantages

• Known environment
• Domino is a supported platform
• Domino is a developed platform
• No-code/low-code optimization
• Business developed applications
• Rapid application framework

Understand your tools

Many tools are available to help evaluate or migrate your Domino Platform. Here are a few common tools for you to consider.

• SWING Software
  • Lotus Notes application archiving with Seascape for Notes: Seascape for Notes and Lotus Notes migration
  • Lotus Notes to SharePoint Migration
• Rivit MigrateMe
• SkyBow Notes to SharePoint
• CIMtrek: CIMtrek SharePoint Migrator and Migrating Lotus Notes Applications to the Cloud Using the CIMtrek migration tools [PDF]
• 4WS.Platform

Notes Archiving & Notes to SharePoint

Summary of Vendor

“SWING Software delivers content transformation and archiving software to over 1,000 organizations worldwide. Our solutions uniquely combine key collaborative platforms and standard document formats, making document production, publishing, and archiving processes more efficient.”*

Tools

Lotus Notes Data Migration and Archiving: Preserve historical data outside of Notes and Domino

Lotus Note Migration: Replacing Lotus Notes. Boost your migration by detaching historical data from Lotus Notes and Domino.

Headquarters

Croatia

Best fit

• Application archive and retire
• Migration to SharePoint

* swingsoftware.com

Domino Migration to SharePoint

Summary of Vendor

“Providing leading solutions, resources, and expertise to help your organization transform its collaborative environment.”*

Tools

Notes Domino Migration Solutions: Rivit’s industry-leading solutions and hardened migration practice will help you eliminate Notes Domino once and for all.

Rivive Me: Migrate Notes Domino applications to an enterprise web application

Headquarters

Canada

Best fit

• Application Archive & Retire
• Migration to SharePoint

* rivit.ca

Lotus Notes to M365

Summary of Vendor

“More than 300 organizations across 40+ countries trust skybow to build no-code/no-compromise business applications & processes, and skybow’s community of customers, partners, and experts grows every day.”*

Tools

SkyBow Studio: The low-code platform fully integrated into Microsoft 365

Headquarters:

Switzerland

Best fit

• Application Archive & Retire
• Migration to SharePoint

* skybow.com | About skybow

Notes to SharePoint Migration

Summary of Vendor

“CIMtrek is a global software company headquartered in the UK. Our mission is to develop user-friendly, cost-effective technology solutions and services to help companies modernize their HCL Domino/Notes® application landscape and support their legacy COBOL applications.”*

Tools

CIMtrek SharePoint Migrator: Reduce the time and cost of migrating your IBM® Lotus Notes® applications to Office 365, SharePoint online, and SharePoint on premises.

Headquarters

United Kingdom

Best fit

• Application replatform
• Migration to SharePoint

* cimtrek.com | About CIMtrek

Domino replatform/Rapid application selection framework

Summary of Vendor

“4WS.Platform is a rapid application development tool used to quickly create multi-channel applications including web and mobile applications.”*

Tools

4WS.Platform is available in two editions: Community and Enterprise.
The Platform Enterprise Edition, allows access with an optional support pack.

4WS.Platform’s technical support provides support services to the users through support contracts and agreements.

The platform is a subscription support services for companies using the product which will allow customers to benefit from the knowledge of 4WS.Platform’s technical experts.

Headquarters

Italy

Best fit

• Application replatform

* 4wsplatform.org

Activity

Understand your Domino options

Application Rationalization Exercise

Info-Tech Insight

Application rationalization is the perfect exercise to fully understand your business-developed applications, their importance to business process, and the potential underlying financial impact.

This activity involves the following participants:

• IT strategic direction decision-makers.
• IT managers responsible for an existing Domino platform
• Organizations evaluating platforms for mission-critical applications.

Outcomes of this step:

• Completed Application Rationalization Tool

Application rationalization exercise

Use this Application Rationalization Tool to input the outcomes of your various application assessments

In the Application Entry tab:

• Input your application inventory or subset of apps you intend to rationalize, along with some basic information for your apps.

In the Business Value & TCO Comparison tab, determine rationalization priorities.

• Input your business value scores and total cost of ownership (TCO) of applications.
• Review the results of this analysis to determine which apps should require additional analysis and which dispositions should be prioritized.

In the Disposition Selection tab:

• Add to or adapt our list of dispositions as appropriate.

In the Rationalization Inputs tab:

• Add or adapt the disposition criteria of your application rationalization framework as appropriate.
• Input the results of your various assessments for each application.

In the Disposition Settings tab:

• Add or adapt settings that generate recommended dispositions based on your rationalization inputs.

In the Disposition Recommendations tab:

• Review and compare the rationalization results and confirm if dispositions are appropriate for your strategy.

In the Timeline Considerations tab:

• Enter the estimated timeline for when you execute your dispositions.

In the Portfolio Roadmap tab:

• Review and present your roadmap and rationalization results.

Follow the instructions to generate recommended dispositions and populate an application portfolio roadmap.

Info-Tech Insight

Watch out for misleading scores that result from poorly designed criteria weightings.

Related Info-Tech Research

Build an Application Rationalization Framework

Manage your application portfolio to minimize risk and maximize value.

Embrace Business-Managed Applications

Empower the business to implement their own applications with a trusted business-IT relationship.

Satisfy Digital End Users With Low- and No-Code

Extend IT, automation, and digital capabilities to the business with the right tools, good governance, and trusted organizational relationships.

Maximize the Benefits from Enterprise Applications with a Center of Excellence

Optimize your organization’s enterprise application capabilities with a refined and scalable methodology.

Drive Successful Sourcing Outcomes With a Robust RFP Process

Leverage your vendor sourcing process to get better results.

Research Authors

Darin Stahl, Principal Research Advisor,
Info-Tech Research Group

Darin is a Principal Research Advisor within the Infrastructure practice, leveraging 38+ years of experience. His areas of focus include IT operations management, service desk, infrastructure outsourcing, managed services, cloud infrastructure, DRP/BCP, printer management, managed print services, application performance monitoring, managed FTP, and non-commodity servers (zSeries, mainframe, IBM i, AIX, Power PC).

Troy Cheeseman, Practice Lead,
Info-Tech Research Group

Troy has over 24 years of experience and has championed large enterprise-wide technology transformation programs, remote/home office collaboration and remote work strategies, BCP, IT DRP, IT operations and expense management programs, international right placement initiatives, and large technology transformation initiatives (M&A). Additionally, he has deep experience working with IT solution providers and technology (cloud) startups.

Research Contributors

Rob Salerno, Founder & CTO, Rivit Technology Partners

Rob is the Founder and Chief Technology Strategist for Rivit Technology Partners. Rivit is a system integrator that delivers unique IT solutions. Rivit is known for its REVIVE migration strategy which helps companies leave legacy platforms (such as Domino) or move between versions of software. Rivit is the developer of the DCOM Application Archiving solution.

Bibliography

Cheshire, Nigel. “Domino v12 Launch Keeps HCL Product Strategy On Track.” Team Studio, 19 July 2021. Web.

“Is LowCode/NoCode the best platform for you?” Rivit Technology Partners, 15 July 2021. Web.

McCracken, Harry. “Lotus: Farewell to a Once-Great Tech Brand.” TIME, 20 Nov. 2012. Web.

Sharwood, Simon. “Lotus Notes refuses to die, again, as HCL debuts Domino 12.” The Register, 8 June 2021. Web.

Woodie, Alex. “Domino 12 Comes to IBM i.” IT Jungle, 16 Aug. 2021. Web.

Manage Exponential Value Relationships

Are you ready to manage outcome-based agreements?

Analyst Perspective

Outcome-based agreements require a higher degree of mutual trust.

Exponential IT brings with it an exciting new world of cutting-edge technology and increasingly accelerated growth of business and IT. But adopting and driving change through this paradigm requires new capabilities to grow impactful and meaningful partnerships with external vendors who can help implement technologies like artificial intelligence and virtual reality. Building outcome-based partnerships involves working very closely with vendors who, in many cases, will have just as much to lose as the organizations implementing these new technologies. This requires a greater degree of trust between parties than a standard vendor relationship. It also drastically increases the risks to both organizations; as each loses some control over data and outcomes, they must trust that the other organization will follow through on commitments and obligations. Outcome-based partnerships build upon traditional vendor management practices and create the potential for organizations to embrace emerging technology in new ways. Kim Osborne Rodriguez Research Director, CIO Advisory Info-Tech Research Group

Executive Summary

Info-Tech Insight

Outcome-based partnerships require a higher degree of trust than traditional vendor relationships. Build trust by sharing risks and rewards.

Vendor relationships can be worth billions of dollars

Positive vendor relationships directly impact the bottom line, sometimes to the tune of billions of dollars annually.

• Organizations typically spend 40% to 80% of their total budget on external suppliers.
• Greater supplier trust translates directly to greater business profits, even in traditional vendor relationships.1
• Based on over a decade of data from vehicle manufacturers, greater supplier relationships nearly doubled the unit profit margin on vehicles, contributing over $20 billion to Toyota’s annual profits based on typical sales volume.2
• Having positive vendor relationships can be instrumental in times of crisis – when scarcity looms, vendors often choose to support their best customers.3,4 For example, Toyota protected itself from the losses many original equipment manufacturers (OEMs) faced in 2020 and showed improved profitability that year due to increased demand for vehicles which it was able to supply as a result of top-ranked vendor relationships.

1 PR Newswire, 2022.
2 Based on 10 years of data comparing Toyota and Nissan, every 1-point increase in the company’s Working Relations Index was correlated with a $15.77 net profit increase per unit. Impact on Toyota annual profits is based on 10.5 million units sold in 2021 and 2022.
3 Interview with Renee Stanley, University of Texas at Arlington. Conducted 17 May 2023.
4 Plante Moran, 2020.

Sources: Macrotrends, Plante Moran 2022, Nissan 2022 and 2023, and Toyota 2022. Profit per car is based on total annual profit divided by total annual sales volume.

Outcome-based relationships are a new paradigm

In a new model where organizations are procuring autonomous capabilities, outcomes will govern vendor relationships.

An outcome-based relationship requires a higher level of mutual trust than traditional vendor relationships. This requires shared reward and shared risk.

Don’t forget about traditional vendor management relationships! Not all vendor relationships can (or should) be outcome-based.

Case study

INDUSTRY: Technology

SOURCE: Press Release

Microsoft and OpenAI partner on Azure, Teams, and Microsoft Office suite

In January 2023, Microsoft announced a $10 billion investment in OpenAI, allowing OpenAI to continue scaling its flagship large language model, ChatGPT, and giving Microsoft first access to deploy OpenAI’s products in services like GitHub, Microsoft Office, and Microsoft Teams.

Shared risk

Issues with OpenAI’s platforms could have a debilitating effect on Microsoft’s own reputation – much like Google’s $100 billion stock loss following a blunder by its AI platform Bard – not to mention the financial loss if the platform does not live up to the hype.

Shared reward

This was a particularly important strategic move by Microsoft, as its main competitors develop their own AI models in a race to the top. This investment also gave OpenAI the resources to continue scaling and evolving its services much faster than it would be capable of on its own. If OpenAI’s products succeed, there is a significant upside for both companies.

Adapt your approach to vendor relationships

Both traditional vendors and exponential relationships are important.

Use Info-Tech’s research to Jump Start Your Vendor Management Initiative.

Common obstacles

Exponential relationships require new approaches to vendor management as businesses autonomize:

• Autonomization refers to the shift toward autonomous business capabilities which leverage technologies such as AI and quantum computing to operate independently of human interaction.
• The speed and complexity of technology advancement requires that businesses move quickly and confidently to develop strong relationships and deliver value.
• We are seeing businesses shift from procuring products and services to procuring autonomous business capabilities (sometimes called “as a service,” or aaS). This shift can drive exponential value but also increases complexity and risk.
• Exponential IT requires a shift in emphasis toward more mature relationship and risk management strategies, compared to traditional vendor management.

The shift from technology service agreements to business capability agreements needs a new approach

Eighty-seven percent of organizations are currently experiencing talent shortages or expect to within a few years.

Source: McKinsey, “Mind the [skills] gap”, 2021.

Sixty-three percent of IT leaders plan to implement AI in their organizations by the end of 2023.

Source: Info-Tech Research Group survey, 2022

Insight summary

Assess your readiness for exponential value relationships

Key deliverable:

Exponential Relationships Readiness Assessment

Determine your readiness to build exponential value relationships.

Measure the value of this blueprint

Save thousands of dollars by leveraging this research to assess your readiness, before you lose millions from a relationship gone bad.

Our research indicates that most organizations would take months to prepare this type of assessment without using our research. That’s over 80 person-hours spent researching and gathering data to support due diligence, for a total cost of thousands of dollars. Doesn’t your staff have better things to do?

Start by answering a few brief questions, then return to this slide at the end to see how much your answers have changed.

Use Info-Tech’s research to Exponential Relationships Readiness Assessment.

Establish a baseline

Gauge the effectiveness of this research by asking yourself the following questions before and after completing your readiness assessment:

How to use this research

Five tips to get the most out of your readiness assessment.

1. Each category consists of five competencies, with a maximum of five points each. The maximum score on this assessment is 100 points.
2. Effectiveness levels range from basic (level 1) to advanced (level 5). Level 1 is generally considered the baseline for most effectively operating organizations. If your organization is struggling with level 1 competencies, it is recommended to improve maturity in those areas before pursuing exponential relationships.
3. This assessment is qualitative; complete the assessment to the best of your ability, based on the scoring rubric provided. If you fall between levels, use the lower one in your assessment.
4. The scoring rubric may not perfectly fit the processes and practices within every organization. Consider the spirit of the description and score accordingly.
5. Other industry- and region-specific competencies may be required to succeed at exponential relationships. The competencies in this assessment are a starting point, and internal validation and assessments should be conducted to uncover additional competencies and skills.

Financial management

Manage your budget and spending to stay on track throughout your relationship.

“Most organizations underestimate the amount of time, money, and skill required to build and maintain a successful relationship with another organization. The investment in exponential relationships is exponential in itself – as are the returns.”

– Jennifer Perrier, Principal Research Director,
Info-Tech Research Group

This step involves the following participants:

• Executive leadership team, including CIO
• CFO
• Vendor management leader
• Other internal stakeholders of vendor relationships

Activities:

• Assess your ability to manage scope and budget in exponential IT relationships.

Successfully manage complex finances

Stay on track and keep your relationship running smoothly.

Why is this important?

• Finance is at the core of most business – it drives decision making, acts as a constraint for innovation and optimization, and plays a key role in assessing options (such as return on investment or payback period).
• Effectively managing finances is a critical success factor in developing strong relationships. Each organization must be able to manage their own budget and spending in order to balance the risk and reward in the relationship. Often, these risks and rewards will come in the form of profit and loss or revenue and spend.

Build it into your practice:

1. Ensure your financial decision-making practices are aligned with the organizational and relationship strategy. Do metrics and criteria reflect the organization’s goals?
2. Develop strong accounting and financial analysis practices – this includes the ability to conduct financial due diligence on potential vendors.
3. Develop consistent methodology to track and report on the desired outcomes on a regular basis.

Build your ability to manage finances

The five competencies needed to manage finances in exponential value relationships are:

Relationship management

Drive exponential value by becoming a customer of choice.

“The more complex the business environment becomes — for instance, as new technologies emerge or as innovation cycles get faster — the more such relationships make sense. And the better companies get at managing individual relationships, the more likely it is that they will become “partners of choice” and be able to build entire portfolios of practical and value-creating partnerships.”

(“Improving the management of complex business partnerships.” McKinsey, 2019)

This step involves the following participants:

• Executive leadership team, including CIO
• Vendor management leader
• Other internal stakeholders of vendor relationships

Activities:

• Assess your ability to manage relationships in exponential IT relationships.

Take your relationships to the next level

Maintaining positive relationships is key to building trust.

Why is this important?

• All relationships will experience challenges, and the ability to resolve these issues will rely heavily on the relationship management skills and soft skills of the leadership within each organization.
• Based on a 20-year study of vendor relationships in the automotive sector, business-to-business trust is a function of reasonable demands, follow-through, and information sharing.

(Source: Plante Moran, 2020)

Build it into your practice:

1. Develop the soft skills necessary to promote psychological safety, growth mindset, and strong and open communication channels.
2. Be smart about sharing information – you don’t need to share everything, but being open about relevant information will enhance trust.
3. Both parties need to work hard to develop trust necessary to build a true relationship. This will require increased access to decision-makers, clearly defined guardrails, and the ability for unsatisfied parties to leave.

Build your ability to manage relationships

The five competencies needed to manage relationships in exponential partnerships are:

Performance management

Outcomes management focuses on results, not methods.

According to Jennifer Robinson, senior editor at Gallup, “This approach focuses people and teams on a concrete result, not the process required to achieve it. Leaders define outcomes and, along with managers, set parameters and guidelines. Employees, then, have a high degree of autonomy to use their own unique talents to reach goals their own way.” (Forbes, 2023)

In the context of exponential relationships, vendors can be given a high degree of autonomy provided they meet their objectives.

This step involves the following participants:

• Executive leadership team, including CIO
• Vendor management leader
• Other internal stakeholders of vendor relationships

Activities:

• Assess your ability to manage outcomes in exponential IT relationships.

Manage outcomes to drive mutual success

Build trust by achieving shared objectives.

Why is this important?

• Relationships are based on shared risk and shared reward for all parties. In order to effectively communicate the shared rewards, you must first understand and communicate your objectives for the relationship, then measure outcomes to ensure all parties are benefiting.
• Effectively managing outcomes reduces the risk that one party will choose to leave based on a perception of benefits not being achieved. Parties may still leave the agreement, but decisions should be based on shared facts and issues should be communicated and addressed early.

Build it into your practice:

1. Clearly articulate what you hope to achieve by entering an outcome-based relationship. Each party should outline and agree to the goals, objectives, and desired outcomes from the relationship.
2. Document how rewards will be shared among parties. What type of rewards are anticipated? Who will benefit and how?
3. Develop consistent methodology to track and report on the desired outcomes on a regular basis. This might consist of a vendor scorecard or a monthly meeting.

Build your ability to manage outcomes

The five competencies needed to manage outcomes in exponential value relationships are:

Risk management

Exponential IT means exponential risk – and exponential rewards.

One of the key differentiators between traditional vendor relationships and exponential relationships is the degree to which risk is shared between parties. This is not possible in all industries, which may limit companies’ ability to participate in this type of exponential relationship.

This step involves the following participants:

• Executive leadership team, including CIO
• Vendor management leader
• Risk management leader
• Other internal stakeholders of vendor relationships

Activities:

• Assess your ability to manage risk in exponential IT relationships.

Relationships come with a lot of hidden risks

Successfully managing complex risks can be the difference between a spectacular success and company-ending failure.

Why is this important?

• Relationships inherently involve a loss of control. You are relying on another party to fulfill their part of the agreement, and you depend on the success of the outcome. Loss of control comes with significant risks.
• Sharing in risk is what differentiates an outcome-based relationship from a traditional vendor relationship; vendors must have skin in the game.
• Organizations must consider many different types of risk when considering a relationship with a vendor: fraud, security, human rights, labor relations, ESG, and operational risks. Remember that risk is not inherently bad; some risk is necessary.

Build it into your practice:

1. Build or hire the necessary risk expertise needed to properly assess and evaluate the risks of potential vendor relationships. This includes intellectual property, ESG, legal/regulatory, cybersecurity, data security, and more.
2. Develop processes and procedures which clearly communicate and report on risk on a regular basis.

Info-Tech Insight

Some highly regulated industries (such as finance) are prevented from transferring certain types of risk. In these industries, it may be much more difficult to form vendor relationships.

Don’t forget about third-party ESG risk

Customers care about ESG. You should too.

Protect yourself against third-party ESG risks by considering the environmental and social impacts of your vendors.

Third-party ESG risks can include the following:

• Environmental risk: Vendors with unsustainable practices such as carbon emissions or waste generation of natural resource depletion can negatively impact the organization’s environmental goals.
• Social risk: Unsafe or illegal labor practices, human rights violations, and supply chain management issues can reflect negatively on organizations that choose to work with vendors who engage in such practices.
• Governance risk: Vendors who engage in illegal or unethical behaviors, including bribery and corruption or data and privacy breaches can impact downstream customers.

Working with vendors that have a poor record of ESG carries a very real reputational risk for organizations who do not undertake appropriate due diligence.

A global survey of nearly 14,000 customers revealed that…

Source: EY Future Consumer Index, 2021

Seventy-seven percent of customers believe companies have a responsibility to manufacture sustainably.

Sixty-eight percent of customers believe businesses should ensure their suppliers meet high social and environmental standards.

Fifty-five percent of customers consider the environmental impact of production in their purchasing decisions.

Build your ability to manage risk

The five competencies needed to manage risk in exponential value relationships are:

Contract management

Contract management is a critical part of vendor management.

Well-managed contracts include clearly defined pricing, performance-based outcomes, clear roles and responsibilities, and appropriate remedies for failure to meet requirements. In outcome-based relationships, contracts are generally used as a secondary method of enforcing performance, with relationship management being the primary method of addressing challenges and ensuring performance.

This step involves the following participants:

• Executive leadership team, including CIO
• Vendor management leader
• Risk management leader
• Other internal stakeholders of vendor relationships

Activities:

• Assess your ability to manage risk in exponential IT relationships.

Build your ability to manage contracts

The five competencies needed to manage contracts in exponential value relationships are:

Activity 1: Assess your readiness for exponential relationships

1-3 hours

1. Gather key stakeholders from across your organization to participate in the readiness assessment exercise.
2. As a group, review the core competencies from the previous four sections and determine where your organization’s effectiveness lies for each competency. Record your responses in the Exponential Relationships Readiness Assessment tool.

Download the Exponential Relationships Readiness Assessment tool.

Understand your assessment

This step involves the following participants:

• Executive leadership team, including CIO
• Vendor management leader
• Other internal stakeholders of vendor relationships

Activities:

• Create an action plan.

Understand the results of your assessment

Consider the following recommendations based on your readiness assessment scores:

• The chart to the right shows sample results. The bars indicate the recommended scores, and the line indicates the readiness score.
• Three or more categories below the recommended scores, or any categories more than five points below the recommendation: outcome-based relationships are not recommended at this time.
• Two or more categories below the recommended scores: Proceed with caution and limit outcome-based relationships to low-risk areas. Continue to mature capabilities.
• One category below the recommended scores: Evaluate the risks and benefits before engaging in higher-risk vendor relationships. Continue to mature capabilities.
• All categories at or above the recommended scores: You have many of the core capabilities needed to succeed at exponential relationships! Continue to evaluate and refine your vendor relationships strategy, and identify any additional competencies needed based on your industry or region.

Activity 2: Create an action plan

1 hour

1. Gather the stakeholders who participated in the readiness assessment exercise.
2. As a group, review the results of the readiness assessment. Where there any surprise? Do the results reflect your understanding of the organization’s maturity?
3. Determine which areas are likely to limit the organization’s relationship capability, based on lowest scoring areas and relative importance to the organization.
4. Break out into groups and have each group identify three actions the organization could take to mature the lowest scoring areas.
5. Bring the group back together and prioritize the actions. Note who will be accountable for each next step.

Related Info-Tech Research

Jump Start Your Vendor Management Initiative
Create and implement a vendor management framework to begin obtaining measurable results in 90 days.

Elevate Your Vendor Management Initiative
Transform your VMI from tactical to strategic to maximize its impact and value

Evaluate Your Vendor Account Team to Optimize Vendor Relations
Understand the value of knowing your account team’s influence in the organization, and your influence, to drive results.

Related Info-Tech Research

Build an IT Risk Management Program
Mitigate the IT risks that could negatively impact your organization.

Build an IT Budget
Effective IT budgets are more than a spreadsheet. They tell a story.

Adopt an Exponential IT Mindset
Thrive through the next paradigm shift..

Author

Kim Osborne Rodriguez Research Director, CIO Advisory Info-Tech Research Group

Kim is a professional engineer and Registered Communications Distribution Designer (RCDD) with over a decade of experience in management and engineering consulting spanning healthcare, higher education, and commercial sectors. She has worked on some of the largest hospital construction projects in Canada, from early visioning and IT strategy through to design, specifications, and construction administration. She brings a practical and evidence-based approach, with a track record of supporting successful projects. Kim holds a Bachelor’s degree in Honours Mechatronics Engineering and an option in Management Sciences from the University of Waterloo.

Research Contributors and Experts

	Jack Hakimian Senior Vice President Info-Tech Research Group Jack has more than 25 years of technology and management consulting experience. He has served multibillion-dollar organizations in multiple industries including financial services and telecommunications. Jack also served several large public sector institutions. He is a frequent speaker and panelist at technology and innovation conferences and events and holds a Master’s degree in Computer Engineering as well as an MBA from the ESCP-EAP European School of Management.		Michael Tweedie Practice Lead, CIO Strategy Info-Tech Research Group Mike Tweedie brings over 25 years as a technology executive. He’s led several large transformation projects across core infrastructure, application and IT services as the head of Technology at ADP Canada. He was also the Head of Engineering and Service Offerings for a large French IT services firm, focused on cloud adoption and complex ERP deployment and management. Mike holds a Bachelor’s degree in Architecture from Ryerson University.
	Scott Bickley Practice Lead, VCCO Info-Tech Research Group Scott Bickley is a Practice Lead & Principal Research Director at Info-Tech Research Group, focused on Vendor Management and Contract Review. He also has experience in the areas of IT Asset Management (ITAM), Software Asset Management (SAM), and technology procurement along with a deep background in operations, engineering, and quality systems management. Scott holds a B.S. in Justice Studies from Frostburg State University. He also holds active IAITAM certification designations of CSAM and CMAM and is a Certified Scrum Master (SCM).		Donna Bales Principal Research Director Info-Tech Research Group Donna Bales is a Principal Research Director in the CIO Practice at Info-Tech Research Group, specializing in research and advisory services in IT risk, governance, and compliance. She brings over 25 years of experience in strategic consulting and product development and has a history of success in leading complex, multistakeholder industry initiatives. Donna has a bachelor’s degree in economics from the University of Western Ontario.

Research Contributors and Experts

Jennifer Perrier Principal Research Director Info-Tech Research Group Jennifer has 25 years of experience in the information technology and human resources research space, joining Info-Tech in 1998 as the first research analyst with the company. Over the years, she has served as a research analyst and research manager, as well as in a range of roles leading the development and delivery of offerings across Info-Tech’s product and service portfolio, including workshops and the launch of industry roundtables and benchmarking. She was also Research Lead for McLean & Company, the HR advisory division of Info-Tech, during its start-up years. Jennifer’s research expertise spans the areas of IT strategic planning, governance, policy and process management, people management, leadership, organizational change management, performance benchmarking, and cross-industry IT comparative analysis. She has produced and overseen the development of hundreds of publications across the full breadth of both the IT and HR domains in multiple industries. In 2022, Jennifer joined Info-Tech’s IT Financial Management Practice with a focus on developing financial transparency to foster meaningful dialogue between IT and its stakeholders and drive better technology investment decisions.

Phil Bode Principal Research Director Info-Tech Research Group Phil has 30+ years of experience with IT procurement-related topics: contract drafting and review, negotiations, RFXs, procurement processes, and vendor management. Phil has been a frequent speaker at conferences, a contributor to magazine articles in CIO Magazine and ComputerWorld, and quoted in many other magazines. He is a co-author of the book The Art of Creating a Quality RFP. Phil has a Bachelor of Science in Business Administration with a double major of Finance and Entrepreneurship and a Bachelor of Science in Business Administration with a major of Accounting, both from the University of Arizona.

Research Contributors

Erin Morgan Assistant Vice President, IT Administration University of Texas at Arlington

Renee Stanley Assistant Director IT Procurement and Vendor Management University of Texas at Arlington

Note: Additional contributors did not wish to be identified.

Bibliography

Andrea, Dave. “Plante Moran’s 2022 Working Relations Index® (WRI) Study shows supplier relations can improve amid industry crisis.” Plante Moran, 25 Aug 2022. Accessed 18 May 2023.
Andrea, Dave. “Trust between suppliers and OEMs can better prepare you for the next crisis.” Plante Moran, 9 Sept 2020. Accessed 17 May 2023.
Cleary, Shannon, and Carolan McLarney. “Organizational Benefits of an Effective Vendor Management Strategy.” IUP Journal of Supply Chain Management, Vol. 16, Issue 4, Dec 2019.
De Backer, Ruth, and Eileen Kelly Rinaudo. “Improving the management of complex business partnerships.” McKinsey, 21 March 2019. Accessed 9 May 2023 .
Dennean, Kevin et al. “Let's chat about ChatGPT.” UBS, 22 Feb 2023. Accessed 26 May 2023.
F&I Tools. “Nissan Worldwide Vehicle Sales Report.” Factory Warranty List, 2022. Accessed 18 May 2023.
Gomez, Robin. “Adopting ChatGPT and Generative AI in Retail Customer Service.” Radial, 235, April 2023. Accessed 10 May 2023.
Harms, Thomas and Kristina Rogers. “How collaboration can drive value for you, your partners and the planet.” EY, 26 Oct 2021. Accessed 10 May 2023.
Hedge & Co. “Toyota, Honda finish 1-2; General Motors finishes at 3rd in annual Supplier Working Relations Study.” PR Newswire, 23 May 2022. Accessed 17 May 2023.
Henke Jr, John W., and T. Thomas. "Lost supplier trust, lost profits." Supply Chain Management Review, May 2014. Accessed 17 May 2023.
Information Services Group, Inc. “Global Demand for IT and Business Services Continues Upward Surge in Q2, ISG Index™ Finds.” BusinessWire, 7 July 2021. Accessed 8 May 2023.
Kasanoff, Bruce. “New Study Reveals Costs Of Bad Supplier Relationships.” Forbes, 6 Aug 2014. Accessed 17 May 2023.
Macrotrends. “Nissan Motor Gross Profit 2010-2022.” Macrotrends. Accessed 18 May 2023.
Macrotrends. “Toyota Gross Profit 2010-2022.” Macrotrends. Accessed 18 May 2023.
McKinsey. “Mind the [skills] gap.” McKinsey, 27 Jan 2021. Accessed 18 May 2023.
Morgan, Blake. “7 Examples of How Digital Transformation Impacted Business Performance.” Forbes, 21 Jul 2019. Accessed 10 May 2023.
Nissan Motor Corporation. “Nissan reports strong financial results for fiscal year 2022.” Nissan Global Newsroom, 11 May 2023. Accessed 18 May 2023.

Bibliography

“OpenAI and Microsoft extend partnership.” Open AI, 23 Jan 2023. Accessed 26 May 2023.
Pearson, Bryan. “The Apple Of Its Aisles: How Best Buy Lured One Of The Biggest Brands.“ Forbes, 23 Apr 2015. Accessed 23 May 2023.
Perifanis, Nikolaos-Alexandros and Fotis Kitsios. “Investigating the Influence of Artificial Intelligence on Business Value in the Digital Era of Strategy: A Literature Review.” Information, 2 Feb 2023. Accessed 10 May 2023.
Scott, Tim and Nathan Spitse. “Third-party risk is becoming a first priority challenge.” Deloitte. Accessed 18 May 2023.
Stanley, Renee. Interview by Kim Osborne Rodriguez, 17 May 2023.
Statista. “Toyota's retail vehicle sales from 2017 to 2021.” Statista, 27 Jul 2022. Accessed 18 May 2023.
Tlili, Ahmed, et al. “What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education.” Smart Learning Environments, 22 Feb 2023. Accessed 9 May 2023.
Vitasek, Kate. “Outcome-Based Management: What It Is, Why It Matters And How To Make It Happen.” Forbes, 12 Jan 2023. Accessed 9 May 2023.

Drive Business Value With Off-the-Shelf AI

A practical guide to ensure return on your Off-the-Shelf AI investment

Executive Summary

Info-Tech Insight

To guarantee the success of your Off-the-Shelf AI implementation and ensure it delivers value, you must start with a clear definition of the business case and an understanding of your data.

Info-Tech offers various levels of support to best suit your needs

Diagnostics and consistent frameworks used throughout all four options

Getting value out of AI and machine learning investments

AI vs. ML

Types of Off-the-Shelf AI products and solutions

The data inputs for these models are defined, the developer has to conform to the provided schema, and the data outputs are usually fixed due to the particular task the OTS model is built to solve.

Insight summary

Overarching insight:

To guarantee the success of your Off-the-Shelf AI implementation and ensure it delivers value, you must start with a clear definition of the business case and an understanding of your data.

Business Goals

Question the value that AI adds to the tool you are evaluating. Don’t go after the tool simply because it has an AI label attached to it. AI/ML capabilities might add little value but increase implementation complexity. Define the problem you are solving and document business requirements for the tool or a model.

Data

Know your data. Determine data requirements to:

• Train the model during the implementation and development.
• Run the model in production.

People/Skills

Define the skills required for the implementation and assemble the team that will support the project from requirements to deployment and support, through its entire lifecycle. Don’t forget about production support and maintenance.

Choosing an AI-Powered Tool

No need to reinvent the wheel and build a product you can buy, but be prepared to work around tool limitations, and make sure you understand the data and the model the tool is built on.

Choosing an AI/ML Model

Using Off-the-Shelf-AI models enables an agile approach to system development. Faster POC and validation of ideas and approaches, but the model might not be customizable for your requirements.

Guaranteeing Off-the-Shelf AI Implementation Success

Info-Tech Insight

To guarantee the success of your Off-the-Shelf AI implementation and ensure it delivers value, you must start with a clear definition of the business case and an understanding of your data.

Why do you need AI in your toolset? Business Goals Clearly defined problem statement and business requirements for the tool or a model will help you select the right solution that will deliver business value even if it does not have all the latest bells and whistles. Artificial Intelligence Data Strategy

Do you know the data required for implementation? Data Expected business outcome defines data requirements for implementation. Do you have the right data required to train and run the model? Data Governance Machine Bias

Is your organization ready for AI? People/Team/ Skills New skills and expertise are required through all phases of the implementation: design, build, deployment, support, and maintenance, as well as post-production support, scaling, and adoption. Data Architecture/ Infrastructure New tool or model will impact your cloud and integration strategy. It will have to integrate with the existing infrastructure, in the cloud or on prem. Create an Architecture for AI

What questions do you need to ask when choosing the solution? Product/ Tool or Model Selection Do you know what model powers the AI tool? What data was used to train the tool and what data is required to run it? Ask the right questions. Rapid Application Selection Framework SoftwareReviews

Are you measuring impact on your processes? Business and IT Processes Business processes need to be defined or updated to incorporate the output of the tool back into the business processes to deliver value. IT governance and support processes need to accommodate the new AI-powered tool.

Realize and measure business value of your AI investment Value Do you have a clear understanding of the value that AI will bring to your organization?Optimization?Increased revenue?Operational efficiency?

Introduction of Off-the-Shelf AI Requires a Strategic Approach

Define Business Goals and Objectives

Why do you need AI in your toolset? What value will AI deliver? Have a clear understanding of business benefits and the value AI delivers through the tool. Define a business problem that can be solved with either an AI-powered tool or AI/ML pre-built model. Define expectations and assumptions around the value that AI can bring. Document business requirements for a tool or model. Start with the POC or a prototype to test assumptions, architecture, and components of the solution. Define business metrics and KPIs to measure success of the implementation. Info-Tech Insight Question the value that AI adds to the tool you are evaluating. Don’t go after the tool simply because it has an AI label attached to it. AI/ML capabilities might add little value but increase implementation complexity. Define the problem you are solving and document business requirements for the tool or a model.

AAI solutions and technologies are helping organizations make faster decisions and predict future outcomes such as: Business process automation Intelligent integration Intelligent insights Operational efficiency improvement Increase revenue Improvement of existing products and services Product and process innovation

1. Use Info-Tech’s Off-the-Shelf AI Analysis Tool to define business drivers and document business requirements

		Use the Business Drivers tab to document: Business objectives of the initiative that might drive the AI/ML use case. The business owner or primary stakeholder who will help to define business value and requirements. All stakeholders who will be involved or impacted. KPIs that will be used to assess the success of the POC. Data required for the implementation.
Use the Business Requirements tab to document high-level requirements for a tool or model. These requirements will be used while defining criteria for a tool selection and to validate if the tool or model meets your business goals. You can use either traditional BRD format or a user story to document requirements.

Download the Off-the-Shelf AI Analysis Tool

1. Define business drivers and document business requirements

2. Use Info-Tech’s Off-the-Shelf AI Analysis Tool to document data requirements

2-3 hours

Use the Data tab to document the following for each data source or dataset: Data Domain – e.g. Customer data Data Concept – e.g. Customer Data Internally Accessible – Identify datasets that are required for the implementation even if the data might not be available internally. Work on determining if the data ca be acquired externally or collected internally. Source System – define the primary source system for the data, e.g. Salesforce Target System (if applicable) – Define if the data needs to be migrated/transferred. For example, you might use a datalake or data warehouse for the AI/ML solution or migrate data to the cloud. Classification/Taxonomy/Ontology Data Steward Data Owner Data Quality – Data quality indicator Refresh Rate – Frequency of data refresh. Indicate if the data can be accessed in real time or near-real time

Retention – Retention policy requirements Compliance Requirements – Define if data has to comply with any of the regulatory requirements, e.g. GDPR Privacy, Bias, and Ethics Considerations – Privacy Act, PIPEDA, etc. Identify if the dataset contains sensitive information that should be excluded from the model, such as gender, age, race etc. Indicate fairness metrics, if applicable. Download the Off-the-Shelf AI Analysis Tool

2. Document data requirements

Is Your Organization Ready for AI?

Assess organizational readiness and define stakeholders impacted by the implementation. Build the team with the right skillset to drive the solution.

Info-Tech Insight:

Define the skills required for the implementation and assemble the team that will support the project through its entire lifecycle. Don’t forget about production, support, and maintenance.

3. Build your implementation team

1-2 hours

Input: Solution conceptual design, Current resource availability

Output: Roles required for the implementation of the solution, Resources gap analysis, Training and hiring plan

Materials: Whiteboard/Flip charts, Off-the-Shelf AI Analysis Tool, “People and Team” tab

Participants: Project lead, HR, Enterprise Architect

1. Review your solution conceptual design and define implementation team roles.
2. Document requirements for each role.
3. Review current org chart and job descriptions and identify skillset gaps. Draft an action plan to fill in the roles.
4. Use Info-Tech’s Off-the-Shelf AI Analysis Tool's People and Team tab to document team roles for the entire implementation, including design, build/implement, deployment, support and maintenance, and future development.

Download the Off-the-Shelf AI Analysis Tool

Cloud, SaaS or On Prem – what are my options and what is the impact?

Depending on the architecture of the solution, define the impact on the current infrastructure, including system integration, AI/ML pipeline deployment, maintenance, and data storage

• Data Architecture: use the current data architecture to design the architecture for an AI-powered solution. Assess changes to the data architecture with the introduction of a new tool to make sure it is scalable enough to support the change.
• Define infrastructure requirements for either Cloud, Software-as-a-Service, or on-prem deployment of a tool or model.
• Define how the tool will be integrated with existing systems and into existing infrastructure.
• Define requirements for:
  • Data migration and data storage
  • Security
  • AI/ML pipeline deployment, production monitoring, and maintenance
• Define requirements for operation and maintenance of the tool or model.
• Work with your infrastructure architect and vendor to determine the cost of deploying and running the tool/model.
• Make a decision on the preferred architecture of the system and confirm infrastructure readiness.

Download the Create an Architecture for AI blueprint

4. Use Info-Tech’s Off-the-Shelf AI Analysis Tool to document infrastructure decisions

2-3 hours

Input: Solution conceptual design

Output: Infrastructure requirements, Infrastructure readiness assessment

Materials: Whiteboard/Flip charts, Off-the-Shelf AI Analysis Tool, “Infrastructure” tab

Participants: Infrastructure Architect, Solution Architect, Enterprise Architect, Data Architect, ML/AI Ops Engineer

1. Work with Infrastructure, Data, Solution, and Enterprise Architects to define your conceptual solution architecture.
2. Define integration and storage requirements.
3. Document security requirements for the solution in general and the data specifically.
4. Define MLOps requirements and tools required for ML/AI pipeline deployment and production monitoring.
5. Use Info-Tech’s Off-the-Shelf AI Analysis Tool's Infrastructure tab to document requirements and decisions around Data and Infrastructure Architecture.

Download the Off-the-Shelf AI Analysis Tool

What questions do you need to ask vendors when choosing the solution?

Take advantage of Info-Tech’s Rapid Application Selection Framework (RASF) to guide tool selection, but ask vendors the right questions to understand implications of having AI/ML built into the tool or a model

Use Info-Tech’s Off-the-Shelf AI Analysis Tool, “Vendor Questionnaire” tab to track vendor responses to these questions.

Are you measuring impact on your processes?

Make sure that you understand the impact of the new technology on the existing business and IT processes.

And make sure your business processes are ready to take advantage of the benefits and new capabilities enabled by AI/ML.

Process automation, optimization, and improvement enabled by the technology and AI/ML-powered tools allow organizations to reduce manual work, streamline existing business processes, improve customer satisfaction, and get critical insights to assist decision making.

To take full advantage of the benefits and new capabilities enabled by the technology, make sure that business and IT processes reflect these changes:

• Processes that need to be updated.
• How the outcome of the tool or a model (e.g. predictions) is incorporated into the existing business processes and the processes that will monitor the accuracy of the outcome and monitor performance of the tool or model.
• New business and IT processes that need to be defined for the tool (e.g. chatbot maintenance, analysis of the data generated by the tool, etc.).

5. Document the Impact on Business and IT Processes

2-3 hours

Input: Solution design, Existing business and IT processes

Output: Documented updates to the existing processes, Documented new business and IT processes

Materials: Whiteboard/Flip charts, Off-the-Shelf AI Analysis Tool, “Business and IT Processes” tab

Participants: Project lead, Business stakeholders, Business analyst

1. Review current business processes affected by the implementation of the AI/ML- powered tool or model. Define the changes that need to be made. The changes might include simplification of the process due to automation of some of the steps. Some processes will need to be redesigned and some processes might become obsolete.
2. Document high-level steps for any new processes that need to be defined around the AI/ML-powered tool. An example of such a process would be defining new IT and business processes to support a new chatbot.
3. Use Info-Tech’s Off-the-Shelf AI Analysis Tool's Business and IT Processes tab, to document process changes.

Download the Off-the-Shelf AI Analysis Tool

AI-powered Tools – Considerations

Pre-built/pre-trained models – what to keep in mind when choosing

Info-Tech Insight:

Using Off-the-Shelf AI models enables an agile approach to system development – faster POC and validation of ideas and approaches, but the model might not be customizable for your requirements.

Metrics

Metrics and KPIs for this project will depend on the business goals and objectives that you will identify in Step 1 of the tool selection process.

Metrics might include:

• Reduction of time spent on a specific business process. If the tool is used to automate certain steps of a business process, this metric will measure how much time was saved, in minutes/hours, compared to the process time before the introduction of the tool.
• Accuracy of prediction. This metric would measure the accuracy of estimations or predictions compared to the same estimations done before the implementation of the tool. It can be measured by generating the same prediction or estimation using the AI-powered tool or using any methods used before the introduction of the tool and comparing the results.
• Accuracy of the search results. If the AI-powered tool is a search engine, compare a) how much time it would take a user to find an article or a piece of content they were searching for using new tool vs. previous techniques, b) how many steps it took the user to locate the required article in the search results, and c) the location of the correct piece of content in the search result list (at the top of the search result list or on the tenth page).
• Time spent on manual tasks and activities. This metric will measure how much time, in minutes/hours, is spent by the employees or users on manual tasks if the tool automates some of these tasks.
• Reduction of business process steps (if the steps are being automated). To derive this metric, create a map of the business process before the introduction of the AI-powered tool and after, and determine if the tool helped to simplify the process by reducing the number of process steps.

Bibliography

Adryan, Boris. “Is it all machine learning?” Badryan, Oct. 20, 2015. Accessed Feb. 2022.

“AI-Powered Data Management Platform.” Informatica, N.d. Accessed Feb 2022.

Amazon Rekognition. “Automate your image and video analysis with machine learning.” AWS. N.d. Accessed Feb 2022.

“Artificial Intelligence (AI).” IBM Cloud Education, 3 June 2020. Accessed Feb 2022.

“Artificial intelligence (AI) vs machine learning (ML).” Microsoft Azure Documentation. Accessed Feb. 2022.

“Avante Garde in the Realm of AI” SearchUnify Cognitive Platform. Accessed Feb 2022.

“Azure Cognitive Services.” Microsoft. N.d. Accessed Feb 2022.

“Becoming an AI-fueled organization. State of AI in the enterprise, 4th edition,” Deloitte, 2020. Accessed Feb. 2022.

“Coveo Predictive Search.” Coveo, N.d. Accessed Feb 2022.

”Data and AI Leadership. Executive Survey 2022. Executive Summary of Findings.” NewVantage Partners. Accessed Feb 2022.

“Einstein Discovery in Tableau.” Tableau, N.d. Accessed Feb 2022.

Korolov, Maria. “9 biggest hurdles to AI adoption.” CIO, Feb 26, 2019. Accessed Feb 2022.

Meel, Vidushi. “What Is Deep Learning? An Easy to Understand Guide.” visio.ai. Accessed Feb. 2022.

Mitchell, Tom. “Machine Learning,” McGraw Hill, 1997.

Stewart, Matthew. “The Actual Difference Between Statistics and Machine Learning.” Towards Data Science, Mar 24, 2019. Accessed Feb 2022.

“Sentiment analysis with Cognitive Services.” Microsoft Azure Documentation. Accessed February 2022.

“Three Principles for Designing ML-Powered Products.” Spotify Blog. Oct 2019, Accessed Feb 2022.

“Video Intelligence API.” Google Cloud Platform. N.d. Accessed Feb 2022