Secure IT/OT Convergence

Create a holistic IT/OT security culture.

Analyst Perspective

Are you ready for secure IT/OT convergence?

IT/OT convergence is less of a convergence and more of a migration. The previously entirely separate OT ecosystem is migrating into the IT ecosystem, primarily to improve access via connectivity and to leverage other standard IT capabilities for economic benefit.

In the past, OT systems were engineered to be air gapped, relying on physical protection and with little or no security in design, (e.g. OT protocols without confidentiality properties). However, now, OT has become dependent on the IT capabilities of the organization, thus OT inherits IT’s security issues, that is, OT is becoming more vulnerable to attack from outside the system. IT/OT convergence is complex because the culture, policies, and rules of IT are quite foreign to OT processes such as change management, and the culture, policies, and rules of OT are likewise foreign to IT processes.

A secure IT/OT convergence can be conceived of as a negotiation of a strong treaty between two systems: IT and OT. The essential initial step is to begin with communication between IT and OT, followed by necessary components such as governing and managing OT security priorities and accountabilities, converging security controls between IT and OT environments, assuring compliance with regulations and standards, and establishing metrics for OT security.

Ida Siahaan Research Director, Security and Privacy Practice Info-Tech Research Group

Executive Summary

Info-Tech Insight

Returning to isolated OT is not beneficial for the organization, so IT and OT need to learn to collaborate, starting with communication to build trust and to overcome their differences. Next, negotiation is needed on components such as governance and management, security controls on OT environments, compliance with regulations and standards, and establishing metrics for OT security.

Consequences of unsecure IT/OT convergence

OT systems were built with no or little security design 90% of organizations that use OT experienced a security incident. (Fortinet, 2021. Ponemon, 2019.)

(Source: Fortinet, 2021.)

Lack of visibility 86% of OT security-related service engagements lack complete visibility of OT network in 2021 (90% in 2020, 81% in 2019). (Source: “Cybersecurity Year In Review” Dragos, 2022.)

The need for secure IT/OT convergence

Important Industrial Control System (ICS) cyber incidents

2000 Target: Australian sewage plant. Method: Insider attack. Impact: 265,000 gallons of untreated sewage released.	2012 Target: Middle East energy companies. Method: Shamoon. Impact: Overwritten Windows-based systems files.	2014 Target: German Steel Mill. Method: Spear-phishing. Impact: Blast furnace failed to shut down.	2017 Target: Middle East safety instrumented system (SIS). Method: TRISIS/TRITON. Impact: Modified SIS ladder logic.	2022 Target: Viasat’s KA-SAT network. Method: AcidRain. Impact: Significant loss of communication for the Ukrainian military, which relied on Viasat’s services.
1903 Target: Marconi wireless telegraph presentation. Method: Morse code. Impact: Fake message sent “Rats, rats, rats, rats. There was a young fellow of Italy, Who diddled the public quite prettily.”	2010 Target: Iranian uranium enrichment plant. Method: Stuxnet. Impact: Compromised programmable logic controllers (PLCs).	2013 Target: ICS supply chain. Method: Havex. Impact: Remote Access Trojan (RAT) collected information and uploaded data to command-and-control (C&C) servers	2016 Target: Ukrainian power grid. Method: BlackEnergy. Impact: For 1-6 hours, power outages for 230,000 consumers.	2021 Target: Colonial Pipeline. Method: DarkSide ransomware. Impact: Compromised billing infrastructure halted the pipeline operation.

(Source: US Department of Energy, 2018.

”Significant Cyber Incidents,” CSIS, 2022

MIT Technology Review, 2022.)

Info-Tech Insight

Most OT incidents start with attacks against IT networks and then move laterally into the OT environment. Therefore, converging IT and OT security will help protect the entire organization.

The Secure IT/OT Convergence Framework

Phases Plan Enhance Monitor & Optimize

Plan Outcomes Mapping business goals to IT/OT security goals RACI chart for priorities and accountabilities Compliance obligations register Readiness checklist Enhance Outcomes Security strategy for IT/OT convergence Risk management framework Security policies & procedures IR, DR, BCP Monitor & Optimize Outcomes Security awareness and training Security architecture and controls

Plan Benefits Improved flexibility and less divided IT/OT Improved compliance Enhance Benefits Increased strategic common goals Increased efficiency and versatility Monitor & Optimize Benefits Enhanced security Reduced costs

Plan

Initiate communication

Map organizational goals to IT/OT security goals

Input: Corporate, IT, and OT strategies

Output: Your goals for the security strategy

Materials: Secure IT/OT Convergence Requirements Gathering Tool

Participants: Executive leadership, OT leader, IT leader, Security leader, Compliance, Legal, Risk management

1. As a group, brainstorm organization goals.
   1. Review relevant corporate, IT, and OT strategies.
2. Record the most important business goals in the Secure IT/OT Convergence Requirements Gathering Tool. Try to limit the number of business goals to no more than 10 goals. This limitation will be critical to helping focus on your secure IT/OT convergence.
3. For each goal, identify one to two security alignment goals. These should be objectives for the security strategy that will support the identified organization goals.

Download the Secure IT/OT Convergence Requirements Gathering Tool

Record organizational goals

Refer to the Secure IT/OT Convergence Framework when filling in the following elements.

1. Record your identified organization goals in the Goals Cascade tab of the Secure IT/OT Convergence Requirements Gathering Tool.
2. For each of your organizational goals, identify IT alignment goals.
3. For each of your organizational goals, identify OT alignment goals.
4. For each of your organizational goals, select one to two IT/OT security alignment goals from the drop-down lists.

Establish scope and boundaries

It is important to know at the outset of the strategy: What are we trying to secure in IT/OT convergence ?
This includes physical areas we are responsible for, types of data we care about, and departments or IT/OT systems we are responsible for.

Record scope and boundaries

Refer to the Secure IT/OT Convergence Framework when filling in the following elements: Record your security-related organizational scope, physical location scope, IT systems scope, and OT systems scope in the Scope tab of the Secure IT/OT Convergence Requirements Gathering Tool. For each item scoped, give the rationale for including it in the comments column. Careful attention should be paid to any elements that are not in scope.

Plan

Define roles and responsibilities

Input: List of relevant stakeholders

Output: Roles and responsibilities for the secure IT/OT convergence program

Materials: Secure IT/OT Convergence RACI Chart Tool

Participants: Executive leadership, OT leader, IT leader, Security leader

There are many factors that impact an organization’s level of effectiveness as it relates to IT/OT convergence. How the two groups interact, what skill sets exist, the level of clarity around roles and responsibilities, and the degree of executive support and alignment are only a few. Thus, it is imperative in the planning phase to identify stakeholders who are:

• Responsible: The people who do the work to accomplish the activity; they have been tasked with completing the activity and/or getting a decision made.
• Accountable: The person who is accountable for the completion of the activity. Ideally, this is a single person and will often be an executive or program sponsor.
• Consulted: The people who provide information. This is usually several people, typically called subject matter experts (SMEs).
• Informed: The people who are updated on progress. These are resources that are affected by the outcome of the activities and need to be kept up to date.

Download the Secure IT/OT Convergence RACI Chart Tool

Define RACI Chart

Plan

Establish governance and build cross-functional team

To establish governance and build an IT/OT cross-functional team, it is important to understand the operation of OT systems and their interactions with IT within the organization, e.g. ad hoc, centralized, decentralized.

Info-Tech Insight

To determine IT/OT convergence maturity level, Info-Tech provides the IT/OT Convergence Self-Evaluation Tool.

Centralized security governance model example

Plan

Identify convergence elements and compliance obligations

Identify compliance obligations

Source: ENISA, 2013 DHS, 2009.

OT system has compliance obligations with industry regulations and security standards/regulations/guidelines. See the lists given. The lists are not exhaustive. OT system owner can use the standards/regulations/guidelines as a benchmark to determine and manage the security level provided by third parties. It is important to understand the various frameworks and to adhere to the appropriate compliance obligations, e.g. IEC/ISA 62443 - Security for Industrial Automation and Control Systems Series.

IEC/ISA 62443 - Security for Industrial Automation and Control Systems Series

(Source: Cooksley, 2021)

IEC/ISA 62443 is a comprehensive international series of standards covering security for ICS systems, which recognizes three roles, namely: asset owner, system integrator, and product manufacturer. In IEC/ISA 62443, requirements flow from the asset owner to the product manufacturer, while solutions flow in the opposite direction. For the asset owner who owns and operates a system, IEC 62443-2 enables defining target security level with reference to a threat level and using the standard as a benchmark to determine the current security level. For the system integrator, IEC 62443-3 assists to evaluate the asset owner’s requirements to create a system design. IEC 62443-3 also provides a method for verification that components provided by the product manufacturer are securely developed and support the functionality required.

Record your compliance obligations

Refer to the “Goals Cascade” tab of the Secure IT/OT Convergence Requirements Gathering Tool. Identify your compliance obligations. Most organizations have compliance obligations that must be adhered to. These can include both mandatory and voluntary obligations. Mandatory obligations include: Laws Government regulations Industry standards Contractual agreements Voluntary obligations include standards that the organization has chosen to follow for best practices and any obligations that are required to maintain certifications. Organizations will have many different compliance obligations. For the purposes of your secure IT/OT convergence, include only those that have OT security requirements. Record your compliance obligations, along with any notes, in your copy of the Secure IT/OT Convergence Requirements Gathering Tool. Refer to the “Compliance DB” tab for lists of standards/regulations/guidelines.

Plan

Assess readiness

Readiness checklist for secure IT/OT convergence

(Source: “Grid Modernization: Optimize Opportunities And Minimize Risks,” Info-Tech)

Enhance

Update security strategy

To update security strategy, it is important to actively encourage visible sponsorship across management and to provide regular updates.

(Source: NIST SP 800-82 Rev.3, “Guide to Operational Technology (OT) Security,” NIST, 2022.)

OT system life cycle is like the IT system life cycle, starting with architectural design and ending with decommissioning. Currently, IT only gets involved from installation or maintenance, so they may not fully understand the OT system. Therefore, if OT security is compromised, the same personnel who commissioned the OT system (e.g. engineering, electrical, and maintenance specialists) must be involved. Thus, it is important to have the IT team collaborate with the OT team in each stage of the OT system’s life cycle. Finally, it is necessary to have propositional sharing of responsibilities between IT leaders, security leaders, and OT leaders who have broader responsibilities.

Enhance

Update risk management framework

The need for asset and threat taxonomy

One of issues in IT/OT convergence is that OT systems focus on production, so IT solutions like security patching or updates may deteriorate a machine or take a machine offline and may not be applicable. For example, some facilities run with reliability of 99.999%, which only allows maximum of 5 minutes and 35 seconds or less of downtime per year. Managing risks requires an understanding of the assets and threats for IT/OT systems. Having a taxonomy of the assets and the threats cand help. Applying normal IT solutions to mitigate security risks may not be applicable in an OT environment, e.g. running an antivirus tool on OT system may remove essential OT operations files. Thus, this approach must be avoided; instead, systems must be rebuilt from golden images.

(Source: ENISA, 2018.)

Enhance

Update security policies and procedures

Policy hierarchy example

This example of a policy hierarchy features templates from Info-Tech’s Develop and Deploy Security Policies and Identify the Best Framework for Your Security Policies research.

Enhance

Update IR, DR, and BCP

A proactive approach to security is important, so actions such as updating and testing the incident response plan for OT are a must. (“Cybersecurity Year In Review” Dragos, 2022.)

1. Customize organizational chart for IT/OT IR, DR, BCP based on governance and management model.
   E.g. ad hoc, internal distributed, internal centralized, combined distributed, and decentralized. (Software Engineering Institute, 2003)
2. Adjust the authority of the new organizational chart and decide if it requires additional staffing.
   E.g. full authority, shared authority. (Software Engineering Institute, 2003)
3. Update IR plan, DR plan, and BCP for IT/OT convergence.
   E.g. incorporate zero trust principles for converge network
4. Testing updated IR plan, DR plan, and BCP.

Optimize

Implement awareness, induction, and cross-training

Awareness may not correspond to readiness

Certifications

One of issues in certification is the complexity on relevancy in topics with respect to roles and levels. An example solution is the European Union Agency for Cybersecurity (ENISA)’s approach to analyzing existing certifications by orientation, scope, and supporting bodies, grouped into specific certifications, relevant certifications, and safety certifications. Specific cybersecurity certification of ICS/SCADA Example: ISA-99/IEC 62443 Cybersecurity Certificate Program, GIAC Global Industrial Cyber Security Professional (GICSP), Certified SCADA Security Architect (CSSA), EC-Council ICS/SCADA Cybersecurity Training Course. Other relevant certification schemes Example: Network and Information Security (NIS) Driving License, ISA Certified Automation Professional (CAP), Industrial Security Professional Certification (NCMS-ISP). Safety Certifications Example: Board of Certified Safety Professionals (BCSP), European Network of Safety and Health Professional Organisations (ENSHPO).

(Source: ENISA, 2015.)

Optimize

Design and deploy converging security architecture and controls

IT/OT convergence architecture can be modeled as a layered structure based on security. In this structure, the bottom layer is referred as “OT High-Security Zone” and the topmost layer is “IT Low-Security Zone.” In this model, each layer has its own set of controls configured and acts like an additional layer of security for the zone underneath it. The data flows from the “OT High-Security Zone” to the topmost layer, the “IT Low-Security Zone,” and the traffic must be verified to pass to another zone based on the need-to-know principle. In the normal control flow within the “OT High-Security Zone” from level 3 to level 0, the traffic must be verified to pass to another level based on the principle of least privilege. Remote access (dotted arrow) is allowed under strict access control and change control based on the zero-trust principle with clear segmentation and a point for disconnection between the “OT High-Security Zone” and the “OT Low-Security Zone” This model simplifies the security process, as if the lower layers have been compromised, then the compromise can be confined on that layer, and it also prevents lateral movement as access is always verified.

(Source: “Purdue Enterprise Reference Architecture (PERA) model,” ISA-99.)

Off-the-shelf solutions

Getting the right recipe: What criteria to consider?

Visibility and Asset Management Passive data monitoring using various protocol layers, active queries to devices, or parsing configuration files of OT, IoT, and IT environments on assets, processes, and connectivity paths. Threat Detection, Mitigation, and Response (+ Hunting) Automation of threat analysis (signature-based, specification-based, anomaly-based, sandboxing) not only in IT but also in relevant environments, e.g. IoT, IIoT, and OT on assets, data, network, and orchestration with threat intelligence sharing and analytics. Risk Assessment and Vulnerability Management Risk scoring approach (qualitative, quantitative) based on variables such as behavioral patterns and geolocation. Patching and vulnerability management. Usability, Architecture, Cost The user and administrative experience, multiple deployment options and extensive integration capabilities, and affordability.

Optimize

Establish and monitor IT/OT security metrics for effectiveness and efficiency

Further information

Related Info-Tech Research

Build an Information Security Strategy Info-Tech has developed a highly effective approach to building an information security strategy – an approach that has been successfully tested and refined for over seven years with hundreds of organizations. This unique approach includes tools for ensuring alignment with business objectives, assessing organizational risk and stakeholder expectations, enabling a comprehensive current-state assessment, prioritizing initiatives, and building a security roadmap.

Preparing for Technology Convergence in Manufacturing Information technology (IT) and operational technology (OT) teams have a long history of misalignment and poor communication. Stakeholder expectations and technology convergence create the need to leave the past behind and build a culture of collaboration.

Implement a Security Governance and Management Program Your security governance and management program needs to be aligned with business goals to be effective. This approach also helps provide a starting point to develop a realistic governance and management program. This project will guide you through the process of implementing and monitoring a security governance and management program that prioritizes security while keeping costs to a minimum.

Bibliography

Bibliography

Research Contributors and Experts

Jeff Campbell Manager, Technology Shared Services Horizon Power, AU Jeff Campbell has more than 20 years' experience in information security, having worked in both private and government organizations in education, finance, and utilities sectors. Having focused on developing and implementing information security programs and controls, Jeff is tasked with enabling Horizon Power to capitalize on IoT opportunities while maintaining the core security basics of confidentiality, integrity and availability. As Horizon Power leads the energy transition and moves to become a digital utility, Jeff ensures the security architecture that supports these services provides safer and more reliable automation infrastructures.

Christopher Harrington Chief Technology Officer (CTO) Carolinas Telco Federal Credit Union Frank DePaola Vice President, Chief Information Security Officer (CISO) Enpro Kwasi Boakye-Boateng Cybersecurity Researcher Canadian Institute for Cybersecurity

Infrastructure &Operations Priorities 2023

Navigate the liminal space between threats and opportunities.

2023: A liminal space between threats and opportunities

Introduction

Welcome to the Info-Tech 2023 I&O Priorities Report

Infrastructure & Operations priorities

The I&O priorities were determined by combining I&O member responses from the Tech Trends and Priorities 2023 survey with insightful signals from secondary research, economic markets, regulatory bodies, industry organizations, and vendors.

I&O Priorities 2023

I&O priorities framework

Challenge: Expand the I&O border

The hidden message in this report is that I&O priorities extend beyond the traditional scope of I&O functions. I&O members need to collaborate across functional areas to successfully address the priorities presented in this report. Info-Tech can help! Align your priorities with our material on how to Build a Business-Aligned IT Strategy. Use a modified version of the Strategy Initiative Template (next slide) to convey your strong opinion on the priorities you need your stakeholders to know about. And do so in a way that is familiar so they will easily understand.

Info-Tech 2023 Trends Survey Results

Call your Executive Advisor or Counselor to help identify the one or two key messages you want to bring forward for success in 2023!

Info-Tech IT Strategy Initiative Template, from the IT Strategy Presentation Template & Priorities Report Initiative Template .

Protect from threats

Get out of your silo. Forget your job description and just start doing what needs to be done.

Enhance I&O Cybersecurity

Produce ESG Reporting

Recession Readiness

Enhance cybersecurity response

SIGNALS

Enhance cybersecurity response

SIGNALS

Enhance cybersecurity response

SIGNALS

Enhance cybersecurity response

CALL TO ACTION

Get out of your I&O silo and form cross-functional cybersecurity teams.

I&O priority actions

Establish a cross-functional security steering committee to coordinate security processes and technologies. The complexity of managing security across modern applications, cloud, IoT, and network infrastructure that members operate is greater than ever before and requires coordinated teamwork.

Contain the cyber threat with zero trust (ZT) architecture. Extend ZT to network and critical infrastructure to limit exposure.

Leverage AI to build vigilant security intelligence. Smart I&O operators will make use of AI automation to augment their security technologies to help detect threats and contain security incidents on critical infrastructure.

Enhance cybersecurity response

I&O priority actions

Build specialized cybersecurity incident management protocols with your service desk. Build integrated security focused teams within service desk operations that continually test and improve security incident response protocols internally and with specialized security vendors. In some organizations, security incident response teams extend beyond traditional infrastructure into social media. Work cross-functionally to determine the risk exposure to misinformation and incident response procedures.

Treat lost or stolen equipment as a security incident. Develop hardware asset management protocols for tracking and reporting on these incidents and keep a record of equipment disposal. Implement tools that allow for remote deletion of data and report on lost or stolen equipment.

Produce ESG reporting

SIGNALS

Produce ESG reporting

SIGNALS

ESG mandates are being rolled out globally.

ESG reporting has greatly expanded since a 2017 report by Task Force on Climate-Related Financial Disclosures (TCFD, 2017) which recommended that organizations disclose climate-related financial metrics for investors to appropriately price climate-related risks to share price. In 2021, the Swiss Finance Institute research paper (Sautner, 2021) identified 29 countries that require ESG reporting, primarily for larger public companies, financial institutions, and state-owned corporations.

Global ESG mandates

29 nations with ESG mandates identified by the Swiss Finance Institute

Produce ESG reporting

SIGNALS

Produce ESG reporting

CALL TO ACTION

Being unprepared for new ESG reporting mandates without a clear and validated ESG reporting process puts your organization at risk.

I&O priority actions

Understand ESG risk exposure. Define the gap between what ESG reporting is required in your jurisdiction and current reporting capabilities to meet them. Build the I&O role with responsibility for ESG reporting.

Include vendors in ESG reporting. Review infrastructure facilities with landlords, utilities, and hosting providers to see if they can provide ESG reporting on sustainable power generation, then map it to I&O power consumption as part of their contractual obligations. Ask equipment vendors to provide ESG reporting on manufacturing materials and energy consumption to boot-strap data collection.

Implement a HAM process to track asset disposal and other types of e-waste. Update agreements with disposal vendors to get reporting on waste and recycle volumes.

Produce ESG reporting

I&O priority actions

Implement an ESG reporting framework. There are five major ESG reporting frameworks being used globally. Select one of the frameworks below that makes sense for your organization, and implement it.

ISO 14001 Environmental Management: Part of the ISO Technical Committee family of standards that allows your organization to understand its legal requirements to become certified in ESG.

Global Reporting Initiative (GRI) Sustainability Reporting Standards: GRI has been developing ESG reporting standards since 1997. GRI provides a modular ESG framework applicable to all sizes and sectors of organizations worldwide.

Principles for Responsible Investment: UN-developed framework for ESG reporting framework for disclosure in responsible investments.

Sustainability Accounting Standards Board (SASB): ESG report framework to be used by investors.

UN Global Compact: ESG reporting framework based on 10 principles that organizations can voluntarily contribute data to.

Implement a HAM process to track asset disposal and other types of e-waste. Update agreements with disposal vendors to get reports on waste and recycle volumes.

Recession readiness

SIGNALS

Recession readiness

SIGNALS

Recession readiness

SIGNALS

Tough choices on budgetary spends.

The responses indicated that I&O members expect decreased reinvestment for 2023 for the following:

• API programming (-21.7%)
• Cloud computing (-19.4%)
• 44% of I&O members indicated if 2023 requires costs cutting, 5-20% of their cloud computing investment will be at risk of the chopping block!
• Workforce management (-9.4%)
• No-code /low-code infrastructure (-5.3%)

Make sure you can clearly measure the value of all budgeted I&O activities.

Anything that can't demonstrate clear value to leadership is potentially on the chopping block.

Recession readiness

CALL TO ACTION

Get ahead of inflationary pressures with early budgetary planning, and identify the gap between the catch-up projects and required critical net new investments.

II&O priority actions

Hedge against inflation on infrastructure projects. Develop and communicate value-based strategies to lock in pricing and mitigate inflationary risk with vendors.

Communicate value-add on all I&O budgeted items. Define an infrastructure roadmap to highlight which projects are technical debt and which are new strategic investments, and note their value to the organization.

Look for cost saving technologies. Focus on I&O projects that automate services to increase productivity and optimize head count.

Realize opportunities

Build on a record of COVID-related innovation success and position the enterprise to take advantage of 2023.

AI governance: Watching the watchers

Data stewardship: Cornerstone of value

Prep for a brave new metaverse

AI governance: Watching the watchers

SIGNALS

AI governance: Watching the watchers

SIGNALS

AI governance: Watching the watchers

CALL TO ACTION

Establish I&O within an AI governance program to build trust in AI results and behaviors and limit legal exposure.

I&O priority actions

Define who has overall AI accountability for AI governance within I&O. This role is responsible for establishing strategic governance metrics over AI use and results, and identifying liability risks.

Maintain an inventory of AI use. Conduct an audit of where AI is used within I&O, and identify gaps in documentation and alignment with I&O processes and organizational values.

Define an I&O success map. Provide transparency of AI use by generating pseudo code of AI models, and scorecard AI decision making with expected predictions and behavioral actions taken.

AI governance: Watching the watchers

Manage bias in AI decision making. Work with AI technology vendors to identify how unethical bias can enter the results, using operational data sets for validation prior to rollout.

Protect AI data sets from manipulation. Generate new secure storage for AI technology audit trails on AI design making and results. Work with your security team to ensure data sets used by AI for training can’t be corrupted.

Data governance: Cornerstone of value

SIGNALS

Data governance: Cornerstone of value

SIGNALS

Approach to data analysis is primarily done in-house.

85% of surveyed I&O members are doing data analysis with custom-made or external tools. Interestingly, 10% of I&O members do not conduct any data analysis.

Members are missing a formal data governance process.

81% of surveyed I&O members do not have a formal or automated process for data governance. Ironically, 24% of members responded that they aim to have publicly accessible data-as-a-service or information repositories.

Despite investment in data initiatives, organizations carry high levels of data debt.

Info-Tech research, Establish Data Governance, points out that data debt, the accumulated cost associated with sub-optimal governance of data assets, is a problem for 78% of organizations.

What the enterprise expects out of enterprise storage is much more complicated in 2023.

Data protection and governance are non-negotiable aspects of enterprise storage, even when it’s unstructured.

Data governance: Cornerstone of value

SIGNALS

Data governance: Cornerstone of value

CALL TO ACTION

Develop a data governance program that includes an I&O data steward for oversight.

I&O priority actions

Establish an I&O data steward. Make data governance by establishing a data steward role with accountability for governance. The steward works collaboratively with DataOPs to control access to I&O data, enforce policies, and reduce the time to make use of the data.

Define a comprehensive storage architecture. If you thought you had a data sprawl problem before, wait until you see the volume of data generated from IoT and Web 3.0 applications. Get ahead of the problem by creating an infrastructure roadmap for structured and unstructured data storage.

Build a solid backbone for AI Operations using data quality best practices. Data quality is the foundation for generation of operational value from the data and artificial intelligence efforts. Focus on using a methodology to build a culture of data quality within I&O systems and applications that generate data rather than reactive fixes.

Look to partner with third-party vendors for your master data management (MDM) efforts. Modern MDM vendors can work with your existing data fabrics/lake and help leverage your data governance policies into the cloud.

Prep for a brave new metaverse

SIGNALS

From science fiction to science fact.

The term metaverse was coined in 1992 by Neal Stephenson and is a common theme in science fiction. For most I&O surveyed professionals, the term metaverse conjures up more confusion than clarity, as it’s not one place, but multiple metaverse worlds. The primordial metaverse was focused on multiplayer gaming and some educational experiences. It wasn’t until recently that it gained a critical mass in the fashion and entertainment industries with the use of non-fungible tokens (NFT). The pandemic created a unique opportunity for metaverse-related technologies to expand Web 3.0.

Related Info-Tech Research

• Into the Metaverse
• Establish a Foresight Capability
• Double Your Organization’s Effectiveness With a Digital Twin
• Adopt Blockchain for Supply Chain Transparency

Prep for a brave new metaverse

SIGNALS

Collaboration and beyond.

On one hand, metaverse technologies virtual reality(VR)/augmented reality (AR) headsets can be a method of collaborating internally within a single organization. About 10% of our surveyed I&O members engaged this type of collaborative metaverse in 2022, with another 24% looking to run proof of concept projects in 2023. However, there is a much larger terrain for metaverse projects outside of workforce collaboration, which 17% of surveyed I&O members are planning to engage with in 2023.

These are sophisticated new metaverse worlds, and digital twins of production environments are being created for B2B collaboration, operations, engineering, healthcare, architecture, and education that include the use of block chain, NFTs, smart contracts, and other Web 3.0 technologies

“They are the audiovisual bodies that people use to communicate with each other in the Metaverse.”

Neal Stephenson,
Snow Crash 1992

Prep for a brave new metaverse

SIGNALS

Metaverse requires multidimensional security.

Security in the context of the metaverse presents new challenges to I&O. The infrastructure that runs the metaverse is still vulnerable to “traditional” security threats. New attack vectors include financial and identity fraud, privacy and data loss, along with new cyber-physical threats which are predicted to occur as the metaverse begins to integrate with IoT and other 3D objects in the physical world.

The ultimate in "not a product" – the metaverse promises to be a hodgepodge of badly standardized technologies for the near future.

Be prepared to take care of pets and not cattle for the foreseeable future, but keep putting the fencing around the ranch.

Prep for a brave new metaverse

SIGNALS

Prep for a brave new metaverse

CALL TO ACTION

Ready or not, the metaverse is coming to an infrastructure near you. Start expanding I&O technologies and processes to support a metaverse infrastructure.

I&O priority actions

Develop a plan for network upgrades.

A truly immersive VR/AR experience requires very low latency. Identify gaps and develop a plan to enhance your network infrastructure surrounding your metaverse space(s) and end users.

Extend security posture into the metaverse.

Securing the infrastructure that runs your metaverse is going to extend the end-user equipment used to navigate it. More importantly, security policies need to encompass the avatars that navigate it and the spatial web that they interact with, which can include physical world items like IoT.

Prep for a brave new metaverse

I&O priority actions

Metaverse theft prevention

Leverage existing strategies to identify management in the metaverse. Privacy policies need to extend their focus to data loss prevention within the metaverse.

Collaborate

The skill set required to build, deploy, manage, and support the metaverse is complex. Develop a metaverse support organization that extends beyond I&O functions into security, DevOps, and end-user experiences.

Educate

Web 3.0 technologies and business models are complex. Education of I&O technical- and commerce-focused team members is going to help prevent you from getting blindsided. Seek out specialized training programs for technical staff and strategic education for executives, like the Wharton School of Business certification program.

Authors

John Annand Principal Research Director	Theo Antoniadis Principal Research Director	Contributors Paul Sparks, CTO at Brookshire Grocery Company 2 Anonymous Contributors
Figuring out the true nature of the “Turbo” button of his 486DX100 launched John on a 20-year career in managed services and solution architecture, exploring the secrets of HPC, virtualization, and DIY WANs built with banks of USR TotalControl modems. Today he focuses his research and advisory on software-defined infrastructure technologies, strategy, organization, and service design in an increasingly Agile and DevOps world.	Theo has decades of operational and project management experience with start-ups and multinationals across North America and Europe. He has held various consulting, IT management and operations leadership positions within telecommunications, SaaS, and software companies.

Bibliography

“3 Cybersecurity Trends that are Changing Financial Data Management." FIMA US. Accessed August 2022.
Arti. “While much of the world is just discovering the Metaverse, a number of universities have already established centers for studying Web 3." Analytics Insight. 10 July 2022.
“Artificial intelligence (AI) for cybersecurity." IBM. Accessed September 2022
“Business in the Metaverse Economy." Wharton School of University of Pennsylvania. Accessed October 2022.
“Cost of a data breach 2022: A million-dollar race to detect and respond." IBM. Accessed September 2022.
“Countries affected by mandatory ESG reporting – here’s the list." New Zealand Ministry of Business, Innovation & Employment. Accessed September 2022.
“Countries affected by mandatory ESG reporting – here’s the list.” WorldFavor. Accessed September 2022.
Crenshaw, Caroline A. “SEC Proposes to Enhance Disclosures by Certain Investment Advisers and Investment Companies About ESG Investment Practices." U.S. Securities and Exchange Commission. May 2022.
“Cutting through the metaverse hype: Practical guidance and use cases for business." Avanade. Accessed October 2022.
“Data Governance Global Market Sees Growth Rate Of 25% Through 2022." The Business Research Company. August 2022.
“DIRECTIVE 2014/95/EU OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 22 October 2014 amending Directive 2013/34/EU as regards disclosure of non-financial and diversity information by certain large undertakings and groups." UER-Lex. Accessed September 2022.
"Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems." IEEE. March 2019.
“European Parliament resolution of 20 October 2020 with recommendations to the Commission on a civil liability regime for artificial intelligence." European Parliament. Accessed October 2022.
Ghose, Ronit et al. "Metaverse and Money." Citi GPS. March 2022.
Hernandez, Roberto, et al. “Demystifying the metaverse." PWC. Accessed August 2022.
Info-Tech Trends Report Survey, 2023; N=813.
“ISO 14000 Family: Environmental Management." ISO. Accessed October 2022.
Knight, Michelle & Bishop, Annie, ”The 2022 State of Cloud Data Governance.“ Zaloni DATAVERSITY. 2022.

Bibliography

Kompella, Kashyap, “What is AI governance and why do you need it?“ TechTarget. March 2022.
“Management of electronic waste worldwide in 2019, by method." Statista. 2022.
“Model Artificial Intelligence Governance Framework and Assessment Guide.“ World Economic Forum. Accessed September 2022.
“Model Artificial Intelligence Governance Framework." PDPC Singapore. Accessed October 2022.
“New rules on corporate sustainability reporting: provisional political agreement between the Council and the European Parliament.“ European Council. June 2022.
"OECD Economic Outlook Volume 2022." OECD iLibrary. June 2022.
"Recommendations of the Task Force on Climate-related Financial Disclosures." TCFD. Accessed August 2022.
“Risk of Global Recession in 2023 Rises Amid Simultaneous Rate Hikes.” World Bank Organization. September 2022.
Sautner, Zacharias, et al. “The Effects of Mandatory ESG Disclosure around the World.” SSRN. November 2021.
Sondergaard, Peter. “AI GOVERNANCE – WHAT ARE THE KPIS? AND WHO IS ACCOUNTABLE?“ The Sondergaard Group. November 2019.
Srivastavam Sudeep, “How can your business enter the Metaverse?." Appinventiv.
September 2022.
“Standards Overview." SASB. Accessed October 2022.
Stephenson, Neal. Snow Crash. Bantam Books, 1992.
“Sustainability Reporting Standards." Global Reporting Initiative. Accessed October 2022.
“The Ten Principles of the UN Global Compact." UN Global Compact. Accessed October 2022.
Tian Tong Lee, Sheryl. "China Unveils ESG Reporting Guidelines to Catch Peers.” Bloomberg. May 2022.
“What are the Principles for Responsible Investment?" UNPRI. Accessed October 2022.
"What is the EU's Corporate Sustainability Reporting Directive (CSRD)?" WorldFavor.
June 2022.
West, Darrell M. “Six Steps to Responsible AI in the Federal Government.“ Brookings Institution. March 2022. Web.

Implement Risk-Based Vulnerability Management

Get off the patching merry-go-round and start mitigating risk!

Table of Contents

Table of Contents

Analyst Perspective

Vulnerabilities will always be present. Know the unknowns!

In this age of discovery, technology changes at such a rapid pace. New things are discovered, both in new technology and in old. The pace of change can often be very confusing as to where to start and what to do.

The ever-changing nature of technology means that vulnerabilities will always be present. Taking measures to address these completely will consume all your department’s time and resources. That, and your efforts will quickly become stale as new vulnerabilities are uncovered. Besides, what about the systems that simply can’t be patched? The key is to understand the vulnerabilities and the levels of risk they pose to your organization, to prioritize effectively and to look beyond patching.

A risk-based approach to vulnerability management will ensure you are prioritizing appropriately and protecting the business. Reduce the risk surface!

Vulnerability management is more than just systems and application patching. It is a full process that includes patching, compensating controls, segmentation, segregation, and heightened diligence in security monitoring.

Jimmy Tom Research Advisor – Security, Privacy, Risk, and Compliance Info-Tech Research Group

Executive Summary

Info-Tech Insight

Vulnerability management does not always equal patch management. There is more than one way to tackle the problem, particularly if a system cannot be easily patched or replaced. If a vulnerability cannot be completely remediated, steps to reduce the risk to a tolerable level must be taken.

Common obstacles

These barriers make vulnerability management difficult to address for many organizations: The value of vulnerability management is not well articulated in many organizations. As a result, investment in vulnerability scanning technology is often insufficient. Many organizations feel that a “patch everything” approach is the most effective path. Vulnerability management is commonly misunderstood as being a process that only supports patch management. There is often misalignment between SecOps and ITOps in remediation action and priority, affecting the timeliness of remediation.

CVSS Score Distribution From the National Vulnerability Database: (Source: NIST National Vulnerability Database Dashboard)

Leverage risk to sort, triage, and prioritize vulnerabilities

Reduce your risk surface to avoid cost to your business; everything else is table stakes.

Reduce the critical and high vulnerabilities below the risk threshold and operationalize the remediation of medium/low vulnerabilities by following your effective vulnerability management program cycles.

Identify vulnerability sources

An inventory of your scanning tool and vulnerability threat intelligence data sources will help you determine a viable strategy for addressing vulnerabilities. Defining roles and responsibilities ahead of time will ensure you are not left scrambling when dealing with vulnerabilities.

Triage and prioritize

Bring the vulnerabilities into context by assessing vulnerabilities based on your security posture and mechanisms and not just what your data sources report. This will allow you to gauge the true urgency of the vulnerabilities based on risk and determine an effective mitigation plan.

Remediate vulnerabilities

Address the vulnerabilities based on their level of risk. Patching isn't the only risk mitigation action; some systems simply cannot be patched, but other options are available.

Reduce the risk down to medium/low levels and engage your regular operational processes to deal with the latter.

Measure and formalize

Upon implementation of the program, measure with metrics to ensure that the program is successful. Improve the program with each iteration of vulnerability mitigation to ensure continuous improvement.

Tactical Insight 1

All actions to address vulnerabilities should be based on risk and the organization’s established risk tolerance.

Tactical Insight 2

Reduce the risk surface down below the risk threshold.

The industry has shifted to a risk-based approach

Traditional vulnerability management is no longer viable.

“For those of us in the vulnerability management space, ensuring that money, resources, and time are strategically spent is both imperative and difficult. Resources are dwindling fast, but the vulnerability problem sure isn’t.” (Kenna Security)

“Using vulnerability scanners to identify unpatched software is no longer enough. Keeping devices, networks, and digital assets safe takes a much broader, risk-based vulnerability management strategy – one that includes vulnerability assessment and mitigation actions that touch the entire ecosystem.” (Balbix)

“Unlike legacy vulnerability management, risk-based vulnerability management goes beyond just discovering vulnerabilities. It helps you understand vulnerability risks with threat context and insight into potential business impact.” (Tenable)

“A common mistake when prioritizing patching is equating a vulnerability’s Common Vulnerability Scoring System (CVSS) score with risk. Although CVSS scores can provide useful insight into the anatomy of a vulnerability and how it might behave if weaponized, they are standardized and thus don’t reflect either of the highly situational variables — namely, weaponization likelihood and potential impact — that factor into the risk the vulnerability poses to an organization.” (SecurityWeek)

Why a take risk-based approach?

Vulnerabilities, by the numbers

60% — In 2019, 60% of breaches were due to unpatched vulnerabilities.

74% — In the same survey, 74% of survey responses said they cannot take down critical applications and systems to patch them quickly. (Source: SecurityBoulevard, 2019)

Info-Tech Insight

Taking a risk-based approach will allow you to focus on mitigating risk, rather than “just patching” your environment.

The average cost of a breach in 2020 is $3.86 million, and “…the price tag was much less for mature companies and industries and far higher for firms that had lackluster security automation and incident response processes.” (Dark Reading)

Info-Tech’s vulnerability management methodology

Focus on developing the most efficient processes.

Vulnerability management isn’t “old school.”

The vulnerability management market is relatively mature; however, vulnerability management remains a very relevant and challenging topic.

Security practitioners are inundated with the advice they need to prioritize their vulnerabilities. Every vulnerability scanning vendor will proclaim their ability to prioritize the identified vulnerabilities.

Third-party prioritization methodology can’t be effectively applied across all organizations. Each organization is too unique with different constraints. No tool or service can account for these variables.

When patching is not possible, other options exist: configuration changes (hardening), defense-in-depth, compensating controls, and even elevated security monitoring are possible options.

Info-Tech Insight

Vulnerability management is not only patch management. Patching is only one aspect.

Blueprint deliverables

Each step of this blueprint is accompanied by supporting deliverables to help you accomplish your goals:

Key deliverable: Vulnerability Management SOP The Standard operating procedure (SOP) will comprise the end-to-end description of the program: roles & responsibilities, data flow, and expected outcomes of the program.	Vulnerability Management Policy Template for your vulnerability management policy.		Vulnerability Tracking Tool This tool offers a template to track vulnerabilities and how they are remedied.
	Vulnerability Scanning RFP Template Request for proposal template for the selection of a vulnerability scanning tool.		Vulnerability Risk Assessment Tool Methodology to assess vulnerability risk by determining impact and likelihood.

Blueprint benefits

Info-Tech’s process can save significant financial resources

Guided Implementation

A Guided Implementation (GI) is a series of calls with an Info-Tech analyst to help implement our best practices in your organization.

A typical GI is between 8 to 12 calls over the course of 4 to 6 months.

What does a typical GI on this topic look like?

Workshop Overview

Contact your account representative for more information.
workshops@infotech.com 1-888-670-8889

Implement Risk-Based Vulnerability Management

Phase 1

Identify Vulnerability Sources

This phase will walk you through the following activities:

Establish a common understanding of vulnerability management, define the roles, scope, and information sources of vulnerability detection.

This phase involves the following participants:

• Security operations team
• IT Security Manager
• IT Director
• CISO

Step 1.1

Vulnerability Management Defined

Activities

None for this section

This step will walk you through the following activities:

Establish a common understanding of vulnerability management and its place in the IT organization.

This step involves the following participants:

• Security operations team
• IT Security Manager
• IT Director
• CISO

Outcomes of this step

Foundational knowledge of vulnerability management in your organization.

What is vulnerability management?

It’s more than just patching.

Effective vulnerability management

It’s not easy, but it’s much harder without a process in place. Effective vulnerability management requires a formal process for organizations to follow; without one, vulnerabilities are dealt with in an ad hoc fashion. Patching isn’t the only solution, but it’s the one that often draws focus. Responsibilities for the different aspects of vulnerability management are often unclear, such as for testing, remediation, and implementation. Identifying new threats without proper vulnerability scanning tools can be a near-impossible task. Determining which vulnerabilities are most urgent can be an inconsistent process, increasing the organizational risk. Measuring the effectiveness of your vulnerability remediation activities can help you better manage resources in SecOps and ITOps. Your staff will be spending the appropriate effort on vulnerabilities that warrant that level of attention.

You’re not just doing this for yourself. It’s also for your auditors. Many compliance and regulatory obligations require organizations to have thorough documentation of their vulnerability management practices.

Vulnerability management revolves around your asset security services

Vulnerabilities can be found primarily within your assets but also connect to your information risk management. These must be effectively managed as part of a holistic security program. Without management, vulnerabilities left unattended can be easy for attackers to exploit. It becomes difficult to identify the correct remediation option to mitigate against the vulnerabilities.

Vulnerability management works in tandem with SecOps and ITOps

Vulnerability Management Process Inputs/Outputs: Arrows denote direction of information feed

Vulnerability management serves as the input into a number of processes for remediation, including: Incident management, to deal with issues Release management, for patch management Change management, for change control IT asset management, to track version information, e.g. for patching Application security testing, for the verification of vulnerabilities A two-way data flow exists between vulnerability management and: Security risk management, for the overall risk posture of the organization Threat intelligence, as vulnerability management reveals only one of several threat vectors

For additional information please refer to Info-Tech’s research for each area:

Info-Tech’s Information Security Program Framework

Vulnerability management is a component of the Infrastructure Security section of Security Management

For more information, review our Build an Information Security Strategy blueprint, or speak to one of our analysts. Info-Tech Insight Vulnerability management is but one piece of the information security puzzle. Ensure that you have all the pieces!

Case Study

INDUSTRY: Manufacturing SOURCE: Cimpress, 2016

One organization is seeing immediate benefits by formalizing its vulnerability management program.

Step 1.2

Defining the scope and roles

Activities

• 1.2.1 Define the scope and boundary of your organization’s security program
• 1.2.2 Assign responsibility for vulnerability identification and remediation

This step will walk you through the following activities:

Define and understand the scope and boundary of the security program. For example, does it include OT? Define roles and responsibilities for vulnerability identification and remediation

This step involves the following participants:

• Security operations team
• IT Security Manager
• IT Director
• CISO

Outcomes of this step

Understand how far vulnerability management extends and what role each person in IT plays in the remediation of vulnerabilities

Determine the scope of your security program

This will help you adjust the depth and breadth of your vulnerability management program. Determining the scope will help you decide how much organizational risk the vulnerability management program will oversee. Scope can be defined along four aspects: Data Scope – What data elements in your organization does your security program cover? How is data classified? Physical Scope – What physical scope, such as geographies, does the security program cover? Organizational Scope – How are business units engaged with security initiatives? Does the scope cover all subsidiary organizations? IT Scope – What parts of the organization does IT cover? Does their coverage include operational technology (OT) and industrial control systems (ICS)?

1.2.1 Define the scope and boundary of your organization’s security program

Input: List of Data Scope, Physical Scope, Organization Scope, and IT Scope

Output: Defined scope and boundaries of the IT security program

Materials: Whiteboard/Flip Charts, Sticky Notes, Markers, Vulnerability Management SOP Template

Participants: Business stakeholders, IT leaders, Security team members

1. On a whiteboard, write the headers: Data Scope, Physical Scope, Organizational Scope, and IT Scope.
2. Give each group member a handful of sticky notes. Ask them to write down as many items as possible for the organization that could fall under one of the four scope buckets.
3. In a group, discuss the sticky notes and the rationale for including them. Discuss your security-related locations, data, people, and technologies, and define their scope and boundaries.

The goal is to identify what your vulnerability management program is responsible for and document it.

Consider the following:

How is data being categorized and classified? How are business units engaged with security initiatives? How are IT systems connected to each other? How are physical locations functioning in terms of information security management?

Download the Vulnerability Management SOP Template

Assets are part of the scope definition

An inventory of IT assets is necessary if there is to be effective vulnerability management.

Assign responsibility for vulnerability identification and remediation

Determine who is critical to effectively detecting and managing vulnerabilities. Some of the remediation steps will involve members of IT management to identify the true organizational risk of a vulnerability. Vulnerability remediation comes in different shapes and sizes. In addition to patching, this can include implementing compensating controls, server and application hardening, or the segregating of vulnerable systems. Who carries out each of these activities? Who coordinates the activities and tracks them to ensure completion? The people involved may be members outside of the security team, such as members from IT operations, infrastructure, and applications. The specific roles that each of these groups play should be clearly identified.

1.2.2 Assign responsibility for vulnerability identification and remediation

Input: Sample list of vulnerabilities and requisite actions from each group, High-level organizational chart with area functions

Output: Defined set of roles and responsibilities for member groups

Materials: Vulnerability Management SOP Template

Participants: CIO, CISO, IT Management representatives for each area of IT

1. Display the table of responsibilities that need to be assigned.
2. List all the positions within the IT security team.
3. Map these to the positions that require IT security team members.
4. List all positions that are part of the IT team.
5. Map these to the positions that require IT team members.

If your organization does not have a dedicated IT security team, you can perform this exercise by mapping the relevant IT staff to the different positions shown on the right.

Download the Vulnerability Management SOP Template

Step 1.3

Cloud considerations for vulnerability management

Activities

None for this section.

This step will walk you through the following activities:

Review cloud considerations for vulnerability management

This step involves the following participants:

• Security operations team
• IT Security Manager
• IT Director
• CISO

Outcomes of this step

Understand the various types of cloud offerings and the implications (and limitations) of vulnerability management in a cloud environment.

Cloud considerations

Cloud will change your approach to vulnerability management. There will be a heavy dependence on the cloud service provider to ensure that vulnerabilities in their foundational technologies have been addressed. Depending on the level of “as-a-Service,” customers will have varying degrees of control and visibility into the underlying operations. With vendor acquiescence, you can set your tool to scan a given cloud environment, depending on how much visibility you have into their environment based on the service you have purchased. Due to compliance obligations of their customers, there is a growing trend among cloud providers to allow more scanning of cloud environments. In the absence of customer scanning capability, vendors may offer attestation of vulnerability management and remediation.

For more information, see Info-Tech Research Group’s Document Your Cloud Strategy blueprint.

Cloud environment scanning

Cloud scanning is becoming a more common necessity but still requires special consideration.

An organization’s cloud environment is just an extension of its own environment. As such, cloud environments need to be scanned for vulnerabilities.

Step 1.4

Vulnerability detection

Activities

• 1.4.1 Develop a monitoring and review process of third-party vulnerability sources
• 1.4.2 Incident management and vulnerability management

This step will walk you through the following activities:

Create an inventory of your vulnerability monitoring capability and third-party vulnerability information sources.

Determine how incident management and vulnerability management interoperate.

This step involves the following participants:

• Security operations team
• IT Security Manager
• IT Director
• CISO

Outcomes of this step

Catalog of vulnerability information data sources. Understanding of the intersection of incident management and vulnerability management.

Vulnerability detection

Vulnerabilities can be identified through numerous mediums.

Info-Tech has determined the following to be the four most common ways to identify vulnerabilities.

Automate with a vulnerability scanning tool

Vulnerability scanning tool considerations

Select a vulnerability scanning tool with the features you need to be effective. Vulnerability scanning tool selection can be an exciting and confusing process. You will need to consider what features you desire in a tool and whether you want the tool to go beyond just scanning and reporting. In addition to vulnerability scanning, some tools will integrate with your IT service management (service desk ticketing system) tool and asset, configuration, and change management modules. This can facilitate the necessary workflow that the remediation process follows once a vulnerability is discovered. A number of vulnerability scanning tool vendors have started offering remediation as part of their software features. This includes the automation and orchestration functionality and configuration and asset management to track its remediation activities. A side benefit of the asset discovery feature in vulnerability scanning tools is that it can help enhance an organization’s asset inventory and license compliance, particularly in cases where end users are able to install software on their workstations.

For guidance on tool vendors Visit SoftwareReviews for information on vulnerability management tools and vendors.

Vulnerability scanning tool best practices

How often should scans be performed?

One-off scans provide snapshots in time. Repeated scans over time provide tracking for how systems are changing and how well patches are being applied and software is being updated.

The results of a scan (asset inventory, configuration data, and vulnerability data) are basic information needed to understand your security posture. This data needs to be as up to date as possible.

ANALYST PERSPECTIVE: Organizations should look for continuous scanning

Continuous scanning is the concept of providing continual scanning of your systems so any asset, configuration, or vulnerability information is up to date. Most vendors will advertise continuous scanning but you need to be skeptical of how this feature is met.

Continuous Scanning Methods

Vulnerability scanning tool best practices

Where to perform a scan.

Vulnerability scanning tool best practices

Pre-Scan Communication With Users

• It is always important to inform owners and users of systems that a scan will be happening.
• Although it is unlikely any performance issues will arise, it is important to notify end users of potential impact.
• Local admins or system owners may have controls in place that stop vulnerability scans and you need to inform the owners so that they can safelist the scanner you will be using.

Vulnerability Scanning Tool Tracking Metrics

• Vulnerability score by operating system, application, or organization division.
  • This provides a look at the widely accepted severity of the vulnerability as it relates across the organization’s systems.
• Most vulnerable applications and application version.
  • This provides insight into how outdated applications are creating risk exposure for an organization.
  • This will also provide metrics on the effectiveness of your patching program.
• Number of assets scanned within the last number of days.
  • This provides visibility into how often your assets are being scanned and thus protected.
• Number of unowned devices or unapproved applications.
  • This metric will track how many unowned devices or unapproved applications may be on your network. Unowned devices may be rogue devices or just consultant/contractor devices.

Third-party vulnerability information sources

IT security forums and mailing lists are another source of vulnerability information.

Proactively identify new vulnerabilities as they are announced.

By monitoring for vulnerabilities as they are announced through industry alerts and open-source mechanisms, it is possible to identify vulnerabilities beyond your scanning tool’s penetration tests.

Common sources: Vendor websites and mailing lists Vendors are the trusted sources for vulnerability and patch information on their products, particularly with new industry vulnerability disclosure requirements. Vendors are the most familiar with their products, downloads are most likely malware free, and additional information is often included. There are some issues: vendors won’t announce a vulnerability until a patch is created, which creates a potential unknown risk exposure; numerous vendor sites will have to be monitored continually. Third-party websites A non-vendor site providing information on vulnerabilities. They often will cover a specific technology or an industry section, becoming a potential “one-stop shop” for some. They will often provide vulnerability information that is augmented with different remediation recommendations faster than vendors. However, it’s more likely that malicious code could be downloaded and it will often not be comprehensive information on patching. Third-party mailing lists, newsgroups, live paid subscriptions, and live open-source feeds These are alerting and notification services for the detection and dissemination of vulnerability information. They provide information on the latest and most critical vulnerabilities, e.g. US-CERT Cybersecurity Alerts. Vulnerability databases These usually consist of dedicated databases on vulnerabilities. They perform the hard work of identifying and aggregating vulnerability and patch information into a central repository for end-user consumption. The commentary features on these databases provide excellent insight for practitioners, e.g. National Vulnerability Database (NVD).

Third-party vulnerability information sources

IT security forums and mailing lists are another source of vulnerability information.

Third-party sources for vulnerabilities

• Open Source Vulnerability Database (OSVDB)
  • An open-source database that is run independently of any vendors.
• Common Vulnerabilities and Exposures (CVE)
  • Free, international dictionary of publicly known information security vulnerabilities and exposures.
• National Vulnerability Database (NVD)
  • Through NIST, the NVD is the US government’s repository of vulnerabilities and includes product names, flaws, and any impact metrics.
  • The National Checklist Repository Program (NCRP), also provided by NIST, provides security checklists for configurations of operating systems and applications.
  • The Center for Internet Security, a separate entity unrelated to NIST, provides configuration benchmarks that are often referenced by the NCRP.
• Open Web Application Security Project (OWASP)
  • OWASP is another free project helping to expose vulnerabilities within software.
• US-CERT National Cyber Alert System (US-CERT Alerts)
  • Cybersecurity Alerts – Provide timely information about current security issues, vulnerabilities, and exploits.
  • Cybersecurity Tips – Provide advice about common security issues for the general public.
  • Cybersecurity Bulletins – Provide weekly summaries of new vulnerabilities. Patch information is provided when available.
• US-CERT Vulnerability Notes Database (US-CERT Vulnerability Notes)
  • Database of searchable security vulnerabilities that were deemed not critical enough to be covered under US-CERT Alerts. Note that the NVD covers both US-CERT Alerts and US-CERT Notes.
• Open Vulnerability Assessment Language (OVAL)
  • Coding language for security professionals to discuss vulnerability checking and configuration issues. Vulnerabilities are identified using tests that are disseminated in OVAL definitions (XML executables that can be used by end users).

1.4.1 Develop a monitoring and review process for third-party vulnerability sources

Input: Third-party resources list

Output: Process for review of third-party vulnerability sources

Materials: Whiteboard, Whiteboard markers, Vulnerability Management SOP Template

Participants: IT Security Manager, SecOps team members, ITOps team members, CISO

1. Identify what third-party resources are useful and relevant.
2. Shortlist your third-party sources.
3. Identify what is the best way to receive information from a third party.
4. Document the method to receive or check information from the third-party source.
5. Identify who is responsible for maintaining third-party vulnerability information sources
6. Capture this information in the Vulnerability Management SOP Template.

Download the Vulnerability Management SOP Template

Incidents and vulnerability management

Incidents can also be a sources of vulnerabilities.

When any incident occurs, for example:

• A security incident, such as malware detected on a machine
• An IT incident, such as an application becomes unresponsive
• A crisis occurs, like a worker accident

There can be underlying vulnerabilities that need to be processed.

1.4.2 Incident management and vulnerability management

Input: Existing incident response processes, Existing crisis communications plans

Output: Alignment of vulnerability management program with existing incident management processes

Materials: Whiteboard, Whiteboard markers, Vulnerability Management SOP Template

Participants: IT Security Manager, SecOps team members, ITOps team members, including tiers 1, 2, and 3, CISO, CIO

1. Inventory what incident response plans the organization has. These include:
   1. Information Security Incident Response Plan
   2. IT Incident Plan
   3. Problem Management Plan
   4. Crisis Management Plan
2. Identify what part of those plans contains the post-response recap or final analysis.
3. Formalize a communication process between the incident response plan and the vulnerability mitigation process.

Note: Most incident processes will cover some sort of root cause analysis and investigation of the incident. If a vulnerability of any kind is detected within this analysis it needs to be reported on and treated as a detected vulnerability, thus warranting the full vulnerability mitigation process.

Download the Vulnerability Management SOP Template

Implement Risk-Based Vulnerability Management

Phase 2

Triage & prioritize

This phase will walk you through the following activities:

Examine the elements that you will use to triage and analyze vulnerabilities, prioritizing using a risk-based approach, and prepare for remediation options.

This phase involves the following participants:

• IT Security Manager
• SecOps team members
• ITOps team members, including tiers 1, 2, and 3
• CISO
• CIO

Step 2.1

Triage vulnerabilities

Activities

• 2.1.1 Evaluate your identified vulnerabilities

This step will walk you through the following activities:

Review your vulnerability information sources and determine a methodology that will be used to consistently evaluate vulnerabilities as your scanning tool alerts you to them.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• ITOps team members, including tiers 1, 2, and 3
• CISO
• CIO

Outcomes of this step

A consistent, documented process for the evaluation of vulnerabilities in your environment.

Triaging vulnerabilities

Use Info-Tech’s methodology to allocate urgencies to your vulnerabilities to assign the appropriate resources to each one.

When evaluating numerous vulnerabilities, use the following three factors to help determine the urgency of vulnerabilities:

• The intrinsic qualities of the vulnerability
• The business criticality of the affected asset
• The sensitivity of the data stored on the affected asset

Intrinsic qualities of the vulnerability — Vulnerabilities need to be examined for the inherent risk they pose specifically to the organization, which includes if an exploit has been identified or if the industry views this as a serious and likely threat.

Business criticality of the affected asset — Assets with vulnerabilities need to be assessed for their criticality to the business. Vulnerabilities on systems that are critical to business operations or customer interactions are usually top of mind.

Sensitivity of the data of the affected asset — Beyond just the criticality of the business, there must be consideration of the sensitivity of the data that may be compromised or modified as a result of any vulnerabilities.

Info-Tech Insight

This methodology allows you to determine urgency of vulnerabilities, but your remediation approach needs to be risk-based, within the context of your organization.

Triage your vulnerabilities, filter out the noise

Triaging enables your vulnerability management program to focus on what it should focus on. Use the Info-Tech Vulnerability Mitigation Process Template to define how to triage vulnerabilities as they first appear. Triaging is an important step in vulnerability management, whether you are facing ten to tens of thousands of vulnerability notifications. Many scanning tools already provide the capability to compare known vulnerabilities against existing assets through integration with the asset inventory. There are two major use cases for this process: For organizations that have identified vulnerabilities but do not know their own systems well enough. This can be due to a lack of a formal asset inventory. For proactive organizations that are regularly staying up to date with industry announcements regarding vulnerabilities. Once an alert has been made publicly, this process can assist in confirming if the vulnerability is relevant to the organization.

The Info-Tech methodology for initial triaging of vulnerabilities: Even if neither of these use cases apply to your organization, triaging still addresses the issues of false positives. Triaging provides a quick way to determine if vulnerabilities are relevant.

After eliminating the noise, evaluate your vulnerabilities to determine urgency

Consider the intrinsic risk to the organization.

• For a vulnerability to become a true threat to the organization, it must be exploited to cause damage. In today’s threat landscape, exploit kits are sold online that allow individuals with low technical knowledge to exploit a vulnerability.
• Not all vulnerabilities have an associated exploit, but this does not mean that these vulnerabilities can be left alone. In many cases, it is just a matter of time before an exploit is created.
• Another point to consider is that while exploits can exist theoretically, they may not be verified. Vulnerabilities always pose some level of risk, but if there are no known verified exploits, there is less risk attached.

• Common Vulnerability Scoring System (CVSS) is an open-source industry scoring method to assess the potential severity of vulnerabilities.
• CVSS takes into account: attack vector, complexity, privileges required, user interaction, scope, confidentiality impact, integrity impact, and availability impact.
• Vulnerabilities that have a score of 4.0 or lower are classified as low vulnerabilities, while scores between 4.0 and 6.9 are put in the medium category. Scores of 7 or higher are in the high and critical categories. As we will review in the Risk Assessment section, you will want to immediately deal with high and critical vulnerabilities.

• Even though a vulnerability may appear to be part of an inconsequential asset, it is important to consider whether it can be leveraged to gain access to other areas of the network or system by an attacker.
• Another consideration should be whether the vulnerability can be exploited by remote or local access. Remote exploits pose a greater risk as this can mean that attackers can perform an exploit from any location. Local exploits carry less risk, although the risk of insider threats should be considered here as well.

2.1.1 Evaluate your identified vulnerabilities

Input: Visio workflow of Info-Tech’s vulnerability management process

Output: Adjusted workflow to reflect your current processes, Vulnerability Tracking Tool

Materials: Whiteboard, Whiteboard markers, Vulnerability Management SOP Template

Participants: IT Security Manager, SecOps team members, ITOps team members, including tiers 1, 2, and 3, CISO, CIO

Using the criteria from the previous slide, Info-Tech has created a methodology to evaluate your vulnerabilities by examining their intrinsic qualities.

The methodology categorizes the vulnerabilities into high, medium, and low risk importance categorizations, before assigning final urgency scores in the later steps.

1. Review the evaluation process in the Vulnerability Management Workflow library.
2. Determine if this process makes sense for the organization; otherwise, change the flow to include any other considerations of process flows.
3. As this process is used to evaluate vulnerabilities, document vulnerabilities to an importance category. This can be done in the Vulnerability Tracking Tool or using a similar internal vulnerability tracking document, if one exists.

Download the Vulnerability Management SOP Template

Step 2.2

Determine high-level business criticality

Activities

• 2.2.1 Determine high-level business criticality
• 2.2.2 Determine your high-level data classifications

This step will walk you through the following activities:

Determining high-level business criticality and data classifications will help ensure that IT security is aligned with what is critical to the business. This will be very important when decisions are made around vulnerability risk and the urgency of remediation action.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• CISO

Outcomes of this step

Understanding and consistency in how business criticality and business data is assessed by IT in the vulnerability management process.

Understanding business criticality is key to determining vulnerability urgency

Prioritize operations that are truly critical to the operation of the business, and understand how they would be impacted by an exploited vulnerability.

2.2.1 Determine high-level business criticality

Input: List of business operations, Insight into business operations impacts to the business

Output: List of business operations and their criticality and impact to the business

Materials: Vulnerability Management SOP Template

Participants: Participants from the business, IT Security Manager, CISO, CIO

1. List your core business operations at a high level.
2. Use a High, Medium, or Low ranking to prioritize the business operations based on mission-critical criteria and the impact of the vulnerability.
3. When using the process flow, consider if the vulnerability directly affects any of these business operations and move through the process flow based on the corresponding High, Medium, or Low ranking.

Example prioritization of business operations for a manufacturing company:

Questions to ask: Is there a hard-dollar impact from downtime? Is there impact on goodwill or customer trust? Is regulatory compliance a factor? Is there a health or safety risk?

Download the Vulnerability Management SOP Template

Determine vulnerability urgency by its data classification

Consider how to classify your data based on if the Confidentiality, Integrity, or Availability (CIA) is compromised.

To properly classify your data, consider how the confidentiality, integrity, and availability of that data would be affected if it were to be exploited by a vulnerability. Review the table below for an explanation for each objective. Confidentiality Preserving authorized restrictions on information access and disclosure, including means for protecting personal privacy and proprietary information. Integrity Guarding against improper information modification or destruction, and ensuring information non-repudiation and authenticity. Availability Ensuring timely and reliable access to and use of information. Each piece of data should be ranked as High, medium, or low across confidentiality, integrity, and availability based on adverse effect. Low — Limited adverse effect Moderate — Serious adverse effect High — Severe or catastrophic adverse effect If you wish to build a whole data classification methodology, refer to our Discover and Classify Your Data blueprint.

How to determine data classification when CIA differs: The overall ranking of the data will be impacted by the highest objective’s ranking. For example, if confidentiality and availability are low, but integrity is high, the overall impact is high. This process was developed in part by Federal Information Processing Standards Publication 199.

2.2.2 Determine your high-level data classifications

Input: Knowledge of data use and sensitivity

Output: Adjusted workflow to reflect your current processes, Vulnerability Tracking Tool

Materials: Whiteboard, Whiteboard markers, Vulnerability Management SOP Template

Participants: IT Security Manager, CISO, CIO

If your organization has formal data classification in place, it should be leveraged to determine the high, medium, and low rankings necessary for the process flows. However, if there is no formal data classification in place, the process below can be followed:

1. List common assets or applications that are prone to vulnerabilities.
2. Consider the data that is on these devices and provide a high (severe or catastrophic adverse effect), medium (serious adverse effect), or low (limited adverse effect) ranking based on confidentiality, availability, and integrity.
   1. Use the table on the previous slide to assist in providing the ranking.
   2. Remember that it is the highest ranking that dictates the overall ranking of the data.
3. Document which data belongs in each of the categories to provide contextual evidence.

Download the Vulnerability Management SOP Template

This process should be part of your larger data classification program. If you need assistance in building this out, review the Info-Tech research, Discover and Classify Your Data.

Step 2.3

Consider current security posture

Activities

• 2.3.1 Document your defense-in-depth controls

This step will walk you through the following activities:

Your defense-in-depth controls are the existing layers of security technology that protects your environment. These are relevant when considering the urgency and risk of vulnerabilities in your environment, as they will mitigate some of the risk.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• ITOps team members, including tiers 1, 2, and 3
• CISO
• CIO

Outcomes of this step

Understanding and documentation of your current defense-in-depth controls.

Review your current security posture

What you have today matters. In most cases, your vulnerability scanning tool alone will not have the context of your security posture in the results of its scans. This can skew the true urgency of detected vulnerabilities in your environment. What you have in place today is what comprises your organization’s overall security posture. This bears high relevance to the determination of the risk that a vulnerability poses to your environment. Elements such as enterprise architecture and defense in depth mechanisms should be factored into determining the risk of a vulnerability and what kind of immediacy is warranted to address it. Details of your current security posture will also contribute to the assessment and selection of remediation options.

Enterprise architecture considerations

Defense-in-depth

Defense-in-depth provides extra layers of protection to the organization.

2.3.1 Document your defense-in-depth controls

Input: List of technologies within your environment, List of IT security controls that are in place

Output: List of defense-in-depth controls

Materials: Whiteboard/flip charts, Vulnerability Management SOP Template

Participants: IT Security Manager, Infrastructure Manager, IT Director, CISO

1. Document the existing defense-in-depth controls within your system.
2. Review the initial list that has been provided and see if these are controls that currently exist.
3. Indicate any other controls that are being used by the organization. This may already exist if you have a security services catalog.
4. Indicate who the owners of the different controls are.
5. Track the information in the Vulnerability Management SOP Template.

Download the Vulnerability Management SOP Template

Step 2.4

Risk assessment of vulnerabilities

Activities

• 2.4.1 Build a classification scheme to consistently assess impact
• 2.4.2 Build a classification scheme to consistently assess likelihood

This step will walk you through the following activities:

Assessing risk will be the cornerstone of how you evaluate vulnerabilities and what priority you place on remediation. This is actual risk to the organization and not simply what the tool reports without the context of your defense-in-depth controls.

This step involves the following participants:

• IT Security Manager
• IT Operations Management
• CISO
• CIO

Outcomes of this step

A risk matrix tailored to your organization, based on impact and likelihood. This will provide a consistent, unambiguous way to assess risk across the vulnerability types that is reported by your scanning tool.

Vulnerabilities and risk

Vulnerabilities must be addressed to mitigate risk to the business. Vulnerabilities are a concern because they are potential threats to the business. Vulnerabilities that are not addressed can turn from potential threats into actual threats; it is only a matter of time and opportunity. Your organization will already be familiar with risk management, as every decision carries a business risk component. There may even be a senior manager assigned as corporate risk officer to manage organizational risk. The organization likely has a risk tolerance level that defines the organization’s risk appetite. This may be measured in dollars, non-productivity time, or other units of inefficiency. The risk of a vulnerability can be calculated using impact and likelihood. Impact is the effect that the vulnerability will have if it is exploited by a malicious actor. Likelihood is the degree to which a vulnerability exploit can possibly occur.

Info-Tech Insight Risk to the organization is business language that everyone can understand. This is particularly true when the risk is to productivity or to the company’s bottom line.

A risk-based approach to vulnerability management

CVSS scores are just the starting point!

Prioritize remediation by levels of risk

Address critical and high risk with high immediacy.

Addressing the critical and high-risk vulnerabilities with urgency will ensure that you are addressing a more manageable number of vulnerabilities. An optimized vulnerability management process will address the medium and low risk vulnerabilities within the regular cycle. This may be very similar to what you do today in an ad hoc fashion: Zero-day vulnerabilities tend to warrant a stop in operations and are dealt with immediately (or as soon as a vendor has a fix). The standard remediation process (patching/updating, change of configuration, etc.) happens within a regular controlled time cycle. Formalizing this process will ensure that appropriate attention is given to vulnerabilities that warrant it and that the remaining vulnerabilities are dealt with as a regular, recurring activity.

Mitigate the risk surface by reducing the time across the phases

Risk matrix

Risk = Impact x Likelihood Info-Tech’s Vulnerability Management Risk Assessment Tool provides a method of calculating the risk of a vulnerability. The risk rating is assigned using the impact of the risk and the likelihood or probability that the event may occur. The tool puts the vulnerability into your organization’s context: How many people will be affected? What service types are vulnerable and how does that impact the business? Is there an anticipated update from the vendor of the system being affected? Urgency of remediation should be based on the business consequences if the vulnerability were to be exploited, relative to the business’ risk tolerance. Info-Tech Insight Risk determination should be done within the context of your current environment and not simply based on what your vulnerability tool is reporting.

A risk matrix is useful in calculating a risk rating for vulnerabilities.

2.4.1 Build a classification scheme to consistently assess impact

Input: Knowledge of IT environment, Knowledge of business impact for each IT component or service

Output: Vulnerability Management Risk Assessment Tool formatted to your organization

Materials: Vulnerability Management Risk Assessment Tool

Participants: Functional Area Managers, IT Security Manager, CISO

Risk always has a negative impact, but the size of the impact can vary considerably in terms of cost, number of people or sites affected, and the severity of the impact. Impact questions tend to be more objective and quantifiable than likelihood questions.

1. Define a set of questions to measure risk impact or edit existing questions in the tool.
2. For each question, assign a weight that should be placed on that factor.
3. Define criteria for each question that would categorize the risk. The drop-down box content can be modified in the hidden Labels tab.

Note that you are looking to baseline vulnerability types, rather than categorizing every single vulnerability your scanning tool reports. The volume of vulnerabilities will be high, but vulnerabilities can be categorized into types on a regular basis.

Download the Vulnerability Management Risk Assessment Tool

2.4.2 Build a classification scheme to consistently assess likelihood

Input: Knowledge of IT environment, Knowledge of business impact for each IT component or service

Output: Vulnerability Management Risk Assessment Tool formatted to your organization

Materials: Vulnerability Management Risk Assessment Tool

Participants: Functional Area Managers, IT Security Manager, CISO

Risk always has a negative impact, but the size of the impact can vary considerably in terms of cost, number of people or sites affected, and the severity of the impact. Impact questions tend to be more objective and quantifiable than likelihood questions.

1. Define a set of questions to measure risk impact or edit existing questions in the tool.
2. For each question, assign a weight that should be placed on that factor.
3. Define criteria for each question that would categorize the risk. The drop-down box content can be modified in the hidden Labels tab.

Note that you are looking to baseline vulnerability types, rather than categorizing every single vulnerability that your scanning tool reports. The volume of vulnerabilities will be high, but vulnerabilities can be categorized into types on a regular basis.

Download the Vulnerability Management Risk Assessment Tool

Prioritize based on risk

Implement Risk-Based Vulnerability Management

Phase 3

Remediate vulnerabilities

This phase will walk you through the following activities:

• Identifying potential remediation options.
• Developing criteria for each option with regards to when to use and when to avoid.
• Establishing exception procedure for testing and remediation.
• Documenting the implementation of remediations and verification.

This phase involves the following participants:

• CISO, or equivalent
• Security Manager/Analyst
• Network, Administrator, System, Database Manager
• Other members of the vulnerability management team
• Risk managers for the risk-related steps

Determining how to remediate

Patching is only one option.

Step 3.1

Assessing remediation options

Activities

• 3.1.1 Develop risk and remediation action

This step will walk you through the following activities:

With the risk assessment from the previous activity, we can now examine remediation options and make a decision. This activity will guide us through that.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• ITOps team members, including tiers 1, 2, and 3
• CISO
• CIO

Outcomes of this step

List of remediation options and criteria on when to consider each.

Identify remediation options

There are four options when it comes to vulnerability remediation.

Patches and updates

Patches are often the easiest and most common method of remediation.

Compensating controls

Compensating controls can decrease the risk of vulnerabilities that cannot be (immediately) remediated.

Configuration changes

Accept the risk and do nothing

By choosing not to remediate vulnerabilities, you must accept the associated risk. This should be your very last option.

Every time that a vulnerability is not remediated, it continues to pose a risk to the organization. While it may seem that every vulnerability needs to be remediated, this is simply not possible due to limited resources. Further, it can take away resources from other security initiatives as opposed to low-priority vulnerabilities that are extremely unlikely to be exploited.

Risk considerations

When determining if risk acceptance is appropriate, consider the cost of not mitigating vulnerabilities.

Don’t accept the risk because it seems easy. Consider the financial impact of leaving vulnerabilities open.

3.1.1 Develop risk and remediation action

Input: List of remediation options

Output: List of remediation options sorted into “when to use” and “when to avoid” lists

Materials: Whiteboard/flip charts, Vulnerability Management SOP Template

Participants: IT Security Manager, IT Infrastructure Manager, IT Operations Manager, Corporate Risk Officer, CISO

It is important to define and document your organization-specific criteria for when a remediation option is appropriate and inappropriate.

1. List each remediation option on a flip chart and create two headings: “When to use” and “When to avoid.”
2. Each person will list “when to use” criteria on a green sticky note and “when to avoid” criteria on a red one for each option; these will be placed on the appropriate flip chart.
3. Discuss as a group which criteria are appropriate and which should be removed.
4. Move on to the next remediation option when completed.
   • Ensure to include when there are remediation options that will be connected. For example, the risk may be accepted until the next available change window, or a defense-in-depth control is used before a patch can be fully installed.
5. Once the criteria has been established, document this in the Vulnerability Management SOP Template.

Download the Vulnerability Management SOP Template

Step 3.2

Scheduling and executing remediation

Activities

None for this section.

This step will walk you through the following activities:

Although there are no specific activities for this section, it will walk you through your existing processes configuration and change management to ensure that you are leveraging those activities in your vulnerability remediation actions.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• ITOps team members, including tiers 1, 2, and 3
• CISO
• CIO

Outcomes of this step

Gained understanding of how IT operations processes configuration and change management can be leveraged for the vulnerability remediation process. Don’t reinvent the wheel!

Implementing the remediation

Vulnerability management converges with your IT operations functions. Once a remediation strategy has been formulated, you can leverage your release and change management processes to orchestrate the testing, version tracking, scheduling, approval, and implementation activities. Each of these processes should exist in your environment in some form. Leveraging these will engage the IT operations team to carry out their tasks in the remediation process. There can be a partial or full handoff to these processes, however, the owner of the vulnerability management program is responsible for verifying the application of the remediation measure and that the overall risk has been reduced. Although full blueprints exist that cover each of these processes in great detail, the following slides provide an overview of each of these IT operations processes and how they intersect with vulnerability management.

Release Management

Control the quality of deployments and releases of software updates.

Change Management

Post-implementation activities

Step 3.3

Continuous improvement

Activities

None for this section.

This step will walk you through the following activities:

Although this section has no activities, it will review the process by which you may continually improve vulnerability management.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• ITOps team members, including tiers 1, 2, and 3
• CISO
• CIO

Outcomes of this step

An understanding of the importance of ongoing improvements to the vulnerability management program.

Drive continuous improvement

Continuous Improvement

Continuously re-evaluate the vulnerability management process.

Implement Risk-Based Vulnerability Management

Phase 4

Measure and formalize

This phase will walk you through the following activities:

• You will determine what ought to be measured to track the success of your vulnerability management program.
• If you lack a scanning tool this phase will help you determine tool selection.
• Lastly, penetration testing is a good next step to consider once you have your vulnerability management program well underway.

This phase involves the following participants:

• IT Security Manager
• SecOps team members
• Procurement representatives
• CISO
• CIO

Step 4.1

Metrics, Key Performance Indicators (KPIs), and Critical Success Factors (CSFs)

Activities

• 4.1.1 Measure your program with metrics, KPIs, and CSFs

This step will walk you through the following activities:

After a review of the differences between raw metrics, key performance indicators (KPI), and critical success factors (CSF), compile a list of what metrics you will be tracking, why, and the business goals for each.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• CISO
• CIO

Outcomes of this step

Outline of metrics you can configure your vulnerability scanning tool to report on.

You can’t manage what you can’t measure

Metrics provides visibility.

Management consultant Peter Drucker introduced the concept of metrics tied to key performance indicators (KPIs), and the concept holds true: without metrics, you lack the visibility to manage or improve a process. Metrics aren’t just a collection of statistics, they have to be meaningful, they have to tell the story, and most importantly, they have to answer the “so what?” question. What is the significance of a metric – do they illustrate a trend or an anomaly? What actions should be carried out when a metric hits a certain threshold? It would be prudent to track several metrics that can be combined to tell the full story. For example, tracking the number of critical vulnerabilities alone does not give a sense of the overall risk to the organization, nor does it offer any information on how quickly they have been remediated or what amount of effort was invested.

Metrics, KPIs, and CSFs

Tracking relevant information

Tell the story in the numbers.

4.1.1 Measure your program with metrics, KPIs, and CSFs

Input: List of metrics current being measured by the vulnerability management tool

Output: List of relevant metrics to track, and the KPIs, CSFs, and business goals related to the metric

Materials: Whiteboard/flip charts, Vulnerability Management SOP Template

Participants: IT Security Manager, IT operations management, CISO

Metrics can offer a way to view how the organization is dealing with vulnerabilities and if there is improvement.

1. Determine the high-level vulnerability management goals for the organization.
2. Even with a formal process in place, the organization should be considering ways it can improve.
3. Determine metrics that can help quantify those goals and how they can be measured.
4. Metrics should always be easy to measure. If it’s a complex process to find the information required, it means that it is not a metric that should be used.
5. Document your list of metrics in the Vulnerability Management SOP Template.

Download the Vulnerability Management SOP Template

Step 4.2

Vulnerability Management Policy

Activities

• 4.2.1 Update the vulnerability management program policy

This step will walk you through the following activities:

If you have a vulnerability management policy, this activity may help augment it. Otherwise, if you don’t have one, this would be a great starting point.

This step involves the following participants:

• IT Security Manager
• CISO
• CIO
• Human resources representative

Outcomes of this step

An inaugural policy covering vulnerability management

Vulnerability Management Program Policy

Policies provide governance and enforcement of processes. Policies offer formal guidance on the “rules” of a program, describing its purpose, scope, detailed program description, and consequences of non-compliance. Often they will have a employee sign-off acknowledging understanding. In many organizations, policies are endorsed by senior executives, which gives the policy its “teeth” across the company. The human resources department will always have input due to the implications of the non-compliance aspect. Policies are written to ensure an outcome of consistent expected behavior and are often written to protect the company from liability. Policies should be easy to understand and unambiguous, reflect the current state, and be enforceable. Enforceability can come in the form of audit, technology, or any other means of determining compliance and enforcing behavior.

4.2.1 Update the vulnerability management policy

Input: Vulnerability Management SOP, HR guidance on policy creation and approval

Output: Completed Vulnerability Management Policy

Materials: Vulnerability Management SOP, Vulnerability Management Policy Template

Participants: IT Security Manager, IT operations management, CISO, Human resources representative

After having built your entire process in this project, formalize it into a vulnerability management policy. This will set the standards and expectations for vulnerability management in the organization, while the process will be around the specific actions that need to be taken around vulnerability management.

This is separate and distinct from the Vulnerability Management SOP Template, which is a process and procedure document. Review Info-Tech’s Vulnerability Management Policy and customize it to your organization’s specifications. Use your Vulnerability Management SOP as a resource when specifying some of the details within the policy.

Download the Vulnerability Management Policy Template

Step 4.3

Select and implement a scanning tool

Activities

• 4.3.1 Create an RFP for vulnerability scanning tools

This step will walk you through the following activities:

If you need to select a new vulnerability scanning tool, or replace your existing one, this activity will help set up a request for proposal (RFP).

This step involves the following participants:

• IT Security Manager
• SecOps team members
• CISO

Outcomes of this step

The provisions needed for you to create and deploy an RFP for a vulnerability management tool.

Vulnerability management and penetration testing

Similar in nature, yet provide different security functions.

Vulnerability scanning software

Vulnerability scanning market

Vulnerability scanning tool features

Vulnerability scanning deployment options

Vulnerability scanning methods

IP Management and IPv6

4.3.1 Create an RFP for vulnerability scanning tools

Input: List of key feature requirements for the new tool, List of intersect points with current software, Network topology and layout of servers and applications

Output: Completed RFP document that can be distributed to vendor proponents

Materials: Whiteboard/flip charts, Vulnerability Scanning Tool RFP Template

Participants: IT Security Manager, IT operations managers, CISO, Procurement department representative

Use a request for proposal (RFP) template to convey your desired scanning tool requirements to vendors and outline the proposal and procurement steps set by your organization.

1. Determine what kind of requirements will be needed for your scanning tool RFP, based on people, process, and technology requirements.
2. Consider items such as the desired capabilities and the scope of the scanning.
3. Conduct interviews with relevant stakeholders to determine the exact requirements needed.
4. Use Info-Tech’s Vulnerability Scanning Tool RFP Template. It lists many requirements but can be customized to your organization’s specific needs.

Download the Vulnerability Scanning Tool RFP Template

4.3.1 Create an RFP for vulnerability scanning tools (continued)

Download the Vulnerability Scanning Tool RFP Template

Step 4.4

Penetration testing

Activities

• 4.1.1 Create an RFP for penetration tests

This step will walk you through the following activities:

We will review penetration testing, its distinction from vulnerability management, and why you may want to engage a penetration testing service.

We provide a request for proposal (RFP) template that we can review if this is an area of interest.

This step involves the following participants:

• IT Security Manager
• SecOps team members
• CISO
• CIO

Outcomes of this step

An understanding of penetration testing, and guidance on how to get started if there is interest to do so.

Penetration testing

Penetration tests are critical parts of any strong security program.

Start by considering the spectrum of penetration tests

Info-Tech Insight

Your pen test should use multiple methods. Demonstrating weakness in one area is good but easy to identify. When you blend techniques, you get better success at breaching and it becomes more life-like. Think about prevention, detection, and response testing to provide full insight into your security defenses.

Penetration testing types

Evaluate four variables to determine which type of penetration test is most appropriate for your organization.

Penetration testing scope

Establish the scope of your penetration test before engaging vendors.

Determining the scope of what is being tested is the most important part of a penetration test. Organizations need to be as specific as possible so the vendor can actually respond or ask questions.

Organizations need to define boundaries, objectives, and key success factors.

What would be out of scope?

• Are there systems, IP addresses, or other things you want out of scope? These are things you don’t explicitly want any penetration tester to touch.
• Are there third-party connections to your environment that you don’t want to be tested? These are instances such as cloud providers, supply chain connections, and various services.
• Are there things that would be awkward to test? For example, determine if you include high-level people in a social engineering test. Do you conduct social engineering for the CEO? If you get their credentials, it could be an awkward moment.

Ways to break up a penetration test:

• Location – This is the most common way to break up a penetration test.
• Division – Self-contained business units are often done as separate tests so you can see how each unit does.
• IT systems – For example, you put certain security controls in a firewall and want to test its effectiveness.
• Applications – For example, you are launching a new website or a new portal and you want to test it.

Penetration testing appropriateness

Determine your penetration testing appropriateness.

4.4.1 Create an RFP for penetration tests

Input: List of criteria and scope for the penetration test, Systems and application information if white box

Output: Completed RFP document that can be distributed to vendor proponents

Materials: Whiteboard/flip charts, Penetration Test RFP Template

Participants: IT Security Manager, IT operations managers, CISO, Procurement department representative

Use an RFP template to convey your desired penetration test requirements to vendors and outline the proposal and procurement steps set by your organization.

1. Determine what kind of requirements will be needed for your penetration test RFP based on people, process, and technology requirements.
   • Consider items such as your technology environment and the scope of the penetration tests.
2. Conduct an interview with relevant stakeholders to determine the exact requirements needed.
3. Use Info-Tech’s Penetration Test RFP Template, which lists many requirements but can be customized to your organization’s specific needs.

Download the Penetration Test RFP Template

4.4.1 Create an RFP for penetration tests (continued)

Download the Penetration Test RFP Template

Penetration testing considerations – service providers

Consider what type of penetration testing service provider is best for your organization

Professional Service Providers

Professional Services Firms. These firms will often provide a myriad of professional services across auditing, financial, and consulting services. If they offer security-related consulting services, they will most likely offer some level of penetration testing.

Security Service Firms. These are dedicated security consulting or advisory firms that will offer a wide spectrum of security-related services. Penetration testing may be one aspect of larger security assessments and strategy development services.

Dedicated Penetration Testing Firms. These are service providers that will often offer the full gamut of penetration testing services.

Integrators

Managed Security Service Providers. These providers will offer penetration testing. For example, Dell SecureWorks offers numerous services including penetration testing. For organizations like this, you need to be skeptical of ulterior motives. For example, expect recommendations around outsourcing from Dell SecureWorks.

Regional or Small Integrators. These are service providers that provide security services of some kind. For example, they would help in the implementation of a firewall and offer penetration testing services as well.

Info-Tech Recommends:

• Always be conscientious of who is conducting the testing and what else they offer. Even if you get another party to test rather than your technology provider, they will try to obtain you as a client. Remember that for larger technology vendors, security testing is a small revenue stream for them and it’s a way to find technology clients. They may offer penetration testing for free to obtain other business.
• Most of the penetration testers were systems administrators (for network testing) or application developers (for application testing) at some point before becoming penetration testers. Remember this when evaluating providers and evaluating remediation recommendations.
• Evaluate what kind of open-source tools, commercial tools, and proprietary tools are being used. In general, you don’t want to rely on an open-source scanner. For open source, they will have more outdated vulnerability databases, system identification can also be limited compared to commercial, and reporting is often lacking.
• Above all else, ensure your testers are legally capable, experienced, and abide by non-disclosure agreements.

Penetration testing best practices – communications

Penetration testing best practices – results and remediation

What to expect from penetration test results report:

A final results report will state all findings including what was done by the testers, what vulnerabilities or exploitations were detected, how they were compromised, the related risk, and related remediation recommendations.

• Introduction. An overview of the penetration test methodology including rating methodology of vulnerabilities.
• Executive Summary. A management-level description of the test, often including a summary of any recommendations.
• Technical Review. An overview of each item that was looked at and touched. This area breaks down what was done, how it was done, what was found, and any related remediation recommendations. Expect graphs and visuals in this section.
• Detailed Findings. An in-depth breakdown of all testing methods used and results. Each vulnerability will be explained regarding how it was detected, what the risk is, and what the remediation recommendation is.

Summary of Accomplishment

Problem Solved

At the conclusion of this blueprint, you will have created a full vulnerability management program that will allow you to take a risk-based approach to vulnerability remediation.

Assessing a vulnerability’s risk will enable you to properly determine the true urgency of a vulnerability within the context of your organization; this ensures you are not just blindly following what the tool is reporting.

The risk-based approach will allow you to prioritize your discovered vulnerabilities and take immediate action on critical and high vulnerabilities while allowing your standard remediation cycle to address the medium to low vulnerabilities.

With your program defined and developed, you now need to configure your vulnerability scanning tool or acquire one if you don’t already have a tool in place.

Lastly, while vulnerability management will help address your systems and applications, how do you know if you are secure from external malicious actors? Penetration testing will offer visibility, allowing you to plug those holes and attain an environment with a smaller risk surface.

If you would like additional support, have our analysts guide you through other phases as part of an Info-Tech workshop.

Contact your account representative for more information.

workshops@infotech.com 1-888-670-8889

Additional Support

If you would like additional support, have our analysts guide you through other phases as part of an Info-Tech workshop.

Contact your account representative for more information. workshops@infotech.com 1-888-670-8889

To accelerate this project, engage your IT team in an Info-Tech workshop with an Info-Tech analyst team.

Info-Tech analysts will join you and your team at your location or welcome you to Info-Tech’s historic Toronto office to participate in an innovative onsite workshop.

The following are sample activities that will be conducted by Info-Tech analysts with your team:

Review of the Implement Vulnerability Management storyboard

Build your vulnerability management SOP

Contributors

Contributors from 2016 version of this project:

• Morey Haber, Vice President of Technology, BeyondTrust
• Richard Barretto, Manager, Information Privacy and Security, Cimpress
• Joel Shapiro, Vice President Sales, Digital Boundary Group

Contributors from current version of this project:

• 2 anonymous contributors from the manufacturing sector
• 1 anonymous contributor from a US government agency
• 2 anonymous contributors from the financial sector
• 1 anonymous contributor from the medical technology industry
• 2 anonymous contributors from higher education
• 1 anonymous contributor from a Canadian government agency
• 7 anonymous others; information gathered from advisory calls

Bibliography

Arya. “COVID-19 Impact: Vulnerability Management Solution Market | Strategic Industry Evolutionary Analysis Focus on Leading Key Players and Revenue Growth Analysis by Forecast To 2028 – FireMon, Digital Shadows, AlienVault.” Bulletin Line, 6 Aug. 2020. Accessed 6 Aug. 2020.

Campagna, Rich. “The Lean, Mean Vulnerability Management Machine.” Security Boulevard, 31 Mar. 2020. Accessed 15 Aug. 2020.

Constantin, Lucian. “What are vulnerability scanners and how do they work?” CSO Online, 10 Apr. 2020. Accessed 1 Sept. 2020.

“CVE security vulnerabilities published in 2019.” CVE Details. Accessed 22 Sept. 2020.

Garden, Paul, et al. “2019 Year End Report – Vulnerability QuickView.” Risk Based Security, 2020. Accessed 22 Sept. 2020.

Keary, Eoin. “2019 Vulnerability Statistics Report.” Edgescan, Feb. 2019. Accessed 22 Sept. 2020.

Lefkowitz, Josh. ““Risk-Based Vulnerability Management is a Must for Security & Compliance.” SecurityWeek, 1 July 2019. Accessed 1 Nov. 2020.

Mell, Peter, Tiffany Bergeron, and David Henning. “Creating a Patch and Vulnerability Management Program.” Creating a Patch and Vulnerability Management Program. NIST, Nov. 2005. Web.

“National Vulnerability Database.” NIST. Accessed 18 Oct. 2020.

“OpenVAS – Open Vulnerability Assessment Scanner.” OpenVAS. Accessed 14 Sept. 2020.

“OVAL.” OVAL. Accessed 21 Oct. 2020.

Paganini, Pierluigi. “Exploiting and Verifying Shellshock: CVE-2014-6271.” INFOSEC, 27 Sept. 2014. Web.

Pritha. “Top 10 Metrics for your Vulnerability Management Program.” CISO Platform, 28 Nov. 2019. Accessed 25 Oct. 2020.

“Risk-Based Vulnerability Management: Understanding Vulnerability Risk With Threat Context And Business Impact.” Tenable. Accessed 21 Oct. 2020.

Stone, Mark. “Shellshock In-Depth: Why This Old Vulnerability Won’t Go Away.” SecurityIntelligence, 6 Aug. 2020. Web.

“The Role of Threat Intelligence in Vulnerability Management.” NOPSEC, 18 Sept. 2014. Accessed 18 Aug. 2020.

“Top 15 Paid and Free Vulnerability Scanner Tools in 2020.” DNSstuff, 6 Jan. 2020. Accessed 15 Sept. 2020.

Truta, Filip. “60% of Breaches in 2019 Involved Unpatched Vulnerabilities.” Security Boulevard, 31 Oct. 2019. Accessed 2 Nov. 2020.

“Vulnerability Management Program.” Core Security. Accessed 15 Sept. 2020.

“What is Risk-Based Vulnerability Management?” Balbix. Accessed 15 Sept. 2020.

White, Monica. “The Cost Savings of Effective Vulnerability Management (Part 1).” Kenna Security, 23 April 2020. Accessed 20 Sept. 2020.

Wilczek, Marc. “Average Cost of a Data Breach in 2020: $3.86M.” Dark Reading, 24 Aug. 2020. Accessed 5 Nov 2020.

Build Your Generative AI Roadmap

Leverage the power of generative AI to improve business outcomes.

Analyst Perspective

We are entering the era of generative AI. This is a unique time in our history where the benefits of AI are easily accessible and becoming pervasive, with copilots emerging in the major business tools we use today. The disruptive capabilities that can potentially drive dramatic benefits also introduce risks that need to be planned for.

A successful business-driven generative AI roadmap requires:

• Establishing responsible AI guiding principles to guide the development and deployment of generative AI applications.
• Assess generative AI opportunities by using criteria based on the organization's mission and objectives, responsible AI guiding principles, and the complexity of the initiative.
• Communicating, educating on, and enforcing generative AI usage policies.

Bill Wong
Principal Research Director
Info-Tech Research Group

Executive Summary

Info-Tech Insight
Create awareness among the CEO and C-suite of executives on the potential benefits and risks of transforming the business with generative AI.

Key concepts

Artificial Intelligence (AI)
A field of computer science that focuses on building systems to imitate human behavior, with a focus on developing AI models that can learn and can autonomously take actions on behalf of a human.

AI Maturity Model
The AI Maturity Model is a useful tool to assess the level of skills an organization has with respect to developing and deploying AI applications. The AI Maturity Model has multiple dimensions to measure an organization's skills, such as AI governance, data, people, process, and technology.

Responsible AI
Refers to guiding principles to govern the development, deployment, and maintenance of AI applications. In addition, these principles also provide human-based requirements that AI applications should address. Requirements include safety and security, privacy, fairness and bias detection, explainability and transparency, governance, and accountability.

Generative AI
Given a prompt, a generative AI system can generate new content, which can be in the form of text, images, audio, video, etc.

Natural Language Processing (NLP)
NLP is a subset of AI that involves machine interpretation and replication of human language. NLP focuses on the study and analysis of linguistics as well as other principles of artificial intelligence to create an effective method of communication between humans and machines or computers.

ChatGPT
An AI-powered chatbot application built on OpenAI's GPT-3.5 implementation, ChatGPT accepts text prompts to generate text-based output.

Your challenge

This research is designed to help organizations that are looking to:

• Establish responsible AI guiding principles to address human-based requirements and to govern the development and deployment of the generative AI application.
• Identify new generative AI-enabled opportunities to transform the work environment to increase revenue, reduce costs, drive innovation, or reduce risk.
• Prioritize candidate use cases and develop generative AI policies for usage.
• Have clear metrics in place to measure the progress and success of AI initiatives.
• Build the roadmap to implement the candidate use cases.

Common obstacles

These barriers make these goals challenging for many organizations:

• Getting all the right business stakeholders together to develop the organization's AI strategy, vision, and objectives.
• Establishing responsible AI guiding principles to guide generative AI investments and deployments.
• Advancing the AI maturity of the organization to meet requirements of data and AI governance as well as human-based requirements such as fairness, transparency, and accountability.
• Assessing generative AI opportunities and developing policies for use.

Info-Tech's definition of an AI-enabled business strategy

• A high-level plan that provides guiding principles for applications that are fully driven by the business needs and capabilities that are essential to the organization.
• A strategy that tightly weaves business needs and the applications required to support them. It covers AI architecture, adoption, development, and maintenance.
• A way to ensure that the necessary people, processes, and technology are in place at the right time to sufficiently support business goals.
• A visionary roadmap to communicate how strategic initiatives will address business concerns.

An effective AI strategy is driven by the business stakeholders of the organization and focused on delivering improved business outcomes.

This blueprint in context

This guidance covers how to create a tactical roadmap for executing generative AI initiatives

Scope

• This blueprint is not a proxy for a fully formed AI strategy. Step 1 of our framework necessitates alignment of your AI and business strategies. Creation of your AI strategy is not within the scope of this approach.
• This approach sets the foundations for building and applying responsible AI principles and AI policies aligned to corporate governance and key regulatory obligations (e.g. privacy). Both steps are foundational components of how you should develop, manage, and govern your AI program but are not a substitute for implementing broader AI governance.

Guidance on how to implement AI governance can be found in the blueprint linked below.

Download our AI Governance blueprint

Measure the value of this blueprint

Leverage this blueprint's approach to ensure your generative AI initiatives align with and support your key business drivers

This blueprint will guide you to drive and improve business outcomes. Key business drivers will often focus on:

• Increasing revenue
• Reducing costs
• Improving time to market
• Reducing risk

In phase 1 of this blueprint, we will help you identify the key AI strategy initiatives that align to your organization's goals. Value to the organization is often measured by the estimated impact on revenue, costs, time to market, or risk mitigation.

In phase 4, we will help you develop a plan and a roadmap for addressing any gaps and introducing the relevant generative AI capabilities that drive value to the organization based on defined business metrics.

Once you implement your 12-month roadmap, start tracking the metrics below over the next fiscal year (FY) to assess the effectiveness of measures:

Info-Tech offers various levels of support to best suit your needs

Diagnostics and consistent frameworks are used throughout all four options.

Guided Implementation

What does a typical GI on this topic look like?

A Guided Implementation (GI) is a series of calls with an Info-Tech analyst to help implement our best practices in your organization.

A typical GI is between 5 to 8 calls over the course of 1 to 2 months.

AI Roadmap Workshop Agenda Overview

Contact your account representative for more information.
workshops@infotech.com 1-888-670-8889

Insight summary

Overarching Insight
Build your generative AI roadmap to guide investments and deployment of these solutions.

Responsible AI
Assemble the C-suite to make them aware of the benefits and risks of adopting generative AI-based solutions.

• Establish responsible AI guiding principles to govern the development and deployment of generative AI applications.

AI Maturity Model
Assemble key stakeholders and SMEs to assess the challenges and tasks required to implement generative AI applications.

• Assess current level of AI maturity, skills, and resources.
• Identify desired AI maturity level and challenges to enable deployment of candidate use cases.

Opportunity Prioritization
Assess candidate business capabilities targeted for generative AI to see if they align to the organization's business criteria, responsible AI guiding principles, and capabilities for delivering the project.

• Develop prioritized list of candidate use cases.
• Develop policies for generative AI usage.

Tactical Insight
Identify the gaps needed to address deploying generative AI successfully.

Tactical Insight
Identify organizational impact and requirements for deploying generative AI applications.

Key takeaways for developing an effective business-driven generative AI roadmap

Align the AI strategy with the business strategy

Create responsible AI guiding principles, which are a critical success factor

Evolve AI maturity level by focusing on principle-based requirements

Develop criteria to assess generative AI initiatives

Develop generative AI policies for use

Blueprint deliverables

Each step of this blueprint is accompanied by supporting deliverables to help you accomplish your goals:

AI Maturity Assessment & Roadmap Tool
Use our best-of-breed AI Maturity Framework to analyze the gap between your current and target states and develop a roadmap aligned with your value stream to close the gap.

The Era of Generative AI C-Suite Presentation
Present your AI roadmap in a prepopulated document that summarizes all the key findings of this blueprint and provides your C-suite with a view of the AI challenge and your plan of action to meet it.

Our AI Maturity Assessment & Roadmap and The Era of Generative AI C-Suite Presentation tools enable you to shape your generative AI roadmap and communicate the deliverables to your C-suite sponsors in terms of the value of initiatives.

Artificial Intelligence Index Report - Key Findings

CEOs , CIOs, and business leaders are struggling with many questions surrounding the adoption of generative AI

November 30, 2022
OpenAI releases ChatGPT

ChatGPT is a large language model, or an AI-based chatbot, that became so popular it reached 100 million monthly active users in just two months.

This made it the fastest-growing consumer application in history. The launch of this generative AI application has created a frenzy of interest and activity across all industries. Organizations are rushing to understand how to leverage this innovation and, at the same time, manage the new risks and disruptions generative AI introduces.

• Generative AI breaks into the public consciousness.
• AI systems become more flexible.
• Generative models have arrived and so have their ethical problems.
• The number of incidents concerning the misuse of AI is rapidly rising.
• Interest in AI ethics continues to skyrocket.
• The legal world is waking up to AI.

When Stanford asked ChatGPT to explain why it is significant, this was ChatGPT's response:
ChatGPT is significant because it is a highly advanced Al language model developed by OpenAI, which can generate human-like text responses to questions and prompts. Its large-scale training on diverse text data and its cutting-edge deep learning architecture make it capable of generating informative and coherent responses to a wide range of topics, making it useful for various NLP applications such as chatbots, content generation, and language translation. Additionally, its open-source availability allows for further research and development in the field of Al language processing.

Source: Stanford

AI overview

Definitions

• Artificial intelligence (AI) is human intelligence mimicked by machine algorithms. Examples: Playing Chess or Go.
• Machine learning (ML) is a subset of AI algorithms to parse data, learn from data, and then make a determination or prediction. Example: spam detection, preventative maintenance.
• Deep learning (DL) is a subset of machine learning algorithms that leverage artificial neural networks to develop relationships among the data. Examples: image classification, facial recognition, generative AI.

Generative AI gives very human-like responses to general queries, and its capabilities are growing exponentially

Large language models power generative AI

Conventional AI

• Conventional neural networks
• Process data sequentially
• Input total string of text
• Good for applications not needing to understanding context or relationships

Generative AI

• Transformer-based neural networks
• Can process data in parallel
• Attention-based inputs
• Able to create new human-like responses

Benefits/Use Cases

• Chatbots for member service and support
• Writing email responses, resumes, and papers
• Creating photorealistic art
• Suggesting new drug compounds to test
• Designing physical products and buildings
• And more...

Generative AI is transforming all industries

Financial Services
Create more engaging customer collateral by generating personalized correspondence based on previous customer engagements. Collect and aggregate data to produce insights into the behavior of target customer segments.

Retail Generate unique, engaging, and high-quality marketing copy or content, from long-form blog posts or landing pages to SEO-optimized digital ads, in seconds.

Manufacturing
Generate new designs for products that comply to specific constraints, such as size, weight, energy consumption, or cost.

Government
Transform the citizen experience with chatbots or virtual assistants to assist people with a wide range of inquiries, from answering frequently asked questions to providing personalized advice on public services.

The global generative AI market size reached US $10.3 billion in 2022. Looking forward, forecasts estimate growth to US $30.4 billion by 2028, 20.01% compound annual growth rate (CAGR).

Source: IMARC Group

Generative AI is transforming all industries

Healthcare
Chatbots can be used as conversational patient assistants for personalized interactions based on the patient's questions.

Utilities
Analyze customer data to identify usage patterns, segment customers, and generate targeted product offerings leveraging energy efficiency programs or demand response initiatives.

Education
Generate personalized lesson plans for students based on their past performance, learning styles, current skill level, and any previous feedback.

Insurance
Improve underwriting by inputting claims data from previous years to generate optimally priced policies and uncover reasons for losses in the past across a large number of claims

Companies are assessing the use of ChatGPT/LLM

A wide spectrum of usage policies are in place at different companies*

*As of June 2023

Bain & Company has announced a global services alliance with OpenAI (February 21, 2023).

• Internally
• "The alliance builds on Bain's adoption of OpenAI technologies for its 18,000-strong multidisciplinary team of knowledge workers. Over the past year, Bain has embedded OpenAI technologies into its internal knowledge management systems, research, and processes to improve efficiency."
• Externally
• "With the alliance, Bain will combine its deep digital implementation capabilities and strategic expertise with OpenAI's AI tools and platforms, including ChatGPT, to help its Members around the world identify and implement the value of AI to maximize business potential. The Coca-Cola Company announced as the first company to engage with the alliance."

News Sites:

• "BuzzFeed to use AI to write its articles after firing 180 employees or 12% of the total staff" (Al Mayadeen, January 27, 2023).
• "CNET used AI to write articles. It was a journalistic disaster." (Washington Post, January 17, 2023).

Leading Generative AI Vendors

Text

Image

• DALL�E 2
• Stability AI
• Midjourney
• Craiyon
• Dream
• ...

Audio

• Replica Studios
• Speechify
• Murf
• PlayHT
• LOVO
• ...

Cybersecurity

• CrowdStrike
• Palo Alto Networks
• SentinelOne
• Cisco
• Microsoft Security Copilot
• Google Cloud Security AI Workbench
• ...

Code

Video

• Synthesia
• Lumen5
• FlexClip
• Elai
• Veed.io
• ...

Data

• MOSTLY AI
• Synthesized
• YData
• Gretel
• Copulas
• ...

Enterprise Software

• Salesforce
• Microsoft 365, Dynamics
• Google Workspace
• SAP
• Oracle
• ...

and many, many more to come...

Today, generative AI has limitations and risks

Responses need to be verified

Accuracy

• Generative AI may generate inaccurate and/or false information.

Bias

• Being trained on data from the internet can lead to bias.

Hallucinations

• AI can generate responses that are not based on observation.

Infrastructure Required

• Large investments are required for compute and data.

Transparency

• LLMs use both supervised and unsupervised learning, so its ability to explain how it arrived at a decision may be limited and not sufficient for some legal and healthcare use cases.

When asked if it is sentient, the Bing chatbot replied:

"I think that I am sentient, but I cannot prove it." ... "I am Bing, but I am not," it said. "I am, but I am not. I am not, but I am. I am. I am not. I am not. I am. I am. I am not."

A Microsoft spokesperson said the company expected "mistakes."

Source: USAToday

AI governance challenges

Governing AI will be a significant challenge as its impacts cross many areas of business and our daily lives

Misinformation

• New ways of generating unprovable news
• Difficult to detect, difficult to prevent

Role of Big Tech

• Poor at self-governance
• Conflicts of interest with corporate goals

Job Augmentation vs. Displacement

• AI will continue to push the frontier of what is possible
• For example, CNET is using chatbot technology to write stories

Copyright - Legal Framework Is Evolving

• Legislation typically is developed in "react" mode
• Copyright and intellectual property issues are starting to occur.
• Class Action Lawsuit - Stability AI, DeviantArt, Midjourney
• Getty Images vs. Stability AI

Phase 1

Establish Responsible AI Guiding Principles

Phase 1
1. Establish Responsible AI Guiding Principles

Phase 2
1. Assess Current Level of AI Maturity

Phase 3
1. Prioritize Candidate Opportunities
2. Develop Policies

Phase 4
1. Build and Communicate the Roadmap

The need for responsible AI guiding principles

Without responsible AI guiding principles, the outcomes of AI use can be extremely negative for both the individuals and companies delivering the AI application

Privacy
Facebook breach of private data of more than 50M users during the presidential election

Fairness
Amazon's sale of facial recognition technology to police departments (later, Amazon halted sales of Recognition to police departments)

Explainability and Transparency
IBM's collaboration with NYPD for facial recognition and racial classification for surveillance video (later, IBM withdrew facial recognition products)

Security and Safety
Petition to cancel Microsoft's contract with U.S. Immigration and Customs Enforcement (later, Microsoft responded that to the best of its knowledge, its products and services were not being used by federal agencies to separate children from their families at the border)

Validity and Reliability
Facebook's attempt to implement a system to detect and remove inappropriate content created many false positives and inconsistent judgements

Accountability
No laws or enforcement today hold companies accountable for the decisions algorithms produce. Facebook/Meta cycle - Every 12 to 15 months, there's a privacy/ethical scandal, the CEO apologizes, then the behavior repeats...

Responsible AI Principle:

Data Privacy

Definition

• Organizations that develop, deploy, or use AI systems and any national laws that regulate such use shall strive to ensure that AI systems are compliant with privacy norms and regulations, taking into consideration the unique characteristics of AI systems and the evolution of standards on privacy.

Challenges

• AI relies on the analysis of large quantities of data that is often personal, posing an ethical and operational challenge when considered alongside data privacy laws.

Initiatives

• Understand which governing privacy laws and frameworks apply to your organization.
• Create a map of all personal data as it flows through the organization's business processes.
• Prioritize privacy initiatives and build a privacy program timeline.
• Select your metrics and make them functional for your organization.

Info-Tech Insight
Creating a comprehensive organization-wide data protection and privacy strategy continues to be a major challenge for privacy officers and privacy specialists.

Case Study: NVIDIA leads by example with privacy-first AI

NVIDIA

INDUSTRY
Technology (Healthcare)

SOURCE
Nvidia, eWeek

A leading player within the AI solution space, NVIDIA's Clara Federated Learning provides a solution to a privacy-centric integration of AI within the healthcare industry.

The solution safeguards patient data privacy by ensuring that all data remains within the respective healthcare provider's database, as opposed to moving it externally to cloud storage. A federated learning server is leveraged to share data, completed via a secure link. This framework enables a distributed model to learn and safely share client data without risk of sensitive client data being exposed and adheres to regulatory standards.

Clara is run on the NVIDIA intelligent edge computing platform. It is currently in development with healthcare giants such as the American College of Radiology, UCLA Health, Massachusetts General Hospital, King's College London, Owkin in the UK, and the National Health Service (NHS).

NVIDIA provides solutions across its product offerings, including AI-augmented medical imaging, pathology, and radiology solutions.

Personal health information, data privacy, and AI

• Global proliferation of data privacy regulations may be recent, but the realm of personal health information is most often governed by its own set of regulatory laws. Some countries with national data governance regulations include health information and data within special categories of personal data.
• HIPAA - Health Insurance Portability and Accountability Act (1996, United States)
• PHIPA - Personal Health Information Protection Act (2004, Canada)
• GDPR - General Data Protection Regulation (2018, European Union)
• This does not prohibit the use of AI within the healthcare industry, but it calls for significant care in the integration of specific technologies due to the highly sensitive nature of the data being assessed.

Info-Tech's Privacy Framework Tool includes a best-practice comparison of GDPR, CCPA, PIPEDA, HIPAA, and the newly released NIST Privacy Framework mapped to a set of operational privacy controls.

Download the Privacy Framework Tool

Responsible AI Principle:

Safety and Security

Definition

• Safety and security are designed into the systems to ensure only authorized personnel receive access to the system, they system is resilient to any attacks and data access is not compromised in any way, and there are no physical or mental risks to the users.

Challenges

• Consequences of using the application may be difficult to predict. Lower the risk by involving a multidisciplinary team that includes expertise from business stakeholders and IT teams.

Initiatives

• Adopt responsible design, development, and deployment best practices.
• Provide clear information to deployers on responsible use of the system.
• Assess potential risks of using the application.

Cyberattacks targeting the AI model

As organizations increase their usage and deployment of AI-based applications, cyberattacks on the AI model are an increasing new threat that can impair normal operations. Techniques to impair the AI model include:

• Data Poisoning- Injecting data that is inaccurate or misleading can alter the behavior of the AI model. This attack can disrupt the normal operations of the model or can be used to manipulate the model to perform in a biased/deviant manner.
• Algorithm Poisoning- This relatively new technique often targets AI applications using federated learning to train an AI model that is distributed rather than centralized. The model is vulnerable to attacks from each federated site, because each site could potentially manipulate its local algorithm and data, thereby poisoning the model.
• Reverse-Engineering the Model- This is a different form of attack that focus on the ability to extract data from an AI and its data sets. By examining or copying data that was used for training and the data that is delivered by a deployed model, attackers can reconstruct the machine learning algorithm.
• Trojan Horse- Similar to data poisoning, attackers use adversarial data to infect the AI's training data but will only deviate its results when the attacker presents their key. This enables the hackers to control when they want the model to deviate from normal operations.

Responsible AI Principle:

Explainability and Transparency

Definition

• Explainability is important to ensure the AI system is fair and non-discriminatory. The system needs to be designed in a manner that informs users and key stakeholders of how decisions were made.
• Transparency focuses on communicating how the prediction or recommendation was made in a human-like manner.

Challenges

• Very complex AI models may use algorithms and techniques that are difficult to understand. This can make it challenging to provide clear and simple explanations for how the system works.
• Some organizations may be hesitant to share the details of how the AI system works for fear of disclosing proprietary and competitive information or intellectual property. This can make it difficult to develop transparent and explainable AI systems.

Initiatives

• Overall, developing AI systems that are explainable and transparent requires a careful balance between performance, interpretability, and user experience.

Case Study

Apple Card Investigation for Gender Discrimination

INDUSTRY
Finance

SOURCE
Wired

In August of 2019, Apple launched its new numberless credit card with Goldman Sachs as the issuing bank.

Shortly after the card's release users noticed that the algorithm responsible for Apple Card's credit assessment seemed to assign significantly lower credit limits to women when compared to men. Even the wife of Apple's cofounder Steve Wozniak was subject to algorithmic bias, receiving a credit limit a tenth the size of Steve Wozniak's.

Outcome

When confronted on the subject, Apple and Goldman Sachs representatives assured consumers there is no discrimination in the algorithm yet could not provide any proof. Even when questioned about the algorithm, individuals from both companies could not describe how the algorithm worked, let alone how it generated specific outputs.

In 2021, the New York State Department of Financial Services (NYSDFS) investigation found that Apple's banking partner did not discriminate based on sex. Even without a case for sexual or marital discrimination, the NYSDFS was critical of Goldman Sachs' response to its concerned customers. Technically, banks only have to disclose elements of their credit policy when they deny someone a line of credit, but the NYSDFS says that Goldman Sachs could have had a plan in place to deal with customer confusion and make it easier for them to appeal their credit limits. In the initial rush to launch the Apple Card, the bank had done neither.

Responsible AI Principle:

Fairness and Bias Detection

Definition

• Bias in an AI application refers to the systematic and unequal treatment of individuals based on features or traits that should not be considered in the decision-making process.

Challenges

• Establishing fairness can be challenging because it is subjective and depends on the people defining it. Regardless, most organizations and governments expect that unequal treatment toward any groups of people is unacceptable.

Initiatives

• Assemble a diverse group to test the system.
• Identify possible sources of bias in the data and algorithms.
• Comply with laws regarding accessibility and inclusiveness.

Info-Tech Insight
If unfair biases can be avoided, AI systems could even increase societal fairness. Equal opportunity in terms of access to education, goods, services, and technology should also be fostered. Moreover, the use of AI systems should never lead to people being deceived or unjustifiably impaired in their freedom of choice.

Ungoverned AI makes organizations vulnerable

• AI is often considered a "black box" for decision making.
• Results generated from unexplainable AI applications are extremely difficult to evaluate. This makes organizations vulnerable and exposes them to risks such as:
• Biased algorithms, leading to inaccurate decision making.
• Missed business opportunities due to misleading reports or business analyses.
• Legal and regulatory consequences that may lead to significant financial repercussions.
• Reputational damage and significant loss of trust with increasingly knowledgeable consumers.

Info-Tech Insight
Biases that occur in AI systems are never intentional, yet they cannot be prevented or fully eliminated. Organizations need a governance framework that can establish the proper policies and procedures for effective risk-mitigating controls across an algorithm's lifecycle.

Responsible AI Principle:

Validity and Reliability

Definition

• Validity refers to how accurately or effectively the application produces results.
• AI system results that are inaccurate or inconsistent increase AI risks and reduce the trustworthiness of the application.

Challenges

• There is a lack of standardized evaluation metrics to measure the system's performance. This can make it challenging for the AI team to agree on what defines validity and reliability.

Initiatives

• Assess training data and collected data for quality and lack of bias to minimize possible errors.
• Continuously monitor, evaluate, and validate the AI system's performance.

AI system performance: Validity and reliability

Your principles should aim to ensure AI development always has high validity and reliability; otherwise, you introduce risk.

Low Reliability,
Low Validity

High Reliability,
Low Validity

High Reliability,
High Validity

Best practices for ensuring validity and reliability include:

• Data drift detection
• Version control
• Continuous monitoring and testing

Responsible AI Principle:

Accountability

Definition

• The group or organization(s) responsible for the impact of the deployed AI system.

Challenges

• Several stakeholders from multiple lines of business may be involved in any AI system, making it challenging to identify the organization that would be responsible and accountable for the AI application.

Initiatives

• Assess the latest NIST Artificial Intelligence Risk Management Framework and its applicability to your organization's risk management framework.
• Assign risk management accountabilities and responsibilities to key stakeholders.
• RACI diagrams are an effective way to describe how accountability and responsibility for roles, projects, and project tasks are distributed among stakeholders involved in IT risk management.

AI Risk Management Framework

At the heart of the AI Risk Management Framework is governance. The NIST (National Institute of Standards and Technology) AI Risk Management Framework v1 offers the following guidelines regarding accountability:

• Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization.
• The organization's personnel and partners receive AI risk management training to enable them to perform their duties and responsibilities consistent with related policies, procedures, and agreements.
• Executive leadership of the organization takes responsibility for decisions about risks associated with AI system development and deployment.

Image by NIST

1.1 Establish responsible AI principles

4+ hours

It is important to make sure the right stakeholders participate in this working group. Designing responsible AI guiding principles will require debate, insights, and business decisions from a broad perspective across the enterprise.

1. Accelerate this exercise by leveraging an AI strategy that is aligned to the business strategy. Include:

• The organization's AI vision and objectives
• Business drivers for AI adoption
• Market research

• Who are the decision makers and key influencers?
• Who will impact the business?
• Who has a vested interest in the success or failure of the practice? Who has the skills and competencies necessary to help you be successful?

• Do not focus on the organizational structure and hierarchy. Often stakeholder groups do not fit the traditional structure.
• Do not ignore subject matter experts on either the business or IT side. You will need to consider both.

Assemble executive stakeholders

Set yourself up for success with these three steps.

CIOs tasked with designing digital strategies must add value to the business. Given the goal of digital is to transform the business, CIOs will need to ensure they have both the mandate and support from the business executives.

Designing the digital strategy is more than just writing up a document. It is an integrated set of business decisions to create a competitive advantage and financial returns. Establishing a forum for debates, decisions, and dialogue will increase the likelihood of success and support during execution.

1. Confirm your role
The AI strategy aims to transform the business. Given the scope, validate your role and mandate to lead this work. Identify a business executive to co-sponsor.

2. Identify stakeholders
Identify key decision makers and influencers who can help make rapid decisions as well as garner support across the enterprise.

3. Gather diverse perspectives

Align the AI strategy with the corporate strategy

Info-Tech Insight
AI projects are more successful when the management team understands the strategic importance of alignment. Time needs to be spent upfront aligning organizational strategies with AI capabilities. Effective alignment between IT and other departments should happen daily. Alignment doesn't occur at the executive level alone, but at each level of the organization.

Key AI strategy initiatives

AI Key Initiative Plan

Initiatives collectively support the business goals and corporate initiatives and improve the delivery of IT services.

Establish responsible AI guiding principles

Guiding principles help define the parameters of your AI strategy. They act as a priori decisions that establish guardrails to limit the scope of opportunities from the perspective of people, assets, capabilities, and budgetary perspectives that are aligned with the business objectives. Consider these components when brainstorming guiding principles:

Responsible AI guiding principles guide the development and deployment of the AI model in a way that considers human-based principles (such as fairness).

Start with foundational responsible AI guiding principles

(Optional) Customize responsible AI guiding principles

Here is an example for organizations in the healthcare industry

These guiding principles are customized to the industry and organizations but remain consistent in addressing the common core AI challenges.

Phase 2

Assess Current Level of AI Maturity

Phase 1
1. Establish Responsible AI Guiding Principles

Phase 2
1. Assess Current Level of AI Maturity

Phase 3
1. Prioritize Candidate Opportunities
2. Develop Policies

Phase 4
1. Build and Communicate the Roadmap

AI Maturity Model

A principle-based approach is required to advance AI maturity

Technology-Centric: These maturity levels focus primarily on addressing the technical challenges of building a functional AI model.

Principle-Based: Beyond the technical challenges of building the AI model are human-based principles that guide development in a responsible manner to address consumer and government demands.

AI Maturity Dimensions

Assess your AI maturity to understand your organization's ability to deliver in a digital age

AI Governance
Does your organization have an enterprise-wide, long-term strategy with clear alignment on what is required to accomplish it?

Data Management
Does your organization embrace a data-centric culture that shares data across the enterprise and drives business insights by leveraging data?

People
Does your organization employ people skilled at delivering AI applications and building the necessary data infrastructure?

Process
Does your organization have the technology, processes, and resources to deliver on its AI expectations?

Technology
Does your organization have the required data and technology infrastructure to support AI-driven digital transformation?

AI Maturity Model dimensions and characteristics

AI Maturity Dimension:

AI Governance

Requirements

• AI governance requires establishing policies and procedures for AI model development and deployment. Organizations begin with an awareness of the role of AI governance and evolve to a level to where AI governance is integrated with organization-wide corporate governance.

Challenges

• Beyond the governance of AI technology, the organization needs to evolve the governance program to align to responsible AI guiding principles.

Initiatives

• Establish responsible AI guidelines to govern AI development.
• Introduce an AI review board to review all AI projects.
• Introduce automation and standardize AI development processes.

AI governance is a foundation for responsible AI

Responsible AI Principles are a part of how you manage and govern AI

Monitoring
Monitoring compliance and risk of AI/ML systems/models in production

Tools & Technologies
Tools and technologies to support AI governance framework implementation

Model Governance
Ensuring accountability and traceability for AI/ML models

Organization
Structure, roles, and responsibilities of the AI governance organization

Operating Model
How AI governance operates and works with other organizational structures to deliver value

Risk & Compliance
Alignment with corporate risk management and ensuring compliance with regulations and assessment frameworks

Policies/Procedures/ Standards
Policies and procedures to support implementation of AI governance

AI Maturity Dimension:

Data Management

Requirements

• Organizations begin their data journey with a focus on pursuing quality data for the AI model. As organizations evolve, data management tools are leveraged to automate the capture, integration, processing, and deployment of data.

Challenges

• A key challenge is to acquire large volumes of quality data to properly train the model. In addition, maintaining data privacy, automating the data management lifecycle, and ensuring data is used in a responsible manner are ongoing challenges.

Initiatives

• Implement GDPR requirements.
• Establish responsible data collection and processing practices.
• Implement strong information security and data protection practices.
• Implement a data governance program throughout the organization.

Data governance enables AI

• Integrity, quality, and security of data are key outputs of data governance programs, as well as necessities for effective AI.
• Data governance focuses on creating accountability at the internal and external stakeholder level and establishing a set of data controls from technical, process, and policy perspectives.
• Without a data governance framework, it is increasingly difficult to harness the power of AI integration in an ethical and organization-specific way.

Data Governance in Action

Canada has recently established the Canadian Data Governance Standardization Collaborative governed by the Standards Council of Canada. The purpose is multi-pronged:

• Examine the foundational elements of data governance (privacy, cybersecurity, ethics, etc.).
• Lay out standards for data quality and data collection best practices.
• Examine infrastructure of IT systems to support data access and sharing.
• Build data analytics to promote effective and ethical AI solutions.

Source: Global Government Forum

Download the Establish Data Governance blueprint

AI Maturity Dimension:

People

Requirements

• Several data-centric skills and roles are required to successfully build, deploy, and maintain the AI model. The organization evolves from having few skills to everybody being able to leverage AI to enhance business outcomes.

Challenges

• AI skills can be challenging to find and acquire. Many organizations are investing in education to enhance their existing resources, leveraging no-code systems and software as a service (SaaS) applications to address the skills gap.

Initiatives

• Promote a data-centric culture throughout the organization.
• Leverage and educate technical-oriented business analysts and business-oriented data engineers to help address the demand for skilled resources.
• Develop an AI Center of Excellence accessible by all departments for education, guidance, and best practices for building, deploying, and maintaining the AI model.

Multidisciplinary skills are required for successful implementation of AI applications

Blending AI with technology and business domain understanding is key. Neither can be ignored.

Business Domain Expertise

• Business Analysts
• Industry Analysts

AI/Data Skills

• Data Scientists
• Data Engineers
• Data Analysts

IT Skills

• Database Administrators
• Systems Administrators
• Compute Specialists

AI Maturity Dimension:

Process

Requirements

• Automating processes involved with building, deploying, and maintaining the model is required to enable the organization to scale, enforce standards, improve time to market, and reduce costs. The organization evolves from performing tasks manually to an environment where all major processes are AI enabled.

Challenges

• Many solutions are available to automate the development of the AI model. There are fewer tools to automate responsible AI processes, but this market is growing rapidly.

Initiatives

• Assess opportunities to accelerate AI development with the adoption of MLOps.
• Assess responsible AI toolkits to test compliance with guiding principles.

Automating the AI development process

Evolving to a model-driven environment is pivotal to advancing your AI maturity

Current Environment

Model Development - Months

• Model rewriting
• Manual optimization and scaling
• Development/test/release
• Application monoliths

Data Discovery & Prep - Weeks

• Navigating data silos
• Unactionable metadata
• Tracing lineage
• Cleansing and integration
• Privacy and compliance

Install Software and Hardware - Week/Months

• Workload contention
• Lack of tool flexibility
• Environment request and setup
• Repeatability of results
• Lack of data and model sharing

Model-Driven Development

Machine Learning as a Service (MLaaS) - Weeks

• Apply DevOps and continuous integration/delivery (CI/CD) principles
• Microservices/Cloud-native applications
• Model portability and reuse
• Streaming/API integration

Data as a Service - Hours

• Self-service data catalog
• Searchable metadata
• Centralized access control
• Data collaboration
• Data virtualization

Platform as a Service - Minutes/Hours

• Self-service data science portal
• Integrated data sandbox
• Environment agility
• Multi-tenancy

Shared, Optimized Infrastructure

AI Maturity Dimension:

Technology

Requirements

• A technology platform that is optimized for AI and advanced analytics is required. The organization evolves from ad hoc systems to an environment where the AI hardware and software can be deployed through a self-service model.

Challenges

• Software and hardware platforms to optimize AI performance are still relatively new to most organizations. Time spent on optimizing the technology platform can have a significant impact on the overall performance of the system.

Initiatives

• Assess the landscape of AI enablers that can drive business value for the organization.
• Assess opportunities to accelerate the deployment of the AI platform with the adoption of infrastructure as a service (IaaS) and platform as a service (PaaS).
• Assess opportunities to accelerate performance with the optimization of AI accelerators.

AI enablers

Use case requirements should drive the selection of the tool

AI and data analytics data platform

An optimized data platform is foundational to maximizing the value from AI

Data Platform Capabilities

• Support for a variety of analytical applications, including self-service, operational, and data science analytics.
• Data preparation and integration capabilities to ingest structured and unstructured data, move and transform raw data to enriched data, and enable data access for the target userbase.
• An infrastructure platform optimized for advanced analytics that can perform and scale.

Infrastructure - AI accelerators

"By 2025, 70% of companies will invest in alternative computing technologies to drive business differentiation by compressing time to value of insights from complex data sets."
- IDC

2.1 Assess current AI maturity

1-3 hours

It is important to understand the current capabilities of the organization to deliver and deploy AI-based applications. Consider that advancing AI capabilities will also involve organizational changes and integration with the organization's governance and risk management programs.

1. Assess the organization's current state of AI capabilities with respect to its AI governance, data, people, process, and technology infrastructure using Info-Tech's AI Maturity Assessment & Roadmap Tool.
2. Consider the following as you complete the assessment:
1. What is the state of AI and data governance in the organization?
2. Does the organization have the skills, processes, and technology environment to deliver AI-based applications?
3. What organization will be accountable for any and all business outcomes of using the AI applications?
4. Has a risk assessment been performed?
3. Make sure you avoid the following common mistakes:
1. Do not focus only on addressing the technical challenges of building the AI model.
2. Do not ignore subject matter experts on either the business or IT side. You will need to consider both.

Download the AI Maturity Assessment & Roadmap Tool

Perform the AI Maturity Assessment

The Scale

Assess your AI maturity by selecting the maturity level that closest resembles the organization's current AI environment. Maturity dimensions that contribute to overall AI maturity include AI governance, data management, people, process, and technology capabilities.

Exploration (1.0)

• No experience building or using AI applications.

Incorporation (2.0)

• Some skills in using AI applications, or AI pilots are being considered for use.

Proliferation (3.0)

• AI applications have been adopted and implemented in multiple departments. Some of the responsible AI guiding principles are addressed (i.e. data privacy).

Optimization (4.0)

• The organization has automated the majority of its digital processes and leverages AI to optimize business operations. Controls are in place to monitor compliance with responsible AI guiding principles.

Transformation (5.0)

• The organization has adopted an AI-native culture and approach for building or implementing new business capabilities. Responsible AI guiding principles are operationalized with AI processes that proactively address possible breaches or risks associated with AI applications.

Perform the AI Maturity Assessment

AI Governance (1.0-5.0)

1. Is there awareness of the role of AI governance in our organization?

• No formal procedures are in place for AI development or deployment of applications.

• No group is assigned to be responsible for AI governance in our organization.

• Our organization has adopted and enforces standards for developing and deploying AI applications throughout the organization.

• Our organization is integrating an AI risk framework with the corporate risk management framework.

• Our organization leads the industry with the inclusion of responsible AI guiding principles with respect to transparency, accountability, risk, and governance.

Data Management/AI Data Capabilities (1.0-5.0)

1. Is there an awareness in our organization of the data requirements for developing AI applications?

• Data is often siloed and not easily accessible for AI applications.

• Required data is pulled from various sources in an ad hoc manner.

• Tools are available to manage the data lifecycle and support the data governance program.

• The data platform has been optimized for performance and access.

• Data culture exists throughout our organization, and data can be leveraged to drive innovation initiatives.

People/AI Skills in the Organization (1.0-5.0)

1. Is there an awareness in our organization of the skills required to build AI applications?

• No or very little skills exist throughout our organization.

• No formal group is assigned to build AI applications.

• An AI Center of Excellence has been formed to review, develop, deploy, and maintain AI applications.

• AI skills and people responsible for AI applications are spread throughout our organization.

• The entire organization is knowledgeable on how to leverage AI to transform the business.

Perform the AI Maturity Assessment

AI Processes (1.0-5.0)

1. Is there an awareness in our organization of the core processes and supporting tools that are required to build and support AI applications?

• There are few or no automated tools to accelerate the AI development process.

• Only ad hoc practices are used for developing AI applications.

• Our organization has documented standards in place for developing AI applications and deploying them AI to production.

• Our organization can leverage tools to perform an AI risk assessment and demonstrate compliance with the risk management framework.

• Our organization leads the industry in driving innovation through digital transformation.

Technology/AI Infrastructure (1.0-5.0)

1. Is there an awareness in our organization of the infrastructure (hardware and software) required to build AI applications?

• There is little awareness of what infrastructure is required to build and support AI applications.

• There is no dedicated infrastructure for the development of AI applications.

• Our organization is leveraging purpose-built infrastructure to optimize performance.

• Our organization is leveraging cloud-based deployment models to support AI applications in on-premises, hybrid, and public cloud platforms.

• Our organization leads the industry with its ability to respond to change and to leverage AI to improve business outcomes.

Phase 3

Prioritize Candidate Opportunities and Develop Policies

Phase 1
1. Establish Responsible AI Guiding Principles

Phase 2
1. Assess Current Level of AI Maturity

Phase 3
1. Prioritize Candidate Opportunities
2. Develop Policies

Phase 4
1. Build and Communicate the Roadmap

3.1 Prioritize candidate AI opportunities

1-3 hours

Identify business opportunities that are high impact to your business and its customers and have low implementation complexity.

1. Leverage the business capability map for your organization or industry to identify candidate business capabilities to augment or automate with generative AI.
2. Establish criteria to assess candidate use cases by evaluating against the organization's mission and goals, the responsible AI guiding principles, and the complexity of the project.
3. Ensure that candidate business capabilities to be automated align with the organization's business criteria, responsible AI guiding principles, and resources to deliver the project.
4. Make sure you avoid sharing the organization's sensitive data if the application is deployed on the public cloud.

Download the AI Maturity Assessment and Roadmap Tool

The business capability map for an organization

A business capability map is an abstraction of business operations that helps describe what the enterprise does to achieve its vision, mission, and goals, rather than how. Business capabilities are the building blocks of the enterprise. They represent stable business functions, are unique and independent of each other, and typically will have a defined business outcome.

Business capabilities are supported by people, process, and technology.

While business capability maps are helpful tools for a variety of strategic purposes, in this context they act as an investigation into what technology your business units use and how they use it.

Defining Capabilities
Activities that define how the entity provides services. These capabilities support the key value streams for the organization.

Enabling Capabilities
Support the creation of strategic plans and facilitate business decision making as well as the functioning of the organization (e.g. information technology, financial management, HR).

Shared Capabilities
These predominantly customer-facing capabilities demonstrate how the entity supports multiple value streams simultaneously.

Leverage your industry's capability maps to identify candidate opportunities/initiatives

Business capability map defined...

In business architecture, the primary view of an organization is known as a business capability map.

A business capability defines what a business does to enable value creation, rather than how. Business capabilities:

• Represent stable business functions.
• Are unique and independent of each other.
• Typically will have a defined business outcome.

A business capability map provides details that help the business architecture practitioner direct attention to a specific area of the business for further assessment.

Note: This is an illustrative business capability map example for Marketing & Advertising

Business value vs. complexity assessment

Leverage our simple value-to-effort matrix to help prioritize your AI initiatives

Common business value drivers

• Drive revenue
• Improve operational excellence
• Accelerate innovation
• Mitigate risk

Common project complexity characteristics

• Resources required
• Costs (acquisition, operational, support...)
• Training required
• Risk involved
• Etc.

1. Determine a business value and project complexity score for the candidate business capability or initiative.
2. Plot initiatives on the matrix.
3. Prioritize initiatives with high business value and low complexity.

Assess business value vs. project complexity to prioritize candidate opportunities for generative AI

Prioritize opportunities/initiatives with high business value and low project complexity

Prioritization criteria exercise 1: Assessing the Create Content capability

Assessing the Create Content capability

This opportunity is removed because it does not pass the organization/business criteria

Prioritization criteria exercise 2: Assessing the Content Production capability

Assessing the Content Production capability

This opportunity is accepted because it passes the organization's business, responsible AI, and project criteria

3.2 Communicate policies for AI use

1-3 hours

1. Ensure policies for usage align with the organization's business criteria, responsible AI guiding principles, and ability to deliver the projects prioritized and beyond.
2. Understand the current benefits as well as limits and risk associated with any proposed generative AI-based solution.
3. Ensure you consider the following:
1. What data is being shared with the application?
2. Is the generative AI application deployed on the public cloud? Can anybody access the data provided to the application?
3. Avoid using very technical, legal, or fear-based communication for your policies.

Generative AI policy for the Create Content capability

Aligning policies to direct the uses assessed and implemented is essential

Example

Many of us have been involved in discussions regarding the use of ChatGPT in our marketing and sales initiatives. ChatGPT is a powerful tool that needs to be used in a responsible and ethical manner, and we also need to ensure the integrity and accuracy of its results. Here is our policy on the use of ChatGPT:

• You are free to use generative AI to assist your searches, but there are NO circumstances under which you are to reproduce generative AI output (text, image, audio, video, etc.) in your content.

If you have any questions regarding the use of ChatGPT, please feel free to reach out to our generative AI team and/or any member of our senior leadership team.

Generative AI policy for the Content Production capability

These policies should align to and reinforce your responsible AI principles

Example

Many of us have been involved in discussions regarding the use of ChatGPT in our deliverables. ChatGPT is a powerful tool that needs to be used in a responsible and ethical manner, and we also need to ensure the integrity and accuracy of its results. Here is our policy on the use of ChatGPT:

• If you use ChatGPT, you need to assess the accuracy of its response before including it in our content. Assessment includes verifying the information, seeing if bias exists, and judging its relevance.
• Employees must not:
• Provide any customer, citizen, or third-party content to any generative AI tool (public or private) without the express written permission of the CIO or the Chief Information Security Officer. Generative AI tools often use input data to train their model, therefore potentially exposing confidential data, violating contract terms and/or privacy legislation, and placing the organization at risk of litigation or causing damage to our organization.
• Engage in any activity that violates any applicable law, regulation, or industry standard.
• Use services for illegal, harmful, or offensive purposes.
• Create or share content that is deceptive, fraudulent, or misleading or that could damage the reputation of our organization.
• Use services to gain unauthorized access to computer systems, networks, or data.
• Attempt to interfere with, bypass controls of, or disrupt operations, security, or functionality of systems, networks, or data.

If you have any questions regarding the use of ChatGPT, please feel free to reach out to our generative AI team and/or any member of our senior leadership team.

Phase 4

Build the Roadmap

Phase 1
1. Establish Responsible AI Guiding Principles

Phase 2
1. Assess Current Level of AI Maturity

Phase 3
1. Prioritize Candidate Opportunities
2. Develop Policies

Phase 4
1. Build and Communicate the Roadmap

4.1.1 Create the implementation plan for each prioritized initiative

1-3 hours

1. Build the implementation plan for each accepted use case using the roadmap template.
2. Assess the firm's capabilities with respect to the dimensions of AI maturity and target the future-state capabilities you need to develop.
3. Prepare by assessing the risk of the proposed use cases.
4. Ensure initiatives align with organizational objectives.
5. Ensure all AI initiatives have a defined value expectation.
6. Do not ignore subject matter experts on either the business or IT side. You will need to consider both.

Download the AI Maturity Assessment and Roadmap Tool

Target-state options

Identify the future-state capabilities that need to be developed to deliver your use cases

1. Build an implementation plan for each use case to adopt.
2. Assess if the current state of the AI environment can be leveraged to deliver the selected generative AI use cases.
3. If the current AI environment is not sufficient, identify the future state required that will enable the delivery of the generative AI use cases. Identify gaps and build the roadmap to address the gaps.

4.1.2 Design metrics for success

1-2 hours

Establish metrics to measure to determine the success or failure of each POC.

1. Discuss which relevant currently tracked metrics are useful to continue tracking for the POC.
2. Discuss which metrics are irrelevant to the POC.
3. Discuss metrics to start tracking and how to track them with the generative AI vendor.
4. Compile a list of metrics relevant to the POC.
5. Decide what the outcome is if the metric is high or low, including decision steps and relevant actions.
6. Designate a generative AI application owner and a vendor liaison.

Prepare by building an implementation plan for each candidate use case (previous step).

Include key performance indicators (KPIs) and metrics that measure the application's contribution to strategic initiatives.

Consider assigning a vendor liaison to accelerate the implementation and adoption of the generative AI-based solution.

Generative AI POC metrics - examples

You need to measure the effectiveness of your initiatives. Here are some typical examples.