Must Learn AI Security Epilogue: Securing AI is a Three-Pronged Approach

Summary

Oct 25, 2023

This post is part of an ongoing series to educate about new and known security vulnerabilities against AI.

The full series index (including code, queries, and detections) is located here:

The book version (pdf) of this series is located here: https://github.com/rod-trent/OpenAISecurity/tree/main/Must_Learn/Book_Version

The book will be updated when each new part in this series is released.

Artificial intelligence (AI) is transforming the world in unprecedented ways, from enhancing productivity and efficiency to enabling new forms of creativity and innovation. However, as AI becomes more powerful and ubiquitous, it also poses significant challenges and risks for security, privacy, and ethics. How can we ensure that AI systems are trustworthy, reliable, and aligned with human values and interests?

As we wrap up, we will explore how securing AI boils down to three main aspects: secure code, secure data, and secure access. We will also discuss some of the best practices and tools that can help developers and users achieve these goals.

Secure Code

Secure code refers to the quality and integrity of the software that implements AI algorithms and models. It is essential to ensure that the code is free of errors, vulnerabilities, and malicious components that could compromise the functionality, performance, or safety of the AI system.

Some of the common threats to secure code include:

Code injection: This occurs when an attacker inserts malicious code into an AI system, such as a web application or a machine learning model, that can execute arbitrary commands or manipulate data. For example, an attacker could inject code into a chatbot that would send sensitive information to a remote server or display inappropriate content to users.
Backdoors: These are hidden features or functions in an AI system that can be activated by an attacker to bypass security measures or gain unauthorized access. For example, an attacker could embed a backdoor into a facial recognition system that would grant access to anyone wearing a specific hat or glasses.
Trojans: These are malicious components that are disguised as legitimate ones in an AI system. They can perform unwanted actions or leak information without the user’s knowledge or consent. For example, an attacker could replace a benign image classifier with a trojanized one that would misclassify certain images or send them to a remote server.

To prevent these threats, developers should follow secure coding practices and standards, such as:

Code review: This is the process of examining the source code of an AI system to identify and fix errors, vulnerabilities, and malicious components. Code review can be done manually by human experts or automatically by tools such as static analyzers, dynamic analyzers, or fuzzers.
Testing: This is the process of verifying the functionality, performance, and security of an AI system by running it under various scenarios and inputs. Testing can be done at different levels, such as unit testing, integration testing, system testing, or penetration testing.
Auditing: This is the process of evaluating the quality and integrity of an AI system by checking its compliance with security standards and regulations. Auditing can be done internally by the developers or externally by independent third parties.

Secure Data

Secure data refers to the protection and privacy of the data that is used to train, test, or run AI systems. It is crucial to ensure that the data is accurate, consistent, and representative of the intended domain and task. It is also important to prevent unauthorized access, modification, or leakage of the data.

Some of the common threats to secure data include:

Data poisoning: This occurs when an attacker alters or injects malicious data into an AI system’s training or testing dataset, with the aim of degrading its performance or causing it to behave in unexpected or harmful ways. For example, an attacker could poison a spam filter’s training dataset with legitimate emails that would cause it to misclassify them as spam.
Data theft: This occurs when an attacker steals sensitive or valuable data from an AI system’s storage or transmission channels. For example, an attacker could steal personal information from a facial recognition system’s database or intercept images from a self-driving car’s camera.
Data inference: This occurs when an attacker infers private or confidential information from an AI system’s output or behavior. For example, an attacker could infer a user’s preferences, habits, or identity from a recommender system’s recommendations or a voice assistant’s responses.

To prevent these threats, developers and users should follow data security practices and techniques, such as:

Data encryption: This is the process of transforming data into an unreadable form that can only be decrypted by authorized parties who have the key. Data encryption can be applied to data at rest (stored on disks or databases) or data in transit (transmitted over networks).
Data anonymization: This is the process of removing or modifying personally identifiable information (PII) from data to prevent re-identification of individuals. Data anonymization can be done by techniques such as masking (replacing PII with random values), generalization (reducing the granularity of PII), perturbation (adding noise to PII), or aggregation (combining PII into groups).
Data minimization: This is the principle of collecting and processing only the minimum amount of data that is necessary for the intended purpose. Data minimization can help reduce the risk of data exposure, misuse, or abuse.

Secure Access

Secure access refers to the control and management of the access rights and privileges of the users and entities that interact with AI systems. It is vital to ensure that only authorized and authenticated parties can access or modify the AI system’s code, data, or functionality.

Some of the common threats to secure access include:

Unauthorized access: This occurs when an attacker gains access to an AI system’s code, data, or functionality without proper authorization or authentication. For example, an attacker could access a medical diagnosis system’s code and alter its logic or parameters.
Privilege escalation: This occurs when an attacker exploits a vulnerability or flaw in an AI system’s access control mechanism to gain higher privileges or permissions than intended. For example, an attacker could exploit a buffer overflow in a speech recognition system’s code to execute arbitrary commands or access restricted resources.
Denial of service: This occurs when an attacker overwhelms or disrupts an AI system’s availability or functionality by sending excessive or malicious requests or inputs. For example, an attacker could flood a natural language processing system’s server with gibberish texts that would consume its resources and prevent legitimate users from accessing it.

To prevent these threats, developers and users should follow access control practices and methods, such as:

Authentication: This is the process of verifying the identity of a user or entity that requests access to an AI system. Authentication can be done by factors such as passwords, tokens, biometrics, or certificates.
Authorization: This is the process of granting or denying access rights and privileges to a user or entity based on their identity, role, or context. Authorization can be done by mechanisms such as access control lists (ACLs), role-based access control (RBAC), or attribute-based access control (ABAC).
Monitoring: This is the process of tracking and logging the activities and events that occur in an AI system. Monitoring can help detect and respond to anomalous or malicious behaviors, such as unauthorized access, privilege escalation, or denial of service.

Shared Responsibility

Artificial intelligence (AI) is a powerful technology that can enhance the productivity and creativity of individuals and organizations. However, AI also poses unique challenges and risks that require careful consideration and management. To use AI safely and responsibly, it is important to understand the shared responsibility model between you and the AI platform or application provider.

The shared responsibility model defines the tasks and security responsibilities that are handled by the AI provider and the ones that are handled by you. The model depends on the type of AI deployment, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). Generally, the more control you have over the AI capabilities, the more responsibility you have for securing them.

Let’s take the Microsoft model as an example.

The following diagram illustrates the areas of responsibility between you and Microsoft according to the type of AI deployment:

See: Artificial intelligence (AI) shared responsibility model

An AI enabled application consists of three layers of functionality that group together tasks, which you or an AI provider perform. These layers are:

The AI platform layer provides the AI capabilities to the applications. This layer requires building and safeguarding the infrastructure that runs the AI model, training data, and specific configurations that change the behavior of the model. This layer also provides access to functionality via APIs, which pass text known as a Metaprompt to the AI model for processing, then return the generated outcome, known as a Prompt-Response.
The AI application layer accesses the AI capabilities and provides the service or interface that the user consumes. This layer can vary from simple to complex, depending on the application. This layer may include components such as user interface, data connectors, plugins, semantic index, persistence layer, and other AI applications.
The AI usage layer describes how the AI capabilities are ultimately used and consumed. This layer involves user behavior and accountability, as well as security assurances for identity and access controls, device protections and monitoring, data protection and governance, administrative controls, and other controls.

Each layer has its own security considerations and challenges that need to be addressed by either you or the AI provider. For example, at the AI platform layer, there is a need to protect the AI model from malicious inputs and outputs. At the AI application layer, there is a need to protect the application from malicious activities and interactions. At the AI usage layer, there is a need to educate users on the difference of standard IT applications to AI enabled applications and on the potential risks of AI based attacks.

Microsoft offers various options for implementing AI capabilities for your organization. Depending on which option you choose, you take responsibility for different parts of the necessary operations and policies needed to use AI safely. Microsoft recommends starting with SaaS based approaches like Copilot solutions for your initial adoption of AI and for all subsequent AI workloads. This minimizes the level of responsibility and expertise your organization has to provide to design, operate, and secure these highly complex capabilities.

Microsoft ensures that every Copilot solution is engineered following strong principles for AI governance.

Conclusion

Securing AI is a complex and challenging task that requires a holistic and multi-layered approach. By following the three aspects of secure code, secure data, and secure access, developers and users can build and use AI systems that are more trustworthy, reliable, and safe.

AI services are becoming more prevalent and influential in various domains and applications, such as healthcare, education, entertainment, and security. However, as AI services become more complex and powerful, they also pose significant challenges and risks for security, privacy, and ethics. Therefore, it is essential to audit and monitor the code, data, and access of AI services to ensure their quality, integrity, and trustworthiness.

This Must Learn AI Security series is intended to help drive awareness to these precepts.

Auditing and monitoring code involves checking the compliance of the software that implements AI algorithms and models with security standards and regulations. It also involves identifying and fixing errors, vulnerabilities, and malicious components that could compromise the functionality, performance, or safety of the AI service. Auditing and monitoring data involves protecting and preserving the accuracy, consistency, and representativeness of the data that is used to train, test, or run AI services. It also involves preventing unauthorized access, modification, or leakage of the data. Auditing and monitoring access involves controlling and managing the access rights and privileges of the users and entities that interact with AI services. It also involves detecting and responding to anomalous or malicious behaviors, such as unauthorized access, privilege escalation, or denial of service.

By auditing and monitoring code, data, and access of AI services, developers and users can mitigate the threats and risks that could harm the AI service itself or its stakeholders. Moreover, they can enhance the reliability, transparency, and accountability of AI services, which are essential for building trust and confidence among users and society. Auditing and monitoring code, data, and access of AI services is not only a technical challenge but also a social responsibility that requires collaboration and coordination among various actors, such as developers, regulators, auditors, users, and researchers. By working together to secure AI services, we can harness their potential for good while avoiding their pitfalls for evil.

[Want to discuss this further? Hit me up on Twitter or LinkedIn]
[Subscribe to the RSS feed for this blog]
[Subscribe to the Weekly Microsoft Sentinel Newsletter]
[Subscribe to the Weekly Microsoft Defender Newsletter]
[Subscribe to the Weekly Azure OpenAI Newsletter]
[Learn KQL with the Must Learn KQL series and book]
[Learn AI Security with the Must Learn AI Security series and book]