Protecting AI Systems: Detecting Common Attacks with KQL in Microsoft Sentinel

Sentinelizing AI

Artificial Intelligence (AI) systems are increasingly integral to modern organizations, powering everything from predictive analytics to automated decision-making. However, their growing adoption makes them prime targets for cyberattacks. Common AI attacks, such as data poisoning, model evasion, and model inversion, can compromise model integrity, leak sensitive data, or degrade performance. Using Microsoft Sentinel, security teams can leverage Kusto Query Language (KQL) to detect and mitigate these threats effectively. This blog post explores prevalent AI attacks and provides practical KQL queries using Microsoft Sentinel’s SecurityEvent and AzureDiagnostics tables to identify suspicious activities.

Common AI Attacks

1. Data Poisoning: Attackers manipulate training data to bias model outputs, often by injecting malicious inputs or outliers. This can lead to incorrect predictions or degraded performance.

2. Model Evasion: Adversaries craft inputs to bypass AI model defenses, such as submitting adversarial examples that trick the model into misclassifying data.

3. Model Inversion: Attackers exploit model outputs to reconstruct sensitive training data, compromising privacy.

4. Model Extraction: Malicious actors repeatedly query the model to replicate its functionality, stealing intellectual property.

5. Probing and Brute-Force Attacks: Attackers send rapid or malformed requests to probe for vulnerabilities or overwhelm the system.

By monitoring logs in Microsoft Sentinel, organizations can detect these attacks early. Below are five KQL query examples tailored to identify these threats using the SecurityEvent and AzureDiagnostics tables, which are commonly available in Sentinel environments.

KQL Queries for AI Attack Detection

The following queries assume a log structure with fields like TimeGenerated, Account, SourceIp, EventID, OperationName, ResultType, request_s, response_s, payloadSize_d, confidenceScore_d, and anomalyScore_d. Adjust field names and thresholds to match your environment.

1. Detecting Rapid Repeated API Calls (Probing or Brute-Force)

Rapid, repeated requests to AI endpoints may indicate probing or brute-force attacks. This query monitors logon events to AI-related accounts or APIs.

// Detect rapid repeated API calls to AI endpoints
SecurityEvent
| where EventID == 4624 or EventID == 4672 // Logon events
| where TargetAccount contains "AI" or LogonProcessName contains "api"
| summarize RequestCount = count() by SourceIp, bin(TimeGenerated, 1m)
| where RequestCount > 50
| project TimeGenerated, SourceIp, RequestCount

Why it works: This query flags IPs making over 50 requests per minute, which is unusual for legitimate users and may indicate automated probing.

2. Identifying High-Confidence Model Outputs from Suspicious IPs

Adversarial inputs often produce unusually high-confidence outputs. This query detects such patterns from uncommon IPs.

// Identify high-confidence AI model outputs from unusual IPs
AzureDiagnostics
| where OperationName == "InferenceRequest"
| where confidenceScore_d > 0.99
| summarize HighConfidenceCount = count() by SourceIp, Account
| where HighConfidenceCount > 10
| project TimeGenerated, SourceIp, Account, HighConfidenceCount

Why it works: High confidence scores (>0.99) from a single IP with many requests suggest potential adversarial inputs designed to exploit the model.

3. Detecting Data Poisoning via Large Payloads

Data poisoning often involves submitting oversized or outlier inputs. This query identifies unusually large payloads in training data uploads.

// Detect potential data poisoning via large input payloads
AzureDiagnostics
| where OperationName == "TrainingDataUpload"
| where payloadSize_d > avg(payloadSize_d) + 3 * stdev(payloadSize_d)
| project TimeGenerated, Account, payloadSize_d, requestUri_s

Why it works: Payloads exceeding three standard deviations from the average size are flagged as potential poisoning attempts.

4. Flagging Model Inversion via Sensitive Data in Outputs

Model inversion attacks aim to extract sensitive data from model outputs. This query searches for sensitive patterns, like Social Security Numbers, in responses.

// Detect model inversion attempts by monitoring sensitive data
AzureDiagnostics
| where OperationName == "InferenceRequest"
| where response_s contains "sensitive" or response_s matches regex "[0-9]{3}-[0-9]{2}-[0-9]{4}"
| project TimeGenerated, Account, response_s, SourceIp

Why it works: Detecting sensitive data patterns in model outputs indicates a potential breach of training data privacy.

5. Detecting Model Extraction via High-Frequency Unique Queries

Model extraction involves querying the model repeatedly to replicate its behavior. This query identifies accounts with a high number of unique requests.

// Detect model extraction via high-frequency unique queries
AzureDiagnostics
| where OperationName == "InferenceRequest"
| summarize UniqueQueries = dcount(request_s) by Account, bin(TimeGenerated, 1h)
| where UniqueQueries > 50
| project TimeGenerated, Account, UniqueQueries

Why it works: A high count of unique queries within an hour suggests an attempt to systematically map the model’s behavior.

Implementing These Queries in Microsoft Sentinel

To use these queries:

1. Ingest Logs: Ensure AI system logs (e.g., inference requests, training data uploads) are ingested into Sentinel via Azure Diagnostics or custom connectors.

2. Create Alerts: Save these queries as analytics rules in Sentinel, configuring thresholds and schedules based on your environment.

3. Tune Thresholds: Adjust thresholds (e.g., RequestCount > 50, confidenceScore_d > 0.99) to minimize false positives while catching true threats.

4. Correlate with Other Logs: Join with SigninLogs or AADNonInteractiveUserSignInLogs for additional context, such as user authentication patterns.

5. Respond to Incidents: Set up automated responses, like blocking IPs or notifying security teams, using Sentinel playbooks.

Best Practices for Securing AI Systems

• Input Validation: Sanitize and validate all inputs to AI models to prevent adversarial examples and poisoning.

• Rate Limiting: Enforce API rate limits to deter probing and extraction attempts.

• Anomaly Detection: Use machine learning in Sentinel to baseline normal behavior and flag deviations.

• Access Controls: Restrict access to AI endpoints using Azure AD and monitor unauthorized access attempts.

• Regular Auditing: Continuously audit logs for suspicious patterns and update KQL queries as new threats emerge.

TLDR

AI systems are powerful but vulnerable to sophisticated attacks. By leveraging Microsoft Sentinel’s robust logging and KQL’s querying capabilities, organizations can proactively detect and respond to threats like data poisoning, model evasion, and model inversion. The provided KQL queries offer a starting point for building a comprehensive AI security monitoring strategy. Regularly refine these queries and integrate them with Sentinel’s alerting and automation features to stay ahead of attackers.

For more advanced setups or custom connectors, refer to Microsoft Sentinel documentation or consult with your security team to tailor these queries to your AI environment.