What Do You Mean, "Resource Not Found?" Demystifying GCP Error Codes for IR & Detections

What Do You Mean, "Resource Not Found?" Demystifying GCP Error Codes for IR & Detections

Speakers: Gabriel Fried

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=8cScKAzuEz0

Overview

Gabriel Fried, a principal security researcher at Mitiga, presented a remote talk on leveraging GCP error codes as a detection and incident response signal. The central thesis is that security teams overwhelmingly focus on successful actions when building detections, but the errors that precede those successes -- failed resource lookups, permission denials, quota exhaustion -- can serve as early warning indicators of reconnaissance, credential probing, and resource abuse. Fried walked through the structure of GCP error codes (rooted in gRPC and the AIP-193 standard), demonstrated how the error detail section provides richer context than the status code alone, explained a nuance around long operations and null principal emails, and presented three practical PySpark-based detection examples with comparative analysis of naive versus context-aware approaches.

Background

▶ Watch: Reframing errors: what if resource-not-found is actually enumeration? (2:00)

GCP's internal service communication uses gRPC, the open-source successor to Google's internal Stubby protocol, built on Protocol Buffers and HTTP/2. When a client sends an API request to GCP, it passes through the Google Front End (GFE) for load balancing and DoS protection, then internal services communicate via gRPC or Stubby. Error codes in GCP audit logs follow the AIP-193 (API Improvement Program) standard, which was established in 2018 to standardize error reporting across Google's many services. This standard was enhanced in 2020 with error info -- a detailed metadata section within the error details.

Despite this structured error reporting, most security teams treat GCP errors as noise to be filtered out rather than signal to be analyzed. Fried argues this is a missed opportunity: error patterns can reveal attacker behavior before successful exploitation occurs, or provide crucial timeline context during incident response.

Key Findings

▶ Watch: Error code anatomy: code, message, and the treasure chest details section (6:00)

GCP error codes have three sections that provide progressively richer context:

1. Code: A canonical number (0-16) indicating the error type -- 0 for OK, 3 for invalid argument, 5 for not found, 6 for already exists, 7 for permission denied, 8 for resource exhausted, 13 for internal error, 16 for unauthenticated
2. Message: Human-readable text intended for developer debugging, though its population varies across services (different services were written by different teams at different times)
3. Details: The richest section, containing error info with domain (e.g., compute, storage, service usage), reason (textual explanation), and metadata (affected resources, specific permissions, scope)

These map closely to HTTP status codes (5 to 404, 7 to 403, 8 to 429/500, 13 to 500, 16 to 401), allowing teams familiar with HTTP-based detection to translate their knowledge.

The error info section is what Fried calls "the treasure chest." For example, a code 7 (permission denied) might include the exact permission that was denied and the service domain, revealing whether an attacker was trying to enable services, access storage, or modify IAM policies. A code 8 (resource exhausted) might reveal that a specific zone's resource pool was depleted -- a potential indicator of cryptojacking that filled the zone with expensive VMs.

Fried also identified a nuance with long operations (builds, exports, large VM creations) where the error log entry has a null principal email. Long operations report status via an operation ID, and the error event is logged as the "last" operation. To find the initiating user, defenders must join on the operation ID where first = true to recover the principal email from the initial triggering event.

Technical Deep Dive

▶ Watch: Long operations and recovering null principal emails via operation IDs (12:00)

Fried presented three detection scenarios with naive and refined implementations in PySpark pseudocode:

Detection 1: Reconnaissance via Resource Enumeration. The naive approach flags any IP with more than 15 errors in 30 minutes -- producing massive false positives because it catches typos, automation errors, and legitimate failures alongside actual reconnaissance. The refined approach filters specifically for code 5 (not found), counts distinct assets that weren't found per user/email/project, and scores based on breadth of the scan (how many different resource types are being probed). This reduced alert volume to approximately 2.04% of what the naive approach produced.

Detection 2: Misconfiguration vs. Malicious Permission Probing. The scenario: an attacker has multiple stolen service account keys and is probing which accounts have access to a specific service. The naive approach flags any account with more than 20 permission denials in 15 minutes. The refined approach extracts the reason from error info to differentiate between service-disabled errors (likely misconfiguration, especially if only one account is affected) and multiple callers failing on the same API with the same reason in a short window (likely credential probing). This yielded only 6 alerts in one test environment and zero in others.

Detection 3: Permission Probing with Stolen Credentials. Similar to Detection 2 but focused on a single stolen service account being tested against multiple services. The refined approach extracts specific permissions from the error message or method name, groups by user and time window, and applies a threshold. This achieved a 0.13% alert rate compared to the naive approach and actually identified a real permission scanning event in test data.

Demo / Proof of Concept

▶ Watch: Detection example: naive vs refined reconnaissance detection (97% FP reduction) (14:00)

No live demo was performed. Detection examples were presented as PySpark pseudocode with comparative statistics showing the false positive reduction from naive to context-aware approaches.

Defensive Implications

▶ Watch: Permission scanning detection at 0.13% false positive rate (18:00)

Fried's recommendations for implementing error-based detection in GCP environments:

1. Map error codes as first-class fields in your SIEM or analytics platform -- do not leave them buried in nested JSON structures
2. Baseline callers and their error patterns: establish what normal error rates look like for each service account and user, then alert on deviations
3. Stitch operation IDs to recover principal identities from long-running operations where the error event has a null principal email
4. Build detections around error info domains and reasons, not just status codes -- the domain (compute, storage, service usage) and reason fields provide the context needed to distinguish reconnaissance from operational noise
5. Map gRPC error codes to HTTP status codes to leverage existing HTTP-based detection logic and team knowledge
6. Start with error-based incident response playbooks before building live detections -- use error patterns to reconstruct attacker timelines during investigations

The suggested implementation timeline: start mapping codes immediately, conduct an error-based IR exercise within a week, and build error-based detections within a month.

Key Takeaways

• GCP error codes follow the AIP-193/gRPC standard with three sections (code, message, details) that provide progressively richer detection context
• The error info section (domain, reason, metadata) is the most valuable for distinguishing attacker behavior from operational noise
• Resource not found errors in bursts can indicate reconnaissance; permission denied patterns across multiple accounts targeting the same service suggest credential probing
• Long operations with null principal emails can be resolved by joining on the operation ID to find the initiating user
• Context-aware detections using error info details reduce false positive rates by 97-99% compared to naive status-code-only approaches
• Error codes map to HTTP status codes, allowing teams to translate existing HTTP detection knowledge to GCP gRPC-based logging

About the Speaker(s)

Gabriel Fried is a principal security researcher at Mitiga with over 10 years of experience in cybersecurity. He specializes in cloud incident response and detection engineering. This was his second presentation at fwd:cloudsec. He presented remotely.