Securing organizations ML & LLMops deployments: A platform architect's journey onboarding LLM & MLops tools and securing multi-cloud data access

Securing Organizations' ML & LLMOps Deployments: A Platform Architect's Journey Onboarding LLM & MLOps Tools and Securing Multi-Cloud Data Access

Speakers: Kyler Middleton; Sai Gunaranjan

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=oxTsbP-iv7Q

Overview

Kyler Middleton (Principal Developer for Internal AI Solutions) and Sai Gunaranjan (Lead Architect, Cloud Platform Team) from Veradyne, a U.S. healthcare company, delivered a dual-track presentation covering the practical realities of securing AI deployments across AWS and Azure. Rather than presenting novel research or vulnerability findings, the talk functions as an architectural reference guide from practitioners who are building and securing production AI systems in a regulated industry. They cover AWS Bedrock's architecture, capabilities, and security gaps; Azure AI services' deployment patterns, network controls, data access patterns, and policy enforcement; and the broader platform engineering principles that underpin secure AI adoption. The talk's value lies in its candid documentation of what works, what is broken, and what is missing from both cloud providers.

Background

▶ Watch: Where is your data and who can access it: bots as data funnels (2:00)

AI adoption in enterprise environments is accelerating rapidly, with the speakers estimating that effectively all security teams will be dealing with AI deployments within a year. The challenge is that AI engineering is fundamentally data engineering with an identity and authentication problem. Organizations need to know where their data is, how it is protected, who can access it, and how AI systems authenticate to data sources. Bots act as a funnel for data -- anything a bot can access is transitively accessible to any user who can interact with that bot.

The speakers identify several converging trends: data is propagating across multiple clouds and SaaS platforms; MCP (Model Context Protocol) has matured rapidly under the Linux Foundation, enabling agentic bots to interact programmatically with tools and APIs; the A2A (Agent-to-Agent) protocol is emerging for agent-to-agent communication; machine identities are increasingly acting as users on corporate networks; and developers are using AI to generate code without clear policies on testing or classification.

Veradyne operates in healthcare, one of the most regulated industries in the United States, which means their AI adoption process involves legal teams, information security, and compliance review. Their conservative approach provides a useful reference point for organizations that need security-first AI deployment patterns.

Key Findings

▶ Watch: AWS Bedrock architecture: knowledge bases, reranker, and guardrails pipeline (12:00)

AWS Bedrock Security Gaps: The speakers documented several significant security limitations in AWS Bedrock. There are no resource policies for Bedrock resources -- you cannot attach a policy to a knowledge base or model deployment to restrict access, unlike S3 buckets or SQS queues. AWS Config has no rules for Bedrock resources because they are not recognized as traditional AWS resources. Guardrails are optional and specified per API call, meaning developers can simply omit the guardrail parameter and bypass all content filtering. Foundational models accessed through Bedrock do not support document types (PDFs, images) despite the underlying models' native support -- a limitation of Bedrock's intermediation layer. The web crawler data source for knowledge bases has no web authentication support, meaning any data behind a login (even basic auth) cannot be ingested.

AWS Bedrock Architecture: Their production architecture uses a Lambda receiver for immediate webhook response, a Lambda worker for Bedrock interactions, knowledge bases backed by Aurora (preferred for cost at ~$1/day vs OpenSearch at ~$40/day), a reranker to winnow vector search results (described as dramatically improving response fidelity at negligible cost and latency), and guardrails policing both input and output tokens. The full pipeline executes in 3-6 seconds and processes approximately 150 requests per day for roughly $1/day total cost.

The Converse API is highlighted as a critical enabler. This meta-API proxies requests to any Bedrock model, transforming the API format on the fly. It enables easy model switching, supports all document types that models natively handle, and insulates applications from model-specific API differences.

Azure AI Security Controls: Sai detailed the deployment security stack for Azure AI services, emphasizing that defaults are not secure across any provider. Key recommendations include: enabling Microsoft-managed networking for ingress/egress control; deploying private endpoints and private link for all supporting services (ACR, storage accounts, Key Vault, databases); integrating Azure Monitor with diagnostic settings and activity logs; carefully selecting model deployment methods to prevent data from being processed outside geographic boundaries (global standard vs. standard deployment); and using Key Vault for all secrets rather than hardcoding credentials.

Azure Policies provide enforcement capability that AWS Bedrock lacks. Organizations can deploy policies that block insecure model deployments, block untrusted marketplace vendors, and enforce security configurations at the platform level. Content filtering in Azure AI workspaces is tightly integrated with model deployments, unlike Bedrock where guardrails are optional per API call.

MCP Security Concerns: The speakers addressed the evolution of MCP from a confused deputy problem (everyone gets the bot's permissions) to token-passing architecture (users prove identity via OAuth 2 tokens). However, they noted that administrative users providing tokens to MCP-connected bots effectively grant those bots administrative access, creating a need for rights pruning during token issuance.

Technical Deep Dive

▶ Watch: CloudWatch logging limitations: cannot scope to specific agents or deployments (20:00)

The AWS Bedrock architecture demonstrates several practical design decisions. The two-Lambda pattern (receiver + worker) solves the webhook timeout problem -- webhooks expect fast HTTP 200 responses, but Bedrock interactions take 3-6 seconds. The receiver acknowledges immediately and invokes the worker asynchronously.

The reranker sits between the knowledge base vector search and the foundational model. The knowledge base returns 50-75 text chunks from the vector store, and the reranker reads both the user's query and all chunks to identify the most relevant ones. This winnowing step takes approximately 250 milliseconds and dramatically improves response quality while reducing the token load sent to the foundational model. The speakers noted that AWS plans to make reranking a checkbox option in the knowledge base API rather than requiring a separate model invocation.

On the Azure side, the network architecture places AI workspaces behind Microsoft-managed networks with firewall rules governing outbound access to specific endpoints (PyPI, model stores, etc.). The identity layer uses managed identities on ML workspaces to authenticate to data sources (SQL databases, storage accounts, potentially Snowflake) via Entra ID rather than static credentials. Data scientist access flows through VPN tunnels to private AI workspace endpoints.

CloudWatch logging for Bedrock records all conversations but cannot be scoped to specific agents or deployments -- it captures all AI requests across all models in a region-account combination into a single log group. This makes disambiguation of different AI applications challenging.

Demo / Proof of Concept

▶ Watch: AWS Bedrock security gaps: no resource policies, no Config rules, optional gu... (22:00)

No formal demo was performed. The talk was structured as an architectural walkthrough with screenshots of real Azure policy deployments, Bedrock guardrail configurations, firewall rules, and deployment settings from the speakers' production environment.

Defensive Implications

▶ Watch: Azure Policies blocking insecure deployments and untrusted model vendors (32:00)

Platform engineering teams need to shift left into AI development processes just as they did with DevSecOps. If security teams are first encountering AI features when they reach production, they are too late. The default configurations for both AWS and Azure AI services prioritize cost and developer convenience over security, meaning secure deployment requires deliberate configuration at every layer.

For AWS Bedrock specifically: build custom monitoring through CloudTrail and CloudWatch since Config rules are not available; enforce guardrail usage through application-level policy or deployment automation since Bedrock does not enforce guardrails architecturally; prefer the Converse API for model interaction to maintain flexibility and document type support; and use Aurora over OpenSearch for knowledge base vector storage to reduce costs by approximately 40x.

For Azure AI: deploy with Microsoft-managed networking and private endpoints from the start; use Azure Policies to enforce secure configurations and block untrusted model deployments; leverage managed identities for data source authentication; carefully select deployment methods to control data geographic processing; and implement content filtering with custom block lists.

Both providers require that organizations build their own monitoring and governance layers on top of incomplete native security controls. The speakers' healthcare context makes their conservative, compliance-driven approach a useful template for any regulated industry.

Key Takeaways

• AI security defaults across both AWS and Azure prioritize cost and developer enablement over security; secure deployment requires deliberate configuration at every layer
• AWS Bedrock has no resource policies, no Config rules, and optional guardrails, creating significant governance gaps that platform teams must address through application-level controls
• Azure AI services offer stronger policy enforcement via Azure Policies and mandatory content filtering on model deployments, but require manual private endpoint and networking configuration
• The Converse API in AWS Bedrock is a critical enabler for model flexibility and document type support
• MCP's evolution to token-passing improves security but introduces risk when administrative users provide tokens to bots without rights pruning
• AI bots are data funnels: anything the bot can access is transitively accessible to users who interact with it, making data access governance the central security challenge

About the Speaker(s)

Kyler Middleton is a Principal Developer for Internal AI Solutions at Veradyne (a healthcare company in the United States), with a background in software engineering. He focuses on building internal AI applications and navigating the security and compliance requirements of the healthcare industry. Sai Gunaranjan is the Lead Architect on the Cloud Platform Team at Veradyne, responsible for securing the cloud infrastructure underlying AI deployments. Together, they represent the practitioner perspective of building production AI systems that must be compliant, secure, and cost-effective.