Service Mesh Security: Shifting Focus to the Application Layer

Service Mesh Security: Shifting Focus to the Application Layer

Speaker: Daniel Popescu

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: https://www.youtube.com/watch?v=_SBWD12S4KE

Reading time: 8 minutes

TL;DR

After years of failed attempts to bolt security onto Yelp's service mesh at the infrastructure layer, security group tech lead Daniel Popescu and his team pivoted to the application layer — using JWTs, Open Policy Agent, and shared middleware libraries to achieve robust service-to-service authentication and authorization with under 5 milliseconds of added latency at the 95th percentile.

Introduction

Securing service-to-service communication is one of the most vexing problems in modern microservices architecture. On paper, the answer seems obvious: push security into the infrastructure layer, handle it transparently, and remove the burden from developers. In practice, that ambition has a stubborn habit of colliding with operational reality.

Daniel Popescu, group tech lead for security at Yelp, spent roughly 75% of his nine-year tenure at the company grappling with this exact problem. In his BSidesSF 2025 talk, he walked the audience through a multi-year journey involving abandoned attempts with Envoy and Open Policy Agent at the infrastructure layer, evaluations of AWS App Mesh, Istio, and Cilium, and ultimately a pragmatic pivot that actually shipped. The lesson is not that infrastructure-layer security is wrong in principle — it's that perfection can be the enemy of progress, and the best security solution is one you can actually deploy.

Yelp operates at significant scale: 29 million unique devices served monthly, 308 million reviews, and hundreds of microservices maintained by dozens of teams. In that environment, every security decision carries broad operational consequences.

The Service Mesh Hierarchy of Needs

▶ Watch: Background on Yelp's infrastructure (02:30)

Popescu opened by framing the problem through a clever adaptation of Maslow's hierarchy of needs applied to service mesh maturity. Basic connectivity sits at the base — if services can't find each other, nothing else matters. Above that come reliability, then observability, then adoption, and only at the apex does security appear. This is not because security is unimportant, but because it cannot be effectively implemented until the foundation beneath it is stable.

Yelp adopted SmartStack (an open-source service mesh from Airbnb) in 2015 when its container workloads ran on Mesos. Security wasn't a feature of that era's mesh ecosystem — the focus was on service discovery and reliability. As the years passed, feature teams began asking a question the security team couldn't answer well: "How do I protect my service from unauthorized requests?" Without a clear, paved path, some teams built custom solutions with mixed results. Others simply left services unprotected — not out of negligence, but because doing it right was too hard.

The Infrastructure-Layer Attempts That Didn't Survive Production

▶ Watch: First attempt with Envoy and OPA (08:15)

The landscape changed in late 2018 when Yelp began migrating from Mesos to Kubernetes and introduced Envoy. Popescu's team saw an opportunity: Envoy's built-in support for mutual TLS and authorization policies was exactly what they had been waiting for. They built a system using native Envoy constructs and Open Policy Agent to handle authentication and fine-grained authorization at the infrastructure layer — fully transparent to application code, language-agnostic, and centralized. The team even presented it at KubeCon 2019.

Then they had to roll it back.

Yelp's service mesh is atypical: it has no sidecars and no ingress proxies, which are the two standard enforcement points most service mesh security models rely on. To make their solution work, the team bolted on dedicated ingress proxies and had to resolve identities without egress sidecars present. The custom components introduced instability. When reliability is on the line in a service mesh, security enhancements have to yield — and they did.

Subsequent evaluations of AWS App Mesh, Istio, and Cilium each produced dead ends. App Mesh's feature set was too limited and the product was later discontinued. Istio had the right features but carried enormous operational complexity. Cilium only supported Layer 4 authorization in the configuration Yelp required, and they needed Layer 7 policies. Each evaluation consumed months of prototyping and testing with nothing reaching production.

The Pivot to the Application Layer

▶ Watch: Decision to shift to the application layer (18:40)

After exhausting infrastructure-layer options, the team asked a different question: what if they stopped trying to solve this at the infrastructure layer entirely? Application-layer security carries a reputation for being harder to scale and more burdensome to developers. But it also offers properties that infrastructure-layer security struggled to deliver at Yelp: decoupling from infrastructure changes, safe incremental deployment, and full control over authentication and authorization logic.

The team settled on four core security requirements. Authentication must use short-lived, cryptographically signed credentials — no shared API keys, no long-lived tokens. Authorization must be centralized and enforce fine-grained policies, such as role-based or method-level access controls, without every service reinventing the wheel. Observability must provide comprehensive logs, metrics, and traces. And usability must be high enough that service owners will actually adopt it — security that is hard to adopt simply will not be adopted.

Encryption was explicitly excluded from the requirements. Traffic between services already runs on hosts with strict least-privilege access controls, and the risk of wire sniffing was outside the primary threat model.

JWT Authentication and the "curl-with-jwt" Wrapper

▶ Watch: Authentication architecture with JWTs (22:00)

The team chose JSON Web Tokens (JWTs) as the identity mechanism. Client certificates were considered but ruled out: implementing TLS at the application layer would interfere with Yelp's existing observability tooling (which relies on inspecting plaintext traffic), and managing certificate issuance, rotation, and revocation across hundreds of services would introduce significant operational complexity.

JWTs are short-lived, cryptographically signed, and support a unified identity model across all actor types. For human users, Yelp uses Okta. For Kubernetes pods, it uses service account tokens issued by the API server. For workloads running on EC2 instances or Lambda functions, it uses HashiCorp Vault, which can issue JWTs based on an IAM role.

Developer experience received serious attention. Humans typically interact with services via curl, so the team built a wrapper called curl-with-jwt that prompts for Okta credentials, fetches a signed JWT, and automatically adds it to the request. Credentials are cached locally for up to one hour (or however long the Okta session is configured). For Kubernetes services using auto-generated client libraries, the authentication step happens transparently — service account tokens are read from disk and appended to outgoing requests with no code changes required, as long as the service uses a recent version of the shared library.

The token cache files are monitored: accessing another user's cache file triggers an alert, a detail Popescu demonstrated by showing a log entry from testing the alert himself.

Open Policy Agent and the YAML Abstraction Layer

▶ Watch: Authorization with OPA and YAML policies (28:50)

Authorization is handled by Open Policy Agent, running as a standalone service in ECS Fargate — deliberately decoupled from Kubernetes and the service mesh. When a request reaches a service, middleware in the shared HTTP serving library extracts the JWT, the request path, and other metadata, sends them to OPA for evaluation, and handles the allow/deny response. Opting in requires setting a single config flag, assuming the service uses the latest version of the middleware library.

The middleware supports a dry-run mode, in which all requests are allowed but authorization decisions are logged. This lets teams understand what would have been blocked before enforcement goes live, and enables them to generate initial policy definitions from real traffic patterns.

OPA's native policy language is Rego, which is powerful but not always approachable for application developers. The team built a YAML abstraction layer on top of it. Service owners define access rules using familiar HTTP concepts — paths, methods, and identity types (owning team, AD group, other services, IAM roles) — without writing Rego. The YAML definitions live in GitHub under branch protection and version control. A custom tool called the OPA Policy Manager processes updates, validates policy bundles, and uploads them to S3; OPA instances periodically pull the latest bundle.

Performance optimizations are layered throughout the system. JWT verification happens offline: public keys from identity providers are prefetched and bundled directly into the policy bundle, eliminating network calls in the critical path. Multi-tiered caching covers client-side JWT reuse, server-side authorization result caching, and OPA-level caching. Ninety-nine percent of authorization decisions are served from one of these caches. The result: under 5 milliseconds of added latency at the 95th percentile. Service owners reported no noticeable impact on response times, which Popescu described as critical for adoption.

Results and Key Takeaways

▶ Watch: Results and takeaways (38:00)

Observability runs at multiple levels: Splunk receives every OPA decision log, including the requestor identity, the resource being accessed, and the reason for the allow or deny decision. A public runbook documents every possible authorization result and how to address it. Dashboards track request volumes and the distribution of authorization decisions in real time, providing early warning of misconfigurations or unexpected access patterns.

After years of infrastructure-layer dead ends, the application-layer approach actually shipped and achieved meaningful adoption across Yelp's service estate.

Notable Quotes

"Perfection was starting to become the enemy of the good. We didn't need a flawless solution. We needed one that we could actually deploy."

"If security ends up being hard for people to adopt, it simply won't be adopted."

"We'd warned about the dragons they might encounter if they tried to build something custom on their own, but we couldn't offer a simple paved path."

Key Takeaways

• Infrastructure-layer security is fragile if your mesh is non-standard. Yelp's lack of sidecars and ingress proxies invalidated standard service mesh security models. Know your infrastructure constraints before committing to a design.
• Application-layer security offers deployment control that infrastructure-layer approaches often can't. Incremental rollout, dry-run modes, and per-service opt-in are easier to achieve when security lives in shared libraries rather than the mesh.
• Usability is a security requirement. The curl-with-jwt wrapper, auto-generated client library integration, and one-flag opt-in exist because if adoption is painful, it won't happen.
• YAML over Rego. Abstracting OPA's policy language behind a YAML DSL removed a significant barrier for service owners who don't want to learn Rego just to define access rules.
• Cache aggressively, verify offline. Bundling identity provider public keys into the policy bundle and caching at multiple layers brought authorization overhead to under 5ms at p95 — acceptable for high-throughput microservices.