Care and Feeding of HSMs: Key Management in Hard Mode

Care and Feeding of HSMs: Key Management in Hard Mode

Speaker: Nick Pelis

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: ~8 minutes

TL;DR

Hardware Security Modules (HSMs) are the right answer for protecting high-value cryptographic keys — but operating them in practice is a catalog of operational disasters waiting to happen. Nick Pelis, a security engineer at Verkada, walks through the full lifecycle of HSM key management with battlefield candor: a $15,000 unit that went on vacation, a quorum that locked itself out, a battery replacement that resembles bomb disposal, and a smart card overwritten months before it was needed.

Introduction

Cryptography solves eavesdropping. But cryptography introduces a new problem: what do you do with the key? This is not a novel insight — NIST SP 800-57, the recommendation for key management, has addressed it for years. What Pelis offers is something the documentation does not: a first-person account of what key management looks like when it collides with real organizations, real human error, and hardware that is simultaneously critical and deeply annoying to operate.

Pelis came to HSMs sideways — "it was not intentional," he admitted — through work in embedded systems security at Verkada. His talk is structured around two contrasting key scenarios (TLS session keys versus IoT secure boot keys), the options available for protecting long-lived private keys, and a series of horror stories that make NIST 800-57's abstract principles suddenly concrete.

▶ Watch: The key management problem and NIST SP 800-57 (02:00)

A Framework for Thinking About Keys

Before reaching the HSM horror stories, Pelis walked through a useful framework for evaluating any cryptographic key against six properties from NIST SP 800-57:

1. Crypto period — How long is the key valid? Session keys last seconds; IoT device keys may last the device lifetime.
2. Security strength — How strong is the key, in bits?
3. Key revocation — What happens if the key is compromised? Can it be revoked, and at what cost?
4. Archive and recovery — What happens if the key is lost?
5. Cost to replace — Is replacement trivially cheap, or potentially company-ending?
6. Auditing — Do you know who has accessed or used the key?

▶ Watch: TLS session key analysis (06:00)

A TLS 1.3 session key scores well on nearly all dimensions: short crypto period, ephemeral generation means zero cost to replace, loss exposes only a single session, auditing is largely irrelevant. The boot ROM private key of an IoT device scores poorly on almost everything that matters most: the crypto period extends for the life of every device ever shipped, revocation requires writing to extremely limited write-once memory, and losing the key may mean recalling every device in the field — potentially a company-bankrupting event. The asymmetry between these two cases is what makes the IoT scenario genuinely difficult and what justifies the operational complexity of HSMs.

Pelis used the house key analogy to make the concepts accessible: a lock with seven or eight pins is harder to pick than one with a single pin; a single master key for every lock in your house means a single point of failure. These are not trivial metaphors — they map directly to key strength, key isolation, and the blast radius of a compromise.

The Options for Protecting High-Value Keys (and Why They All Have Problems)

▶ Watch: Key storage options — laptop, git, paper, USB, cloud (12:01)

Before arriving at HSMs, Pelis ran through the alternatives:

• Laptop: Convenient, but the laptop can die or be stolen.
• Git repository: Accessible to the whole team, but once a key is in Git it is never truly deleted — and leaks happen. The UEFI Secure Boot key leak was cited as a recent real-world example.
• Paper: Offline and thus immune to network attacks. Used by some cryptocurrency exchanges. But paper can be copied, degrades over time, and has no auditing.
• USB stick: Easily stolen; no tamper detection.
• Cloud KMS (AWS KMS, Azure Key Vault, etc.): Excellent availability, managed backups, good IAM integration. But you are renting someone else's infrastructure, and you lose control over the key material itself.

This is the setup for HSMs: purpose-built physical computers designed specifically to store, create, manage, and protect cryptographic keys. They come in form factors ranging from a YubiKey-sized USB device ($600) to rack-mounted appliances costing over $150,000. They offer tamper detection and response — if the device is physically modified or loses power in certain configurations, it wipes itself.

The downside: they are extremely difficult to operate correctly. Pelis offered four war stories to illustrate exactly how.

Horror Story 1: The HSM That Went on Vacation

▶ Watch: The missing HSM (18:02)

An engineer was doing setup work on a $15,000 HSM and borrowed it over the weekend while working late to meet a deadline. Then they left for two weeks. The team spent days in a panic wondering where the device had gone, whether the engineer had taken it with them, and what had happened to it. "There were a lot of questions raised," Pelis said drily.

Lesson: HSMs need physical controls, inventory tracking, and clear policies about who can take them where. A device storing high-value key material should not be treated like a development board.

Horror Story 2: Quorum Failure and Emergency Key Rotation

▶ Watch: PIN failures, quorum lockout, emergency key rotation (20:02)

PKCS#11, the standard interface for HSMs, defines a role hierarchy: Security Officers can administer the device and manage other roles; Crypto Officers can create, use, and destroy keys; Crypto Users have the most limited privileges. Access to these roles is typically protected by PINs.

Pelis needed to reset the Crypto Officer role after someone forgot their PIN. To do this, a Security Officer had to authenticate. The team was using a two-of-five quorum for the Security Officer role — meaning two out of five authorized individuals needed to present their credentials. One of the two individuals attempting authentication entered the wrong PIN three consecutive times. On an HSM, a Security Officer lockout has no higher authority to appeal to: the device simply wiped all keys.

The recovery required an emergency key rotation — including pushing new firmware to IoT devices in the field. "People make mistakes," Pelis noted, with some understatement. Operating procedures, including PIN management policies, must account for human error at the highest privilege levels.

Horror Story 3: Battery Replacement as Bomb Defusal

▶ Watch: The battery replacement procedure (22:02)

HSMs store key material in battery-backed SRAM. When the main power is removed, the battery keeps the keys alive. When the battery dies, the keys die.

The manufacturer's procedure for battery replacement: insert a temporary backup battery holder into port P8; hold it in place with one hand; use the other hand to unscrew and remove the main battery; insert the replacement battery; screw it back in. If the temporary cable is accidentally pulled out, or if the device is dropped, all keys are lost. Pelis described this as "like defusing a bomb" — except that you chose to put your most sensitive keys in this device precisely because they are high value.

Pelis's operational response: always maintain multiple HSMs in multiple locations — a primary and a hot spare on-site, with additional off-site backups. If one device dies during a maintenance procedure, the operation can continue on another unit.

Horror Story 4: The Forgotten Smart Card

▶ Watch: Smart card recovery, tamper state, and the overwritten card (24:03)

Some HSM models can be configured to require a smart card in order to recover the device from a tamper state (triggered by loss of power, physical modification, or certain configuration changes). When Pelis set up the device, he issued a recovery smart card. Three months later, he needed a blank smart card for something else and overwrote it. Some time after that, the HSM entered a tamper state when the battery died.

Without the recovery smart card, the device could not be reset. It had to be physically shipped back to the manufacturer. The device — and its keys — were completely unusable in the meantime.

"When you see things in the manual that look really enticing," Pelis advised, "you need to read them multiple times and think really closely about what the implications are."

Quorum Design and the Ceremony

▶ Watch: Key management ceremonies, PKCS#11 roles, and quorum design (16:02)

Key management ceremonies are formal procedures for creating, distributing, and authorizing cryptographic keys. DNSSEC runs one four times per year to update the root zone signing key — it is live-streamed, and attending in person is possible by request. Zcash ran one at its currency launch (covered in a Radiolab episode Pelis recommended). Every organization operating an HSM needs to design their own.

A key failure mode during ceremonies: mid-ceremony interruptions. In one instance, Pelis's team was transitioning from a two-of-four to a two-of-five quorum for the Security Officer role. As they were reprogramming smart cards round-robin — going from person to person to reprogram each card — the HSM threw an error and locked up on the third person. Three of the four original smart cards had been overwritten, leaving fewer than the required two valid cards for the existing quorum. The device was in an irrecoverable state.

The lessons: always rehearse critical operations on a test HSM first; never overwrite existing smart cards mid-ceremony until the new ones are confirmed working; maintain a test unit that mirrors the production configuration.

Notable Quotes

"You've basically traded the problem of the eavesdropping for the problem of what to do with the key." — Nick Pelis at 02:00

"Like, never mind the fact that you chose to put your keys in a forty-five thousand dollar HSM because they're high value. This is like defusing a bomb." — Nick Pelis at 22:02

"I think there's an area here that's ripe for innovation. I also think that the world market for HSMs is not big enough to justify that innovation." — Nick Pelis at 30:04

Key Takeaways

• Match key management rigor to key criticality. A TLS session key requires no special handling. An IoT device boot ROM private key that cannot be revoked without a device recall justifies the full operational burden of an HSM.
• Cloud KMS is the right answer for most use cases. On-premises HSMs are appropriate when keys are extremely high value, access is infrequent, or offline storage provides a meaningful threat model simplification. Otherwise, the operational overhead is not worth it.
• Treat HSMs as operational systems, not security boxes. Physical access controls, inventory tracking, power redundancy, battery maintenance schedules, and role management are all part of HSM operations. Skipping any of them creates scenarios where the safeguard destroys the thing it was meant to protect.
• Rehearse every critical operation on a test HSM first. Do not perform a quorum re-key, a role reset, or a battery replacement on a production unit without having executed the same procedure on a test unit first.
• PKCS#11 is legacy technology with limited tooling. The standard interface for HSMs is defined in C and loses functionality through every layer of language bindings. The ecosystem is ripe for innovation but lacks the market size to attract it — practitioners should plan for rough edges.