Unix Underworld: Tales from the Dark Side of z/OS

Unix Underworld: Tales from the Dark Side of z/OS

Speakers: Chad Rickensrud, Security Researcher, Broadcom; Phil Young, Security Researcher, Broadcom

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=3wQHhGxVTuo

Reading Time: ~8 minutes

Type: Briefing

TL;DR

Mainframes running IBM z/OS expose a Unix System Services (USS) subsystem that security teams already know how to attack — using the same enumeration scripts, privilege escalation patterns, and credential harvesting techniques that work on Linux. Chad Rickensrud and Phil Young demonstrate a full attack path from low-privileged SSH access to RACF special/operations attributes, using open-source tools written in REXX and shell scripting.

Introduction

IBM z/OS mainframes process trillions of dollars in transactions every day across financial institutions, governments, airlines, healthcare organizations, and retailers. Yet security testing of these systems remains rare, partly because the platform is perceived as arcane and impenetrable. Rickensrud and Young's talk dismantles that assumption by showing that z/OS contains a fully functional Unix subsystem — and anyone who can hack Linux already has most of the skills needed to hack a mainframe.

The Unix System Services (USS) component of z/OS is not a container or a separate OS. It is a Unix-like interface built directly into the operating system, complete with a hierarchical filesystem rooted at /, familiar shell environments, standard file permission octets, and SSH access. The pair have spent a decade building tooling for this environment and releasing it openly, and this talk packages their most effective attack paths into a single narrative aimed at bringing more researchers into the mainframe security space.

What You're Up Against: z/OS Terminology for Unix Hackers

▶ Watch: Platform Overview (04:00)

Before running any tools, it helps to understand three key concepts:

• RACF (Resource Access Control Facility): The default external security manager on z/OS — think of it as Active Directory for the mainframe. It controls authentication and authorization. Competing ESMs include Broadcom's ACF2 and Top Secret. RACF stores passwords in one of two formats: a legacy DES/MD5 scheme and a newer KDF-AES scheme using AES-256 with multiple iterations. The legacy format can be cracked at tens of billions of guesses per second using John the Ripper or Hashcat offline; the AES scheme is substantially slower but still brutable.
• APF (Authorized Program Facility): A mechanism that grants elevated CPU-level privileges to specific programs. APF-authorized programs run in "supervisor state" rather than "problem state," meaning they can execute privileged assembler instructions and access any physical memory via memory key zero. Getting code into APF authorization is the mainframe equivalent of achieving ring 0.
• Memory Keys: z/OS processes are associated with memory keys (0–15). Key zero is special — it can read and write any memory region on the system. Normal user processes run at key 8.

Enumeration: What the Open-Source Tooling Finds

▶ Watch: Enumeration Demo (12:00)

The speakers maintain a suite of open-source tools for z/OS Unix testing, all written in REXX or shell scripting to avoid dependencies on languages like Python that may not be present on older z/OS installations:

• enum.rex: A REXX script that reads information directly from memory control blocks in the running address space — bypassing logging that would be triggered by running conventional commands. It reads RACF database locations, encryption status, privilege information, and APF-related settings.
• OMVS Enum: A C-shell port of LinEnum adapted for z/OS Unix. Noisy but comprehensive; checks file permissions, accessible paths, SUID bits, and security-relevant profiles.
• ZosHog: The z/OS equivalent of TruffleHog — scans the Unix filesystem for plaintext credentials, tokens, and passwords embedded in configuration files (.properties, .java, database config files). "We see this all the time," Young noted. "A db.properties file or a Java properties file with a username and password."
• PortScan: Tests egress connectivity. The speakers have consistently found mainframes with wide-open outbound TCP access — including open SMTP relays — a configuration that enables data exfiltration and, in some cases, direct internet connectivity.

One of the first findings the enum script surfaces is whether KDF-AES encryption is enabled on the RACF database, and the database's physical location. If it is not enabled, offline password cracking is far faster.

Privilege Escalation via APF Authorization

▶ Watch: APF Exploitation (20:01)

The central privilege escalation path demonstrated in the talk exploits access to the BPX.FILEATTR.APF RACF profile. If an attacker has READ or better access to this profile, they can execute the extattr +a command on any file in the Unix filesystem, setting the APF authorized bit on an executable they control.

The attack chain:

1. Compile an assembler exploit that issues the privileged MODESET instruction to switch from problem state to supervisor state, then acquires memory key zero.
2. Mark the compiled binary APF-authorized using extattr +a.
3. Execute the binary. With supervisor state and key zero, the program can generate a new ACEE (Accessor Environment Element) — the control block that represents a user's privileges in their address space — for any high-privileged user (such as a RACF "SPECIAL" or "OPERATIONS" user).
4. Repoint the address in the current address space to the newly built ACEE, effectively impersonating that privileged user from a privilege-check perspective without triggering a re-authentication event.

Rickensrud demonstrated this live, running as a low-privileged user and then reading a file owned by another user whose ACEE he had swapped in. The system's privilege checks were reading the spoofed ACEE rather than the original.

An important nuance: achieving Unix UID 0 (superuser) is not the end of the story on z/OS. UID 0 confers superuser rights within the Unix subsystem but does not grant RACF SPECIAL attributes, which are needed for operations outside USS. The next step is to SSH in as a known admin — Young demonstrated creating an ~/.ssh/authorized_keys file under the target admin's home directory after achieving UID 0, then SSHing back in as that user to obtain full RACF privileges.

The Privileged Mount Attack

▶ Watch: Privileged Mount (28:01)

z/OS Unix filesystems are backed by ZFS data sets, not disk partitions. APF authorization bits for programs residing on a filesystem are stored inside those data sets. An attacker with the ability to perform a privileged mount — requiring UPDATE access to either BPX.SUPERUSER or BPX.FILEATTR.APF in RACF, or UID 0 — can mount a filesystem they built in their lab onto the production system.

The attack: build a ZFS on a lab system containing pre-compiled APF-authorized exploits (complete with APF bit and SUID bit set), FTP that ZFS to the target, and mount it on the target with privileged mount. The production system will honor those authorization bits as if they were set locally. "We run those exploits as APF authorized, even though we built them on a completely different system," Young explained.

Buffer Overflows and Honorable Mentions

The talk addresses buffer overflows in APF-authorized C programs running under z/OS Unix — once thought impossible on mainframes. Finding APF-authorized datasets is straightforward with a single find command querying extattr -E output. If a vulnerable C program in an APF-authorized library has a classic buffer overflow, the same shellcode techniques that work on x86 can be adapted for z/Architecture, leveraging the same ACEE manipulation described above as post-exploitation.

Additional attack surface mentioned includes:

• ESM misconfiguration overriding filesystem permissions: If certain RACF features are enabled, the ESM may grant access to files even when Unix permission bits deny it. The speakers encountered a live engagement where 700-permission private keys were readable due to ESM misconfiguration.
• Local file inclusion vulnerabilities in web applications running on z/OS, providing access to Unix files the attacker could not reach directly.
• Stored credentials in configuration files accessible via world-readable permissions.

Detection and Prevention

▶ Watch: Hardening Guidance (34:02)

Rickensrud and Young's recommended controls:

• Audit file permissions: Search for world-readable and world-writable files routinely. The same scripts used on Linux pen tests apply.
• Lock down three critical RACF profiles: BPX.SUPERUSER, BPX.FILEATTR.APF, and BPX.FILEATTR.PROGCTL. No daily-driver account should have access to these.
• Enable filesystem auditing: The ls -W flag reveals audit settings on z/OS Unix files. Using the chaudit command, administrators can configure audit events per file. Pair this with RACF SMF reporting to generate alerts.
• Monitor for egress: Thousands of outbound TCP connections from a mainframe should be alarming. Many systems the pair have tested have had completely open egress.
• Log analysis: Large numbers of unauthorized access attempts against files indicates someone scanning the filesystem. This activity is visible in SMF logs.

Notable Quotes

"You are all using this platform whether you know it or not, whether you like it or not, multiple times throughout your day if you're participating in society in any way, shape, or form." — Chad Rickensrud (04:00)

"Everyone thinks [mainframe hacking] is super hard. This is the easiest thing to do." — Phil Young (18:01)

"I was on a filesystem, the permissions were set correctly, the ESM was not, and I had read access to every single file, including private keys." — Chad Rickensrud (32:02)

"We stood on the shoulders of giants. We want other people to stand on our shoulders. This should be free and available." — Phil Young (10:00)

Key Takeaways

• USS is a Unix-like attack surface on z/OS. Standard Linux privilege escalation techniques, credential hunting, and enumeration scripts translate directly with minor adaptation.
• APF authorization is the mainframe equivalent of ring 0. Misconfigured RACF access to BPX.FILEATTR.APF enables an attacker to self-authorize shellcode and execute assembler exploits that grant supervisor state and memory key zero.
• RACF misconfiguration is widespread. KDF-AES encryption disabled, overly permissive profiles, and ESM settings overriding filesystem permissions are all frequently found in real engagements.
• Open-source tooling exists today. enum.rex, OMVS Enum, ZosHog, and PortScan are publicly available and provide a complete post-exploitation framework for z/OS Unix environments.
• Egress filtering on mainframes is routinely absent. Many z/OS systems can connect directly to the internet, enabling data exfiltration without any additional infrastructure.

Slides were not listed as available for this briefing.