CryptPEFT: Efficient and Private Neural Network Inference via Parameter-Efficient Fine-Tuning

CryptPEFT: Efficient and Private Neural Network Inference via Parameter-Efficient Fine-Tuning

Speakers: Saisai Xia

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=0DfCFPspOKs

Overview

Saisai Xia presents CryptPEFT, a system that dramatically accelerates private neural network inference by redesigning parameter-efficient fine-tuning (PEFT) architectures specifically for encrypted computation. The core insight is that in a PEFT model, the large backbone is public and can run in plaintext on the user's device, while only the small adapter layers contain proprietary model IP. By enforcing one-way communication -- data flows from backbone to adapter but never back -- CryptPEFT ensures that only the tiny adapter component needs to operate on encrypted data. This approach achieves up to 238.76x speedup over traditional PEFT private inference and 20.85x speedup over simple fine-tuning, while maintaining or improving model accuracy. The work combines a novel adapter architecture with a Neural Architecture Search (NAS) framework tailored for MPC-friendly operations.

Background

▶ Watch: Why private inference matters: protecting both user data and model IP (0:15)

Private neural network inference addresses a fundamental trust problem: users want to use cloud AI models without revealing their sensitive data (medical records, financial information), while model providers want to protect their trained parameters, architecture, and training data from theft. Three main techniques exist: Trusted Execution Environments (TEEs), Secure Multi-Party Computation (MPC), and Homomorphic Encryption (HE). In practice, HE remains extremely expensive, and neural networks have grown dramatically while privacy-preserving computation has not kept pace.

Parameter-Efficient Fine-Tuning (PEFT) offers a potential solution by introducing small trainable adapter layers while keeping the large backbone frozen. Adapters like LoRA and Adapter-Former add minimal parameters for downstream task specialization. However, existing PEFT designs create a fundamental conflict with private inference: once an adapter processes encrypted data, its output feeds back into the backbone, forcing all subsequent backbone layers to also operate on ciphertext -- destroying the efficiency advantage.

Key Findings

▶ Watch: The fundamental conflict: two-way communication forces backbone encryption (4:00)

Fundamental structural conflict identified: Traditional PEFT's two-way communication (adapter output feeds back to backbone) forces the entire downstream computation into encrypted space, negating the efficiency benefits of having a small adapter.

One-way communication as the key design principle: By ensuring data flows only from backbone to adapter (never back), CryptPEFT keeps the backbone in plaintext and limits encrypted computation to the small adapter. This is the single most impactful design decision in the system.

Adapter placement matters: Inserting adapters in deeper backbone layers consistently yields higher model utility than placing them in earlier layers. This means fewer adapters can be used (reducing encrypted computation) without sacrificing accuracy.

Attention is essential but must be MPC-friendly: Traditional adapters without attention mechanisms cannot capture token interactions under one-way communication, leading to accuracy drops. CryptPEFT replaces expensive softmax operations with learnable linear modules and GaLU activations with ReLU for MPC compatibility.

Performance results: The utility-first configuration outperforms standard PEFT by 1.45% on average accuracy. The efficiency-first configuration surpasses simple fine-tuning by 0.8% while reducing trainable parameters by 88.81%. Private inference latency improves by 20.85x over fine-tuning and 238.76x over traditional PEFT.

Technical Deep Dive

▶ Watch: CryptPEFT workflow: offline NAS search and online private inference (6:00)

CryptPEFT's workflow has two phases. In the offline phase, the service provider uses Neural Architecture Search to discover an optimal adapter design within MPC-friendly constraints, then fine-tunes the adapter with private data. The search space is constrained to operations that are efficient in MPC: ReLU instead of GaLU, linear attention instead of softmax, and LoRA for parameter reduction.

In the online phase, the user runs the public backbone locally in plaintext, encrypts intermediate activations using MPC, and sends them to the provider. Both parties jointly execute private inference on the adapter only. The provider returns encrypted results, and the user decrypts the output.

The NAS framework is specifically tailored to CryptPEFT with three components: a dedicated search space restricted to MPC-friendly operations, a new search strategy that balances accuracy and encrypted computation cost, and a latency model that predicts MPC overhead for candidate architectures. The system is implemented on CrypTen and evaluated in both WAN and LAN network settings.

The key architectural choices for the adapter include: replacing softmax with learnable linear modules for attention, using ReLU instead of GaLU for MLP layers, applying LoRA to reduce parameter count, and restricting adapter placement to deeper backbone layers. These constraints are automatically optimized through NAS to find the best accuracy-latency tradeoff.

Demo / Proof of Concept

▶ Watch: MPC-friendly design: ReLU replaces GaLU, linear attention replaces softmax (8:00)

CryptPEFT was evaluated against two baselines (standard PEFT methods and simple fine-tuning) across multiple tasks. Compared to MPC-ViT (which also optimizes architecture for private inference), CryptPEFT significantly outperformed in both model utility and inference efficiency. The evaluation system was built on CrypTen with two network configurations: wide area network (WAN) and local area network (LAN), demonstrating practical applicability in realistic deployment scenarios.

Defensive Implications

▶ Watch: Results: 238x speedup over PEFT and 20x over fine-tuning (11:00)

CryptPEFT moves private inference closer to practical deployment for real-world AI applications. For organizations that need to use third-party AI models on sensitive data (healthcare diagnostics, financial risk assessment, legal document analysis), this work reduces the performance overhead of privacy-preserving computation from prohibitive to potentially acceptable. The 238x speedup over traditional PEFT private inference could make the difference between a system that takes hours and one that responds in seconds.

The broader implication is that privacy-preserving machine learning does not require solving the general problem of efficient homomorphic encryption or MPC. By exploiting the specific structure of modern AI models (public backbones with small private adapters), significant efficiency gains are possible through architectural redesign rather than cryptographic breakthroughs.

Key Takeaways

• Traditional PEFT creates a fundamental conflict with private inference: adapter outputs flowing back to the backbone force all subsequent computation into ciphertext
• One-way communication (backbone-to-adapter only) is the key insight that enables efficient private inference, limiting encrypted computation to the small adapter
• CryptPEFT achieves 238.76x speedup over traditional PEFT private inference while maintaining competitive accuracy
• MPC-friendly replacements (ReLU for GaLU, linear attention for softmax) enable efficient encrypted computation without significant accuracy loss
• Neural Architecture Search tailored for MPC constraints automates the design of optimal adapters for private inference

About the Speaker(s)

Saisai Xia presented the work on behalf of the paper's authors, who were unable to attend. The research explores the intersection of parameter-efficient fine-tuning and privacy-preserving computation, with the goal of making private neural network inference practical for real-world deployment.