When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation
Joshua Ward (Paper author · University of California Los Angeles), Bochao Gu (Paper author · University of California Los Angeles), Chi-Hua Wang (Paper author · University of California Los Angeles), Guang Cheng (Paper author · University of California Los Angeles)
Privacy Enhancing Technologies Symposium (PETS) 2026 · Day 2 · Moose
Overview
In an era increasingly reliant on data-driven insights, the generation of synthetic tabular data has emerged as a critical technique for privacy preservation and data sharing. This approach involves training generative models on real, sensitive datasets to produce artificial data that mimics the statistical properties of the original, without revealing individual records. While traditional methods have long been employed, recent advancements have seen Large Language Models (LLMs) adopted as state-of-the-art generators for tabular data, often accompanied by claims of inherent privacy benefits. This talk, delivered by Joshua Ward from UCLA and co-authored with Bochao Gu, Chi-Hua Wang, and Guang Cheng, critically examines these claims, unveiling a novel and significant privacy vulnerability specific to LLM-based tabular data generation.

Key moments
- 0:00 Introduction: LLMs generating synthetic tabular data
- 3:30 LLMs generate strings close in similarity, far in feature space
- 4:10 Introducing the Levenshtein distance membership inference attack
- 5:05 Catastrophic privacy leakage found in LLM-generated data
- 6:50 Factors increasing leakage: data quantity and sequence length
- 8:00 Proposed defense: adding noise during LLM sampling
- 9:20 Summary: LLMs are not PETs, new attack, effective defense
When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation
Speakers: Joshua Ward (Paper author, University of California Los Angeles); Bochao Gu (Paper author, University of California Los Angeles); Chi-Hua Wang (Paper author, University of California Los Angeles); Guang Cheng (Paper author, University of California Los Angeles)
Conference: PETS
YouTube: https://www.youtube.com/watch?v=s8WO9qPJJdo
Overview
In an era increasingly reliant on data-driven insights, the generation of synthetic tabular data has emerged as a critical technique for privacy preservation and data sharing. This approach involves training generative models on real, sensitive datasets to produce artificial data that mimics the statistical properties of the original, without revealing individual records. While traditional methods have long been employed, recent advancements have seen Large Language Models (LLMs) adopted as state-of-the-art generators for tabular data, often accompanied by claims of inherent privacy benefits. This talk, delivered by Joshua Ward from UCLA and co-authored with Bochao Gu, Chi-Hua Wang, and Guang Cheng, critically examines these claims, unveiling a novel and significant privacy vulnerability specific to LLM-based tabular data generation.
The core of their research challenges the assumption that LLMs, by virtue of their advanced generative capabilities, inherently protect privacy when handling structured data. Unlike prior generative models that operate in the feature space of data, LLMs process and generate data as strings, token by token. This fundamental difference, as the authors demonstrate, introduces a unique attack surface: string memorization. Their findings reveal that LLMs, when generating synthetic tabular rows, can inadvertently memorize and reproduce patterns from the training data in a way that is easily exploitable through simple string similarity metrics, even if the generated data appears statistically distinct in a numerical feature space. This has profound implications for the secure deployment of LLM-based solutions in sensitive data environments.
The paper, titled "When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation," highlights that this string-level leakage can be catastrophic, leading to near-perfect membership inference attacks (MIA) under highly conservative threat models. The research not only exposes this vulnerability but also proposes an effective defense mechanism that can be integrated into existing LLM sampling pipelines, offering a crucial step towards making LLM-based synthetic data generation truly privacy-preserving. This work serves as a vital warning and a call to action for developers and users of LLMs in tabular data contexts, emphasizing the need for a nuanced understanding of privacy implications beyond traditional statistical measures.
Background
▶ Watch: Introduction: LLMs generating synthetic tabular data (0:00)
The concept of tabular synthetic data generation stems from the need to share and analyze sensitive information, such as medical records or financial transactions, without compromising individual privacy. The goal is to create artificial datasets that statistically resemble the original, allowing researchers and analysts to derive insights and build models as if they were using the real data, but without direct access to sensitive records. For years, this field has leveraged various generative models, each with its own approach to learning the complex joint distribution of features within a table.
Historically, privacy-preserving synthetic data generation has relied on models like graphical models, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and more recently, diffusion models. These methods typically operate by learning the joint distribution across the entire feature vector of a tabular record. For instance, a GAN might generate a complete row of data by sampling from a latent space, ensuring that the generated features collectively represent a plausible record within the learned distribution. The atomic unit of these models is the feature vector itself, meaning they reason about the interdependencies and distributions of numerical and categorical values in their native Cartesian space. Privacy attacks against these traditional models often focus on statistical properties or feature-space distances to infer membership.
The advent of Large Language Models (LLMs) has introduced a paradigm shift. LLMs, renowned for their ability to process and generate human-like text, have been adapted for tabular data generation through two primary regimes: in-context learning (ICL) and supervised fine-tuning (SFT). In both scenarios, the original tabular data is first encoded into a textual format (e.g., a CSV-like string representation of rows). For ICL, this encoded data is provided within the LLM's context window, and the model then samples new rows. For SFT, a smaller LLM is fine-tuned on this textual representation of the tabular data. The fundamental difference here is that LLMs generate data token by token, auto-regressively, treating each row as a string. Once generated, these strings are then parsed back into a structured tabular format. This architectural distinction means that the atomic unit for an LLM is a token (a character, a subword, or a word), not a complete feature vector or its constituent numerical values.
This token-level operation, while powerful for natural language, introduces a novel vulnerability in the context of tabular data. While traditional models explicitly model feature relationships in a continuous or categorical space, LLMs are fundamentally engaged in string manipulation and sequence prediction. The researchers hypothesized that this difference could lead to a unique form of privacy leakage, specifically string memorization, where an LLM might generate synthetic data that is numerically distinct from real data but string-similar, thereby exposing sensitive patterns in ways that traditional feature-space attacks would miss. The problem arises because a small string edit distance can correspond to a large numerical difference, yet the underlying string pattern might still betray information about the original record. This distinction forms the bedrock of their investigation into how LLM tabular generators might inadvertently leak privacy through their string representations.
Key Findings
▶ Watch: Introducing the Levenshtein distance membership inference attack (4:10)
The research uncovered several critical findings that challenge the perceived privacy benefits of LLM-based tabular data generation:
- String Similarity as a New Attack Vector: The most significant discovery is that LLMs, when generating synthetic tabular data, often produce strings that, while potentially "far away" in the traditional numerical feature space (e.g., Euclidean distance), are remarkably "close" in terms of string similarity. For instance, a real record's numerical value like
97540might be synthetically rendered as107540. Although these numbers differ significantly, the Levenshtein distance between their string representations is minimal, indicating a high degree of string-level memorization rather than true statistical generalization.
- Catastrophic Membership Inference Leakage: A simple, brain-dead easy membership inference attack (MIA) based on Levenshtein edit distance proved catastrophically effective. Under the most conservative "no-box" threat model – where the adversary only has access to the released synthetic data and no knowledge of the model, training data, or reference data – this attack frequently achieved perfect classification (AUC = 1) for determining if a candidate record was part of the original training data. This was particularly prevalent in the in-context learning (ICL) regime, where many models exhibited this extreme level of leakage.
- LLM-Specific Vulnerability: The string-based attack does not generalize to traditional generative models. When applied to synthetic data generated by non-LLM models like Variational Autoencoders (VAEs) or GANs, the Levenshtein attack yielded results no better than random chance. This clearly demonstrates that the vulnerability is intrinsic to how LLMs process and generate data as strings, rather than a general flaw in synthetic data generation.
- Factors Exacerbating Leakage: Several factors were identified that contribute to increased privacy leakage:
- High Fidelity: The greater the fidelity of the synthetic data (i.e., how closely it statistically matches the original real data), the more pronounced the string-level leakage. This suggests a trade-off where improved data utility comes at a significant privacy cost in LLMs.
- Increased Synthetic Data Quantity: Generating a larger volume of synthetic data provides more opportunities for the LLM to memorize and reproduce patterns from the training data, leading to a proportional increase in leakage.
- Information Content of Training Data: Records containing longer numerical sequences or more complex categorical motifs in the training data were found to be at greater risk of leakage. LLMs tend to memorize these "motifs" in their string representations.
- LLMs as Stochastic Retrieval Systems: The findings suggest that LLMs, in this context, behave less like models learning a true joint distribution from a statistical perspective and more like stochastic retrieval processes. They appear to retrieve approximate rows from their training knowledge and then introduce some noise, rather than generating entirely novel samples based on an abstract understanding of the data's underlying distribution. This interpretation helps explain the observed string memorization.
- Effective Defense Mechanism: The researchers proposed a novel defense mechanism integrated at the sampling stage of an LLM. By injecting a small amount of noise into the LLM's logit distribution just before sampling, the defense effectively disrupts the memorization of character-level patterns and numeric sequences. This approach successfully reduced Levenshtein attack leakage from the high 60s (for SFT models) down to near-random levels, crucially achieving this without significant degradation in the fidelity or utility of the generated synthetic data. This demonstrates a practical path forward for mitigating the identified vulnerability.
Technical Deep Dive
▶ Watch: Catastrophic privacy leakage found in LLM-generated data (5:05)
The fundamental divergence between traditional and LLM-based tabular data generation lies in their operational units. Traditional models like GANs, VAEs, and diffusion models operate directly on the feature space, learning the joint probability distribution of numerical and categorical attributes. For instance, a VAE might map a row of data (e.g., [age=30, salary=50000, occupation=engineer]) to a latent vector, from which it can then decode new, statistically similar rows. Their internal representations and generation processes are inherently structured around the mathematical properties of the data's features.
In contrast, LLMs, by their very nature, process and generate sequences of tokens. To handle tabular data, a row like [age=30, salary=50000, occupation=engineer] must first be serialized into a string format, such as "30,50000,engineer" or "age:30; salary:50000; occupation:engineer". The LLM then operates on this string, predicting the next token in the sequence until a complete synthetic row string is generated. This string is subsequently parsed back into a structured format. This auto-regressive, token-by-token generation is the root cause of the observed privacy leakage.
The core vulnerability, string memorization, arises because the LLM learns to generate specific character sequences that correspond to data values. Even if a generated value is numerically distinct from a training value, its string representation might be incredibly similar. Consider the example provided: a real record might contain 97540, while the LLM generates 107540. As numerical values, these are significantly different (a difference of 10,000 units). However, as strings, they differ by only a single character (9 vs. 1), making their Levenshtein edit distance very low. The LLM, optimizing for statistical fidelity in the output, might be prone to generating such "off-by-one" or "close-edit-distance" strings because they are easier to produce given its training on sequences.
The membership inference attack (MIA) designed by the researchers exploits this characteristic. The attack operates under a highly conservative no-box threat model, which assumes the adversary has minimal information: only the released synthetic dataset (D_synth) and a candidate record (X_star). The adversary does not have access to the generative model, its training data, or any reference data. The attack procedure is strikingly simple:
- Candidate Record Encoding: The candidate record
X_star(which may or may not be in the original training data) is encoded into its string representation. - Nearest Neighbor Search: For each row
S_iin the synthetic datasetD_synth, its string representation is compared toX_star's string representation using the Levenshtein distance metric. - Minimum Distance Calculation: The attack computes the minimum Levenshtein distance between
X_starand its nearest neighbor inD_synth. - Inference: A low minimum Levenshtein distance suggests that
X_staris likely a member of the training data, as the LLM has "memorized" a string very similar to it. A higher distance suggests it is not.
The Levenshtein distance, or edit distance, quantifies the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one string into another. For example, the Levenshtein distance between "kitten" and "sitting" is 3 (k -> s, e -> i, then insert g). In the context of numerical strings like 120.1 and 221.1, the distance is 1 (changing the first 1 to 2), despite the significant numerical difference. This metric is robust to minor perturbations that might appear in LLM-generated strings, effectively revealing memorized patterns.
The experimental results were stark. For LLMs utilizing in-context learning (ICL), the Levenshtein attack frequently achieved an Area Under the Curve (AUC) of 1, indicating perfect classification of training data membership. Even for supervised fine-tuning (SFT) models, which generally exhibited slightly less leakage, the attack still revealed catastrophic privacy issues, with leakage often in the high 60s (measured by AUC, where 0.5 is random and 1 is perfect). Critically, when the same attack was run against synthetic data generated by non-LLM models (e.g., VAEs, GANs), the AUC scores hovered around 0.5, confirming the attack's specificity to LLM's string-based generation.
Further analysis revealed that the quantity of synthetic data generated directly correlated with leakage; more synthetic data meant more opportunities for the LLM to memorize and reproduce patterns. Similarly, the complexity of numerical sequences and categorical motifs in the training data increased the risk, as LLMs were prone to memorizing these specific string patterns. This behavior paints a picture of LLMs not as sophisticated statistical models learning abstract distributions, but as systems prone to stochastic retrieval, where they recall and slightly perturb training data strings.
Demo / Proof of Concept
▶ Watch: Proposed defense: adding noise during LLM sampling (8:00)
While the talk did not feature a live, interactive "demo" in the traditional sense, the researchers presented compelling empirical evidence and experimental results that serve as a robust proof of concept for both the attack and the proposed defense. Their work involved systematically applying the Levenshtein distance-based membership inference attack across a diverse benchmark of LLM-based tabular generators and datasets.
The proof of concept for the attack involved:
- Dataset Preparation: Utilizing various real-world tabular datasets and encoding them into string representations suitable for LLM training or in-context learning.
- LLM Generation: Generating synthetic data using different LLMs configured for both in-context learning (ICL) and supervised fine-tuning (SFT). These included state-of-the-art models known for their high fidelity in tabular generation.
- Attack Execution: For each generated synthetic dataset, a set of candidate records (some from the original training data, some not) was chosen. The Levenshtein distance attack was then executed as described in the "Technical Deep Dive," computing the minimum string edit distance between each candidate record and the nearest neighbor in the synthetic dataset.
- Performance Evaluation: The effectiveness of the attack was quantified using metrics like Area Under the Curve (AUC), where an AUC of 1 signifies perfect membership inference. The results consistently showed catastrophically high AUC scores, often 1.0 for ICL models, demonstrating the attack's practical viability.
- Comparative Analysis: Crucially, the same attack methodology was applied to synthetic data generated by non-LLM models (e.g., VAEs and GANs). The consistently random performance (AUC ≈ 0.5) of the attack on these traditional models provided strong evidence that the vulnerability is specifically tied to the LLM's string-based generation paradigm.
The proof of concept for the defense mechanism was also demonstrated through empirical experiments:
- Defense Integration: The proposed defense algorithm, which involves injecting a small amount of noise into the LLM's logit distribution during the sampling process, was integrated into the generative pipeline of the LLMs. The talk mentioned its compatibility with popular frameworks like Hugging Face and VLM APIs, highlighting its practical applicability.
- Re-generation and Re-attack: Synthetic data was then re-generated using the LLMs with the defense activated. The Levenshtein attack was subsequently re-applied to this newly generated, defended synthetic data.
- Efficacy Measurement: The results showed a dramatic reduction in privacy leakage. For models where leakage was initially in the high 60s (e.g., SFT models), the defense successfully brought the Levenshtein attack's effectiveness down to near-random levels (AUC ≈ 0.5).
- Utility Preservation: A key aspect of the defense's proof of concept was demonstrating that this reduction in privacy leakage did not come at the expense of data utility. The researchers showed that the fidelity of the synthetic data (how well it statistically matched the real data) remained largely unaffected, indicating that the defense effectively breaks string-level memorization patterns without destroying the implied Cartesian space relationships critical for data analysis.
These experimental results, meticulously detailed and quantified, served as a compelling demonstration that the identified string memorization vulnerability is real, significant, and specifically targets LLM-based generators, and that a practical and effective mitigation strategy exists.
Defensive Implications
▶ Watch: Summary: LLMs are not PETs, new attack, effective defense (9:20)
The findings of this research carry significant defensive implications for anyone involved in generating or using synthetic tabular data, particularly when employing LLMs. The primary takeaway is a stark warning: do not use LLM-based tabular generators in their current unmitigated form for sensitive data. The inherent string memorization vulnerability means that privacy claims for these models, without specific countermeasures, are likely unfounded and dangerous.
Defenders should immediately consider the following actions:
- Re-evaluate LLM-based Generators: Organizations currently using or planning to use LLMs for synthetic tabular data generation must re-evaluate their privacy guarantees. Traditional privacy metrics that focus on feature-space distances or statistical properties may entirely miss the string-level leakage identified in this work. A comprehensive privacy audit should include string-similarity attacks.
- Implement Sampling-Time Noise Injection: The most direct and effective defense proposed is to integrate the described sampling-time noise injection algorithm. This defense, which modifies the LLM's logit distribution during token generation, is designed to disrupt the character-level patterns that lead to string memorization without sacrificing data utility. Its compatibility with standard LLM frameworks like Hugging Face and VLM APIs makes it a practical solution for immediate deployment. This should be considered a mandatory step for any LLM-based synthetic data generation pipeline dealing with sensitive information.
- Understand the Unique Attack Surface: Defenders must recognize that LLMs introduce a new, independent attack surface at the string representation level, which is distinct from attacks targeting the Cartesian or feature space. This requires a shift in mindset and the adoption of new evaluation metrics beyond traditional statistical fidelity. Tools for measuring string similarity (like Levenshtein distance) should be incorporated into privacy evaluations.
- Scrutinize Data Characteristics: Be aware that certain characteristics of the training data can exacerbate leakage. Datasets with longer numerical sequences, complex identifiers, or repetitive categorical motifs are at higher risk. If such data must be used, stronger mitigation strategies or alternative generation methods might be necessary.
- Limit Synthetic Data Quantity: Generating an excessively large volume of synthetic data increases the opportunities for an LLM to memorize and leak training records. Defenders should carefully consider the minimum necessary quantity of synthetic data to achieve their analytical goals, rather than generating arbitrarily large datasets.
- Consider Non-LLM Alternatives: For scenarios where privacy is paramount and the proposed LLM defense cannot be fully trusted, or if the LLM-based approach proves too complex to secure, organizations should consider reverting to or exploring traditional generative models (GANs, VAEs, diffusion models). These models, while having their own privacy challenges, do not suffer from the specific string memormemorization vulnerability highlighted in this research.
- Ongoing Research and Monitoring: The field of LLM privacy is rapidly evolving. Defenders should stay abreast of ongoing research into LLM vulnerabilities and defenses. Regular monitoring and updating of privacy-preserving techniques will be crucial as LLM capabilities and attack methods advance.
In summary, the era of unquestioningly trusting LLMs for privacy-preserving data generation is over. A proactive and technically informed approach is required, focusing on the specific mechanisms of LLM operation and implementing targeted defenses to counter novel attack vectors like string memorization.
Key Takeaways
- LLMs introduce a novel string-level privacy attack vector for tabular data generation: Unlike traditional generative models, LLMs process data as strings, leading to string memorization where synthetic data can be numerically distinct but string-similar to real records, enabling privacy leakage.
- A simple Levenshtein distance attack enables catastrophic membership inference: Under a conservative "no-box" threat model, an attack based on Levenshtein edit distance can achieve near-perfect (AUC=1) membership inference, particularly against LLMs using in-context learning (ICL).
- The vulnerability is specific to LLM-based generation: This string-level attack is highly effective against LLMs but fails against traditional models like VAEs and GANs, confirming that the problem stems from the LLM's token-by-token, string-based generation mechanism.
- Factors like fidelity, data quantity, and data complexity increase leakage: Higher synthetic data fidelity, larger quantities of generated data, and more complex numerical/categorical sequences in the training data all contribute to a greater risk of string memorization and privacy leakage.
- An effective defense involves sampling-time noise injection: A proposed defense mechanism, which injects noise into the LLM's logit distribution during sampling, can significantly reduce leakage (e.g., from high 60s to random for SFT models) without compromising the statistical fidelity or utility of the generated synthetic data.
- Caution is advised for LLM-based tabular data generation: Organizations should critically re-evaluate privacy claims for LLM-based synthetic data solutions and implement the proposed defense or alternative strategies to mitigate the significant risk of string memorization.
About the Speaker(s)
The primary presenter for this talk was Joshua Ward, a paper author from the University of California, Los Angeles (UCLA). At the time of the presentation, Joshua was actively working on his dissertation at UCLA, indicating a deep engagement in research within this field. He collaborated with Bochao Gu, Chi-Hua Wang, and Guang Cheng, also paper authors from the University of California, Los Angeles, on this research. Their collective work highlights a specialized focus on the security and privacy implications of advanced machine learning models, particularly in the context of sensitive data generation and analysis.
Reviews
Dr. Zero (Offensive Security Researcher) — MUST SEE
Everyone slapping LLMs onto sensitive tabular data needs to watch this immediately. Ward and his UCLA team just proved that LLMs aren't generating statistical distributions—they're regurgitating strings, and a brain-dead Levenshtein distance attack completely ruins their privacy guarantees. This is what happens when you treat a stochastic text parrot like a VAE.
Heather Calloway (CISO) — STRONG ACCEPT
If your data science teams are using LLMs to generate synthetic tabular data to bypass privacy constraints, this research proves they are likely leaking raw PII. The authors demonstrate that LLMs memorize and regurgitate data as strings rather than statistical values, breaking the core compliance promise of synthetic datasets.
→ Top-rated talks at Privacy Enhancing Technologies Symposium (PETS) 2026
All talks from Privacy Enhancing Technologies Symposium (PETS) 2026