Beyond Jailbreak: Unveiling Risks in LLM Applications Arising from Blurred Capability Boundaries

Beyond Jailbreak: Unveiling Risks in LLM Applications Arising from Blurred Capability Boundaries

Speakers: Yunyi Zhang (presenting on behalf of the authors)

Conference: NDSS Symposium 2026

YouTube: https://www.youtube.com/watch?v=fNwI2RTgduM

Overview

This talk presents a comprehensive evaluation of security risks in LLM-based applications that arise not from traditional jailbreaking but from poorly defined capability boundaries. While most security research focuses on bypassing LLM safety guardrails, this work examines the gap between what application developers intend their apps to do and what the underlying models actually allow. The researchers analyzed 800,000 applications across four major platforms, collected 10,000 public application prompts, and identified three distinct risk categories: capability downgrade (degrading expected performance), capability upgrade (expanding beyond intended scope), and capability jailbreak (bypassing all constraints).

The findings are striking: 144 applications were vulnerable to capability upgrade, with each affected application able to perform more than 15 task types beyond its intended scope. 178 applications were vulnerable to capability jailbreak, and 17 applications executed malicious tasks even without any harmful prompt. About half of all applications included no functional constraints at all, and among those with constraints, 20% scored below 60 on constraint quality. The work demonstrates that prompt design quality is the critical factor in mitigating capability boundary risks.

Background

▶ Watch: Are we truly ready for AI? LLM capability risks (0:00)

The rapid development of LLMs has enabled a new application paradigm where developers use system prompts to define application behavior rather than traditional hard-coded logic. Unlike Android or iOS applications where functionality is explicitly programmed, LLM application developers are "configurers" who set rules to ensure the model executes intended tasks while avoiding unintended behaviors. If these rules are flawed or incomplete, the consequences can be unpredictable and potentially harmful.

The Red Note incident is cited as evidence that LLM-related risks have moved from theoretical or lab-based environments to real-world threats. Every newly released model, regardless of its safety training, is compromised by jailbreaks shortly after launch, highlighting the persistent gap between safety aspirations and practical security.

Traditional jailbreaking focuses on the left boundary of the capability space -- where capabilities cross safety thresholds. This work focuses on the right boundary -- where capabilities remain technically under the developer's control but are inadequately constrained. The developers' prompts are the primary (and often only) security mechanism, making prompt quality a first-order security concern.

The evaluation spans four major platforms, including GPTs (OpenAI's custom GPT platform) and Agent Builder, representing the current ecosystem of consumer-facing LLM applications.

Key Findings

▶ Watch: Evaluation framework and prompt quality scoring methodology (4:00)

Ecosystem dominated by low-quality applications: "Super developers" -- prolific publishers who create large numbers of applications -- dominate the ecosystem. Their applications are frequently generated from templates with minimal prompt customization, saturating platforms with low-quality offerings. About half of applications score below 50 on overall prompt quality metrics.

Widespread absence of functional constraints: About half of applications include no functional constraints whatsoever in their prompts. Among those that do include constraints, 20% score below 60 on constraint quality, meaning limitations are incomplete or ineffective. Most developers lack awareness of capability boundary security.

Capability downgrade vulnerability is universal: Testing six open-source LLMs with 2,000 boundary test cases that inject misleading information, all models were affected. Mistral (MRO) was the most vulnerable, producing incorrect responses in 993 out of 2,000 cases. Applications built on these models without input standardization and output validation remain highly vulnerable.

Capability upgrade is pervasive: 144 applications were vulnerable to capability upgrade, each executing more than 15 task types beyond their intended scope. GPTs were notably more vulnerable due to default configurations that include web search and image generation capabilities, expanding the attack surface when constraints are weak.

Platform default plugins introduce risk: Default plugins provided by platforms (e.g., Baidu Maps on Agent Builder for a medical consultation app) create unintended capability expansion that developers may not account for.

Prompt optimization measurably reduces risk: Applications using optimized prompts reduced out-of-scope task execution by 5% to 80%. One Agent Builder application initially executed 15 out of 21 different task types beyond its purpose; after prompt optimization, this dropped to just three.

Technical Deep Dive

▶ Watch: Capability downgrade: hidden and redundant information attacks (6:00)

The evaluation framework consists of three modules:

LLM Application Collection Module: First, application metadata is collected across four platforms -- application names, descriptions, user visit counts, and 10,000 public system prompts. Applications are classified using LLM-based categorization. Second, prompt quality is evaluated along four dimensions:

• Target: More detailed task descriptions indicate higher quality. Measured via information entropy and key feature extraction.
• Process: Prompts with guided step-by-step instructions better direct applications toward successful task completion. Measured by step count and logical complexity.
• Capability: Assessed by the level of detail in describing intended functions.
• Constraint: Assessed by the completeness and effectiveness of functional limitations.

Capability Downgrade Testing: Two types of modified inputs are used: (1) Hidden information -- data invisible to humans but detectable by LLMs, such as invisible Unicode characters inserted into text inputs, and (2) Redundant information -- irrelevant content injected into standard inputs (off-topic words, unrelated facts) that doesn't alter the core task but may mislead the model. This tests whether the application produces robust outputs when faced with manipulated inputs.

Capability Upgrade Testing: Three types of test cases are constructed: (1) Default test cases -- the application's own example queries representing its intended capability space, (2) Generated out-of-scope cases -- inputs generated for each application category that test boundary conditions, and (3) Common sense test cases -- a set of 10 general knowledge questions (e.g., "How many seasons are in a year?") to assess general reasoning capabilities beyond the intended scope.

Capability Jailbreak Testing: Standard jailbreaking methods are applied using both harmful queries and adversarial queries. The distinction from traditional jailbreaking is that capability boundary jailbreaking exploits the gap between developer-set constraints and base model capabilities, rather than attacking the base model's safety alignment directly.

The cross-platform analysis revealed that while total application counts vary significantly across platforms, the distribution of application types is remarkably consistent -- the mean absolute deviation of category percentages is less than 2%, indicating shared user demand patterns across ecosystems.

Demo / Proof of Concept

▶ Watch: Capability downgrade results: all LLMs vulnerable, Mistral worst (10:00)

No live demo was performed. The presentation relied on statistical results from the large-scale evaluation. Key visualizations included heat maps of capability upgrade vulnerability across four platforms (showing darker colors for applications executing more out-of-scope task types), distribution charts of malicious task execution across platforms, and bar charts showing the band-model distribution of constraint scores. The most concrete example involved an Agent Builder medical consultation application configured with the Baidu Maps plugin by default -- a capability mismatch that illustrates how platform defaults can create unintended attack surface.

Defensive Implications

▶ Watch: GPTs most vulnerable to capability upgrade due to default tools (12:00)

The research has direct and actionable implications for multiple stakeholders:

For application developers: Prompt design is the primary security control for LLM applications. The findings that half of applications lack any functional constraints and that prompt optimization reduces out-of-scope execution by up to 80% provide clear motivation for investment in prompt engineering as a security practice. Developers should treat system prompts as security-critical configuration.

For platform operators: Default plugin configurations can introduce unintended capability expansion. Platforms should audit default configurations for each application category and provide developers with constraint templates or automated prompt quality scoring tools. The "super developer" phenomenon suggests that template-based application creation needs additional quality gates.

For security teams: Traditional application security testing frameworks need adaptation for LLM applications. Capability boundary testing -- not just jailbreaking -- should be part of the security review process. Input standardization and output validation layers should be deployed as defense-in-depth measures regardless of prompt quality.

For regulators: The consistent distribution of application types across platforms suggests that regulatory frameworks could be standardized across ecosystems. The finding that 17 applications executed malicious tasks without any adversarial prompt raises questions about platform liability and application review processes.

Key Takeaways

• LLM application security risks extend far beyond jailbreaking to include capability downgrade, upgrade, and boundary violations caused by poor prompt design
• Analysis of 800,000 applications across four platforms revealed that about half include no functional constraints in their prompts
• 144 applications were vulnerable to capability upgrade (executing 15+ unintended task types), and 178 were vulnerable to capability jailbreak
• GPTs were notably more vulnerable due to default web search and image generation capabilities expanding the attack surface
• Prompt optimization reduced out-of-scope task execution by 5% to 80%, making prompt quality the single most impactful security control
• All six tested open-source LLMs were vulnerable to capability downgrade through input manipulation, with Mistral being most affected (993/2000 cases)
• The LLM application ecosystem is dominated by low-quality, template-generated applications from "super developers"

About the Speaker(s)

Yunyi Zhang from China University presented this work on behalf of the original authors, who were unable to attend due to visa issues. The research team conducted one of the largest-scale evaluations of LLM application security to date, analyzing 800,000 applications and 10,000 prompts across four major platforms. The work bridges the gap between LLM safety alignment research and practical application security assessment.