Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination

Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination

Speakers: Kunal Jha, Wilka Carvalho, Yancheng Liang, Simon Du, Max Kleiman-Weiner, Natasha Jaques

Conference: ICML 2025

YouTube: https://slideslive.com/39043987

Overview

This talk, presented by Kunal Jha at ICML 2025, introduces Cross-Environment Cooperation (CEC), a novel paradigm designed to tackle the formidable challenge of zero-shot multi-agent coordination. The core problem addressed is the inability of current multi-agent reinforcement learning (MARL) systems to robustly collaborate with unfamiliar partners on tasks they have never encountered, especially when the operational environment itself is new. While AI has made significant strides in single-agent and zero-sum multi-agent scenarios, achieving general collaboration—where AI can work flexibly with many people across diverse, unseen problems—remains a critical bottleneck for deploying truly intelligent agents in real-world applications like robotics, healthcare, or personal assistance.

Jha and his collaborators propose CEC as a solution that moves beyond the limitations of traditional self-play and population-based training methods. These prior approaches often lead to agents that either converge on a single, brittle strategy or incur prohibitive retraining costs when environments change. CEC, in contrast, trains agents using self-play across a vast diversity of procedurally generated environments, fostering the development of general cooperative norms that enable robust coordination with new partners and in novel settings. The research demonstrates CEC's superior performance in both quantitative metrics and qualitative human-AI interaction, highlighting its potential to unlock more general and adaptable AI collaborators.

The significance of this work extends to the practical realization of general AI. By enabling agents to learn cooperative strategies that transfer across diverse contexts and partners, CEC lays foundational groundwork for building AI systems that can seamlessly integrate into dynamic human environments. This is crucial for applications demanding flexibility, adaptability, and effective collaboration with humans or other AI entities on unforeseen tasks, marking a significant step towards the promise of truly general and helpful AI.

Background

▶ Watch: Introduction and challenges of multi-agent collaboration (0:00)

The aspiration for AI capable of general collaboration, akin to a robot butler assisting in various household tasks, is a long-standing dream. However, current cutting-edge research in multi-agent reinforcement learning (MARL) has often been constrained by narrow problem definitions. A significant portion of this research, for instance, has focused on the game Overcooked, a collaborative cooking simulator. Despite its apparent simplicity, building agents that can robustly interact even in one of these kitchen environments has proven challenging for the community.

One popular method that has emerged for training MARL agents is self-play, where an AI learns by interacting with a copy of itself. Self-play has famously enabled superhuman performance in zero-sum games like Go and Chess, where agents optimize against an adversarial opponent. However, in fully collaborative or cooperative environments, self-play performs poorly. The fundamental issue is that during training, agents converge on a single optimal strategy or "convention" for solving a task (e.g., "always leave a plate on the right side of the stove"). When such an agent encounters a partner following an equally valid but different convention (e.g., "leave a plate on the left side"), this behavior appears massively out-of-distribution, causing the agents to break down and fail to adapt. This highlights a critical flaw: self-play's training-time performance is not indicative of its success in zero-shot coordination, which requires agents to cooperate effectively with partners they have never seen before. To quantify this, the concept of cross-play is introduced, measuring an agent's reward when interacting with an unfamiliar partner.

To address the limitations of naive self-play, prior work introduced population-based training. This approach attempts to simulate many possible equilibria by first generating diverse partner strategies and then teaching agents a best response to each of them. Population-based methods have shown improvements in enabling AIs to collaborate with new partners on the same task they were trained on. However, this method also faces a significant hurdle: if the environment changes even slightly (e.g., a goal location shifts), these agents completely break down. They require billions of steps to cover every possible partner strategy within a single environment, and the necessity to retrain every time the environment changes makes this approach computationally intensive and unscalable for real-world applications involving diverse people and dynamic settings. The core problem, therefore, is the need for AI agents that can generalize cooperative behaviors not just to new partners, but also to entirely new environments, without requiring extensive retraining.

Key Findings

▶ Watch: Limitations of population-based training methods (4:00)

The research introduces Cross-Environment Cooperation (CEC) as a powerful paradigm for training social AI, yielding several key findings that significantly advance the state of multi-agent coordination:

• Superior Zero-Shot Coordination: CEC agents massively outperform naive self-play and population-based approaches in their ability to cooperate with new partners on problems they have never seen before. This was demonstrated across various settings, from simple dual-destination games to complex Overcooked environments, achieving near-maximum rewards where baselines failed.
• Scalability through Procedural Generation: The use of a procedural generator to create billions of diverse, solvable coordination tasks, combined with an end-to-end training pipeline in JAX, enabled training at "10 million steps per minute on a single GPU." This academic-compute-budget, industry-level scale was crucial for exploring the benefits of environment diversity.
• Specialization vs. Generalization Trade-off: A variant, CEC fine-tune (pre-training with CEC then fine-tuning on a specific handcrafted environment), showed excellent cross-play performance on the environment it was fine-tuned on. However, its performance dropped significantly when evaluated on other unseen environments. This highlights a critical trade-off between specializing for a particular environment and maintaining broad generalization capabilities, and points to challenges like catastrophic forgetting in sequential learning settings.
• Robustness to Ad-Hoc Partners: An empirical game theory analysis using replicator dynamics confirmed that CEC and CEC fine-tune are robust to cooperating with ad-hoc partners. This analysis, which simulated how a population of agents would select a mixture of training algorithms to maximize cross-play, revealed that the gradient flows towards CEC methods, indicating that the learned strategies are general cooperative norms rather than just a collapse to a single, brittle convention.
• Enhanced Human-AI Cooperation: In human-AI experiments using the Nice-WebRL package, CEC agents significantly outperformed naive self-play and population-based methods. While CEC agents achieved competitive reward, slightly lower than state-of-the-art methods that had seen the evaluation environment, they received significantly higher qualitative ratings from 80 human participants across metrics like adaptiveness, least frustration, and best at coordination (on a seven-point Likert scale).
• Emergence of General Cooperative Norms: The qualitative human ratings, coupled with specific behavioral observations (e.g., CEC agents colliding less often with people), suggest that CEC's training process fosters the development of general cooperative norms. These norms facilitate transfer across many people and problems, making the AI appear more human-like and enjoyable to work with, even if prioritizing adaptation might sometimes lead to sub-optimal reward in highly specific scenarios with suboptimal human partners.

Technical Deep Dive

▶ Watch: CEC achieves effective cooperation in new environments (5:50)

The core innovation of this work lies in the Cross-Environment Cooperation (CEC) paradigm. Unlike prior methods that focus on either learning a single optimal strategy (self-play) or enumerating partner strategies within a single environment (population-based training), CEC trains agents in self-play across many diverse problems. The fundamental hypothesis is that by forcing agents to adapt to a constantly changing environment during self-play, they will learn more generalizable cooperative behaviors rather than environment-specific conventions.

The technical implementation of CEC involves several key components:

1. Procedural Environment Generation: To enable training across "many diverse problems," the researchers designed a procedural generator capable of creating "billions of solvable coordination tasks" for the Overcooked environment. This generator randomizes initial goal and agent locations within grids, effectively creating a continuous stream of novel cooperative challenges. This is a critical departure from relying solely on a small set of handcrafted environments, which limits generalization. For simpler tasks like the dual-destination environment, this involved randomizing grid configurations, initial agent positions, and goal locations.

1. Self-Play Across Diverse Environments: Agents are trained using self-play, meaning an agent learns to interact with a copy of itself. However, instead of training in a single, static environment, this self-play process is conducted across the procedurally generated environments. This forces the agents to develop strategies that are robust to environmental variations, as a convention learned for one grid might not be optimal or even applicable in another. The goal is not to find a single optimal strategy for a fixed environment, but rather to discover general principles of cooperation that hold across varying contexts.

1. JAX-based Training Pipeline: To achieve the necessary scale for training across billions of environments, the entire training pipeline was implemented end-to-end in JAX. This high-performance numerical computation library, known for its automatic differentiation and JIT compilation capabilities, allowed for training at an impressive "10 million steps per minute on a single GPU." This efficiency was crucial for making the extensive environment diversity feasible within an academic compute budget.

1. CEC Fine-Tune: A two-stage training procedure was also explored, termed CEC fine-tune. This involves first pre-training agents using the standard CEC approach on many procedurally generated grids. Following this, the agents are fine-tuned for a small amount of experience using self-play on a single, specific handcrafted Overcooked environment. This approach investigates the trade-off between broad generalization and specialization for a particular target environment.

1. Empirical Game Theory Analysis: To rigorously evaluate whether CEC agents truly learn adaptive behaviors or merely collapse onto a single, shared convention (which would artificially inflate cross-play scores), the researchers employed empirical game theory analysis. This involved constructing a large "cross-algorithm cross-play payoff matrix" by having every trained agent from every learning algorithm attempt to cooperate with every other agent. Then, replicator dynamics were applied in a "meta-game" to identify the mixture of training algorithms that would maximize the population's overall cross-play score. The observation that the gradient flows towards CEC and CEC fine-tune in this analysis indicates that these methods produce robust, generalizable cooperative strategies, not just a brittle, shared convention.

1. Human-AI Experimentation with Nice-WebRL: To assess real-world applicability and human perception, the team extended Nice-WebRL, a package designed for human-AI experiments, to support arbitrary JAX-based reinforcement learning environments. This tool significantly reduced the time required to set up and run complex human-computer interaction studies, enabling data collection from 80 participants interacting with the trained models in the two hardest Overcooked environments. This allowed for both quantitative reward measurement and qualitative human feedback, providing a holistic evaluation of the agents' cooperative capabilities.

Experimental Setup & Results

▶ Watch: Scaling CEC using procedural Overcooked environments (6:40)

The experimental evaluation of Cross-Environment Cooperation (CEC) was conducted across multiple settings, ranging from simple synthetic tasks to complex, procedurally generated versions of the Overcooked game.

Environments:

• Dual Destination Environment: A simple cooperative game where red and blue agents are rewarded for moving to opposing green squares. This served as an initial testbed to visually demonstrate the breakdown of self-play and the success of CEC in zero-shot coordination.
• Overcooked Environments: The primary domain for evaluation. This included:
• Five handcrafted, standard Overcooked environments used in prior MARL research.
• Billions of procedurally generated Overcooked-like grids, created by the custom generator, with randomized initial goal and agent locations. These were crucial for CEC training and for evaluating generalization to truly unseen environments. Specific examples mentioned were "Coordination Ring" and "Cramped Room," which involve simple rotational policies.

Baselines:

• Naive Self-Play: Agents trained to cooperate with copies of themselves within a single environment.
• Population-Based Training: Agents trained to learn best responses to a diverse population of partners within a single environment.
• E3T (Emergent Equilibria through Exploration and Exploitation for Transfer): A state-of-the-art zero-shot coordination algorithm. Notably, E3T was allowed to see the evaluation environment during its training, providing a strong benchmark for reward, though not for true zero-shot environment generalization.

Hardware and Software:

• The entire training pipeline was implemented in JAX, leveraging its efficiency for high-throughput training.
• Training was performed on a single GPU, achieving "10 million steps per minute," demonstrating significant computational efficiency for academic-level compute budgets.
• Human-AI experiments utilized Nice-WebRL, extended to support JAX-based RL environments, facilitating efficient crowd-sourced data collection.

Metrics:

• Cross-Play Success: The primary quantitative metric, measuring the reward an agent receives when interacting with a partner it has never seen before, either in a known or an entirely new environment.
• Human Qualitative Ratings: Collected from 80 participants using a seven-point Likert scale across seven different metrics (e.g., adaptiveness, frustration, coordination, helpfulness, intelligence).
• Behavioral Analysis: Observations such as collision frequency.

Headline Results:

• CEC vs. Baselines (Zero-Shot Environment & Partner): In the dual-destination environment, CEC agents effectively learned to move to opposing squares and achieved near-maximum reward even when paired with new partners in unseen environments. Quantitatively, CEC massively outperformed naive self-play and population-based methods in cross-play success across both simple and Overcooked settings. While baselines received the same amount of training compute, they failed to achieve any cooperative success when evaluated on problems they hadn't seen.
• CEC Fine-tune Performance: When pre-trained with CEC and then fine-tuned on a specific handcrafted Overcooked environment, CEC fine-tune significantly boosted agents' ability to cooperate with new partners on that fine-tuned grid. However, when evaluated on other unseen environments (i.e., generalizing beyond its fine-tuned domain), its cross-play performance experienced a significant drop. This indicated a clear trade-off between specialization and generalization.
• Empirical Game Theory Results: The replicator dynamics analysis consistently showed a gradient flowing towards CEC and CEC fine-tune, both for the five original Overcooked environments and a hundred procedurally generated ones. This demonstrated the robustness of CEC's learned strategies to ad-hoc partners and confirmed that the observed cooperation was not due to a collapse onto a single, fragile convention.
• Human-AI Cooperation Results:
• Reward: CEC agents significantly outperformed naive self-play and population-based methods in human-AI reward, despite never having seen the evaluation environment during training. They achieved slightly lower rewards than E3T, which had the advantage of seeing the evaluation environment.
• Qualitative Ratings: CEC and CEC fine-tune agents significantly outperformed all other baselines in human qualitative ratings. They were rated most adaptive, least frustrating, and best at coordination, suggesting that the general cooperative norms learned by CEC are highly valued by human partners.
• Behavioral Observations: CEC agents were observed to collide less often with people compared to other models, providing a concrete piece of evidence for an emergent cooperative norm.

Ablations and Further Observations:

• The learning curves showed that not all strategies are learned at the same rate. Simpler, more generalizable strategies (like rotation in "Coordination Ring" and "Cramped Room") tend to emerge first.
• The observed performance drop of CEC fine-tune when generalizing to new environments, after specializing on one, indicated that catastrophic forgetting (a known issue in sequential deep RL) is still a challenge, similar to observations in domain randomization work with environments like SMAC.

Practical Implications

▶ Watch: Specialization vs. generalization trade-offs with fine-tuning (8:00)

The introduction of Cross-Environment Cooperation (CEC) carries profound practical implications for the development and deployment of intelligent agents, particularly in domains requiring flexible and robust collaboration.

For Practitioners and Infra Teams:

• Scalable Generalization: CEC offers a viable pathway to build AI agents that can generalize cooperative behaviors to unseen partners and entirely new environments without the need for extensive retraining. This directly addresses the scaling problem faced by population-based methods, which become prohibitively expensive when environments are dynamic or diverse. For infrastructure teams, this means potentially simpler, more cost-effective deployment workflows for multi-agent systems, as fewer specialized models would be needed for different environments.
• Robustness in Dynamic Settings: In real-world applications such as robotics, smart manufacturing, or personal assistants, the operational environment is rarely static or fully known beforehand. CEC's ability to learn general cooperative norms makes agents significantly more robust to unexpected changes and novel situations, reducing deployment risks and increasing reliability.
• Efficient Training: The JAX-based implementation, achieving "10 million steps per minute on a single GPU," demonstrates that this paradigm can be realized with academic-level compute budgets, making it accessible for broader research and development efforts.
• Human-AI Teaming: The most impactful practical implication might be for human-AI interaction. CEC agents are rated as more adaptive, less frustrating, and better at coordination by human partners. This is crucial for user adoption and trust in collaborative AI systems. Agents that are perceived as "human-like" and helpful are far more likely to be integrated successfully into workplaces and daily life. The emergence of general cooperative norms, such as collision avoidance, directly translates into smoother, more intuitive interactions.

For Model Builders and Deployers:

• Shift in Training Philosophy: Model builders should consider incorporating procedural generation and environment diversity into their training regimes for collaborative AI, moving away from single-environment optimization. This encourages the development of more fundamental cooperative principles rather than environment-specific heuristics.
• Trade-offs and Limitations: Deployers must be aware of the inherent trade-off between specialization and generalization. While CEC provides broad adaptability, fine-tuning (e.g., CEC fine-tune) might still be necessary for achieving peak performance in highly specific, known environments. However, this specialization comes at the cost of reduced generalization to other unseen environments. This suggests a potential two-stage deployment strategy: a general CEC agent for broad utility, with optional fine-tuning for critical, high-stakes environments.
• Addressing Catastrophic Forgetting: The observation of catastrophic forgetting in fine-tuned agents highlights a continuing challenge. Future model designs and training protocols need to incorporate mechanisms to mitigate this, ensuring that specialized learning does not erase broadly applicable cooperative knowledge.
• Curriculum Design: The finding that simpler cooperative norms emerge first suggests that intelligent curriculum design could further improve sample efficiency and the robustness of learned behaviors. This could involve gradually increasing environment complexity or introducing specific cooperative challenges over time.

In summary, CEC provides a compelling framework for developing truly general-purpose collaborative AI. Its ability to foster adaptable, human-friendly agents that generalize across diverse partners and unseen environments represents a significant step towards realizing the promise of AI that can genuinely improve the quality of our lives.

Key Takeaways

• Limitations of Prior Methods: Traditional multi-agent reinforcement learning approaches like self-play and population-based training struggle with zero-shot multi-agent coordination due to converging on brittle, environment-specific conventions or incurring prohibitive retraining costs for new environments.
• Cross-Environment Cooperation (CEC) for Generalization: CEC trains agents using self-play across a vast diversity of procedurally generated environments. This paradigm enables agents to learn general cooperative norms that allow them to robustly coordinate with new partners in entirely unseen problems.
• Scalability and Efficiency: The use of procedural generation, combined with an end-to-end JAX pipeline, allows for training at "10 million steps per minute on a single GPU," making this approach scalable and computationally efficient for academic research.
• Emergence of Cooperative Norms: CEC agents not only outperform baselines in quantitative cross-play metrics but also receive significantly higher qualitative ratings from human partners (e.g., more adaptive, less frustrating). This suggests the emergence of general cooperative norms, such as collision avoidance, which facilitate effective human-AI interaction.
• Specialization vs. Generalization Trade-off: While CEC offers broad generalization, specializing agents through fine-tuning (CEC fine-tune) on a specific environment improves performance on that particular environment but can lead to a drop in generalization to other novel environments, highlighting the challenge of catastrophic forgetting.
• Future Directions: Further research is needed to better combine partner and environment diversity, develop new algorithms and architectures for continuous learning in dynamic multi-agent worlds, and design more effective curricula to improve sample efficiency.

About the Speaker(s)

The primary speaker for this talk was Kunal Jha, who presented the work on Cross-Environment Cooperation. While his specific affiliation beyond the research is not detailed in the transcript, his presentation clearly demonstrates a deep expertise in multi-agent reinforcement learning and the challenges of zero-shot coordination. He is joined in this research by collaborators Wilka Carvalho, Yancheng Liang, Simon Du, Max Kleiman-Weiner, and Natasha Jaques. Notably, Wilka Carvalho is credited as the "brainchild" behind Nice-WebRL, a package extended in this work to facilitate human-AI experiments on JAX-based reinforcement learning environments. The collaborative nature of this research team is evident in the breadth and depth of the work presented, spanning theoretical advancements, practical implementation, and human-centered evaluation.