Ad-Hoc Agent Coordination

The work

Overview

Most multi-agent systems assume agents are trained together and share a joint policy. That assumption fails immediately in open environments: a warehouse where a new robot joins an existing fleet, a rescue scenario where agents from different organisations must cooperate, or any system where teammates change at runtime. Ad-hoc teamwork research asks whether an agent can infer what its teammates are doing and coordinate with them in real time, from scratch, with no prior shared context. The agent in this project maintains live internal models of itself, each teammate, and each opponent — tracking position, stamina, orientation, ball possession, goal-shot angles, and proximity relationships — and on each game step must decide whether to shoot, pass, or dribble, and to whom. That decision runs through two parallel reasoning channels: a logic planning layer handling what is legally and causally possible, and a neural inference layer handling what is behaviourally likely. An offline RL pipeline then improves the strategic policy from accumulated game logs without requiring live environment interaction. The fundamental difficulty is that the agent has no ground truth about its teammates. It cannot ask them what they will do. It must infer their behaviour from observation, plan around that inferred model, and update continuously as new evidence arrives — all within a single game tick.

Architecture

How it's built

Fig. 003.1 · Each tick, live state feeds parallel symbolic (ASP) and neural reasoning into a single decision; the BREMEN offline-RL pipeline refines the policy from logged play.

Detail

Key features

• Logic planning via SPARC (Answer Set Programming): the live game state is serialised each tick into formal logic predicates — causal laws describing how actions change state, executability conditions constraining which actions are legal, inertia axioms preserving state unless explicitly changed, and a planning goal encoding the target position. These predicates are passed to a SPARC solver at runtime, which returns a valid action plan satisfying all constraints. This layer handles the structured, rule-governed aspects of the game — legality, causality, and goal-directed planning — running at game speed on continuously updated live state.
• Neural behaviour inference (PyTorch): a behaviour cloning network trained on recorded state sequences learns to predict each teammate's action tendencies from observation alone. This feeds directly into the coordination logic — if the agent can model what a teammate is likely to do next, it can pass to where that teammate will be rather than where they are now. The network is the agent's substitute for communication: it replaces explicit signalling with learned predictive inference.
• BREMEN offline RL pipeline — three stages: (1) behaviour cloning pre-trains the policy by imitating recorded gameplay, establishing a safe behavioural baseline; (2) an ensemble of three independent world models is trained on logged (state, action) transition pairs, each learning to predict state deltas; (3) the policy is refined through imaginary rollouts sampled from the ensemble, optimised using Trust Region Policy Optimisation (TRPO) with a KL-divergence constraint that prevents the policy from straying too far from the behaviour-cloned baseline. No live environment interaction is required during training.
• Ensemble uncertainty estimation: because three independent world models are trained separately, their disagreement on any given state-action pair signals low data coverage in that region of state space. The policy optimisation remains conservative in these low-confidence regions, avoiding extrapolations the models cannot reliably support — the same principled approach to distribution shift used in modern offline RL for robotics deployments where live data collection is costly or dangerous.
• Live internal agent models updated every game tick for all teammates and opponents: position, stamina, orientation, ball possession, goal-shot angles, and proximity relationships. These models are the foundation of both reasoning channels — the ASP layer uses them to define legal action contexts, and the neural layer uses their history to generate behavioural predictions.
• Utility layer (1,600+ lines): state discretisation, safe-passing corridor geometry, ASP term generation, forward simulation of candidate actions across multiple game steps, and the full data pipeline feeding both the PyTorch and ASP components. This layer bridges the symbolic and connectionist halves of the architecture — translating continuous game state into discrete ASP predicates and structured tensors simultaneously.

Imagery

In context

Results

Impact

• 30% performance lift over baselines in ad-hoc agent benchmarks — measured against teams with no prior coordination history and no shared policy
• ASP solver running at game speed on live state, fully integrated with the neural inference pipeline in the same per-tick decision loop — a non-trivial symbolic + connectionist integration that remains an open research question
• BREMEN offline RL validated in a multi-agent coordination setting, applying the same technique class used in modern robotics research where live experience is expensive, dangerous, or impossible to collect at scale
• Architecture reflects genuine engagement with the research literature: ad-hoc teamwork, offline RL, world models, and neuro-symbolic integration are active research frontiers, not established engineering practice
• Game logs from live simulation runs and full source code available on GitHub

Open source

How to contribute

This is research code and a full dissertation — built to be read and reproduced more than productionised. To dig in:

• Read the dissertation writeup alongside the source for the full method
• Replay the committed game logs to inspect the agent's per-tick decisions
• Retrain the BREMEN pipeline (BC → world-model ensemble → TRPO) on your own logs
• Open an issue to discuss the neuro-symbolic integration