Multi-Agent AI Collaboration: Orchestrating Specialized AI Roles in Production

Why one agent isn't enough — and how to build a team of AI specialists that research, analyze, synthesize, and critique each other's work. With full observability via Langfuse.

Updated October 2026 with lessons from a conference talk on why one agent isn't enough, and on 4 October 2026 with technical corrections.

Retrieval Was Never the Hard Part

The starting point was straightforward: 23 data sources, 40 specialized tools, one conversational interface. The sources span expression atlases, dependency screens, interaction networks, dose-response curves, ontologies, and literature — including public databases most researchers already use independently: DepMap, STRING, UniProt, PDB/AlphaFold, Open Targets, ChEMBL, GTEx, PubMed, Human Protein Atlas, Reactome, WikiPathways, ClinVar/gnomAD, Cellosaurus, cBioPortal, and NCATS Inxight. The platform work — connecting all of them through a single agent — took substantial effort and is genuinely useful.

But access is not the hard part. Reconciliation is.

When two datasets disagree about whether a gene is essential, retrieval hands you both results and stops. The researcher is still the only thing in the loop capable of resolving it. That ceiling becomes visible with questions like:

• "Compare drug sensitivity across three cancer subtypes and recommend therapeutic targets"
• "What contradicts the current hypothesis about this pathway?"
• "Synthesize findings from five datasets and identify gaps in the evidence"

Each one requires holding competing interpretations simultaneously, weighing them, and being explicit about what would change the answer. That is not a retrieval task. It is a reasoning task with an adversarial component.

Why Single-Agent Systems Hit a Ceiling

When this system encountered those questions, two structural failure modes became clear — and neither was fixable by switching to a more capable model or adding more tools.

Capability dilution. One agent holding 40 tools becomes decent at everything and excellent at nothing. Tool selection becomes a tax on every question — the model spends reasoning budget deciding what to call rather than doing the work well. As the tool surface grows, the decision cost grows with it.

Self-consistency bias. An agent cannot meaningfully critique its own output. It has a single objective and already satisfied it. Asking "are you sure?" invites the same reasoning path to re-run and agree with itself. In scientific synthesis this is disqualifying: an unreviewed conclusion is a hypothesis, not a finding, and the reviewer cannot be the author.

The Multi-Agent Architecture

The system evolved into a four-role collaboration pattern, each role with a distinct objective and a deliberately different incentive structure. Not every query goes through the full pipeline — a lightweight classifier runs first and routes straightforward data lookups to a faster single-agent path, reserving the four-role pipeline for synthesis and reasoning tasks.

Why Different Models for Different Roles

This tiered approach means the majority of tokens are processed by the cheaper Sonnet model (scout + analyst), while frontier-tier reasoning is reserved for the two steps that need it most. Per-role tracing in Langfuse makes this cost breakdown visible and lets you verify that each tier is actually earning its place.

One Question, Four Contributions

To make the architecture concrete, consider a question that a single agent handles poorly: "Across these cancer subtypes, which dependency is the most tractable therapeutic target — and what argues against it?"

The second half — "and what argues against it?" — is what makes this a multi-agent problem. A single agent can answer the first half. Only an adversarial architecture reliably answers the second, because something has to be incentivized to look for the counter-argument.

The Orchestration Pattern

# Simplified multi-agent orchestration
from langfuse import observe
from pydantic import BaseModel
from strands import Agent

class CritiqueVerdict(BaseModel):
    needs_revision: bool
    issues: list[str]

class ResearchReport(BaseModel):
    text: str
    caveats: list[str]

class ResearchOrchestrator:
    def _new_agents(self):
        """Fresh instances per query - agents carry conversational state"""
        scout = Agent(model="sonnet", tools=SCOUT_TOOLS, system_prompt=SCOUT_PROMPT)
        analyst = Agent(model="sonnet", tools=ANALYSIS_TOOLS, system_prompt=ANALYST_PROMPT)
        # Synthesizer: a small set of read tools; writes are blocked until review is done
        synthesizer = Agent(model="opus", tools=READ_TOOLS, system_prompt=SYNTH_PROMPT)
        # Critic: independent lookups only (knowledge graph, literature, skills)
        critic = Agent(model="opus", tools=[KG_LOOKUP, LITERATURE_SEARCH, SKILL_LOOKUP],
                       system_prompt=CRITIC_PROMPT)
        return scout, analyst, synthesizer, critic

    @observe(name="multi-agent-research")
    async def research(self, query: str) -> ResearchReport:
        scout, analyst, synthesizer, critic = self._new_agents()

        # Phase 1: Data gathering — one Scout call that receives the user's question
        # (Strands Agent.__call__ is synchronous; use invoke_async() for await)
        scout_results = await scout.invoke_async(query)

        # Phase 2: Analysis (sequential — needs scout output)
        analysis = await analyst.invoke_async(
            f"Analyze these findings:\n{scout_results}"
        )

        # Phase 3: Synthesis
        report = await synthesizer.invoke_async(
            f"Synthesize into a coherent report:\n{analysis}"
        )

        # Phase 4: Adversarial critique — on the raw retrieved data, not just the report
        # On tiers that reject forced tool choice (the 5.5 models on Bedrock), ask for a
        # prose verdict and parse it instead — see post 31
        critique = await critic.invoke_async(
            f"Find flaws in this report:\n{report}\n\nRaw retrieved data:\n{scout_results}",
            structured_output_model=CritiqueVerdict,
        )

        # Phase 5: Revision (if critique found issues); any writes run only after review
        if critique.structured_output.needs_revision:
            report = await synthesizer.invoke_async(
                f"Revise based on critique:\n{report}\n\nCritique:\n{critique}"
            )

        # Researchers get the Critic's caveats with the report, not just the text
        return ResearchReport(text=str(report), caveats=critique.structured_output.issues)

Observability with Langfuse: You Can't Optimize What You Can't Measure

# Langfuse integration pattern (SDK v4)
from langfuse import observe, get_client, propagate_attributes

@observe(name="multi-agent-research")
async def research(query: str):
    langfuse = get_client()

    # Trace attributes propagate to every span created inside this block
    with propagate_attributes(
        metadata={"query_type": classify_query(query)},
        tags=["multi-agent", "research"],
    ):
        @observe(name="scout-phase")
        async def scout_phase():
            return await run_scout(query)

        @observe(name="analyst-phase")
        async def analyst_phase(data):
            return await analyst(data)

        # Each phase gets its own Langfuse span
        scout_data = await scout_phase()
        analysis = await analyst_phase(scout_data)
        # ... synthesis and critique traced similarly

    # Score the output for quality monitoring
    langfuse.score_current_trace(
        name="user-satisfaction",
        value=1.0,  # Updated later via user feedback webhook
    )

What Langfuse Revealed That Surprised Me

1. The Critic saved money. Counter-intuitively, adding an expensive Opus critic reduced total cost because it caught errors that would have required users to re-run queries. Fewer retries = less total spend.
2. Per-role token tracking revealed waste. The Analyst was receiving the Scout's full output but using only a fraction of it — something only per-role token attribution shows; in aggregate numbers, this inefficiency is invisible.

The Critic Pattern: Why Adversarial AI Matters

The most valuable addition was the Critic agent. Its prompt is explicitly adversarial:

CRITIC_PROMPT = """You are a rigorous scientific reviewer. Your job is to Find Flaws.

For each claim in the report:
1. Is it supported by the evidence provided?
2. Are there alternative interpretations the report ignores?
3. Is the confidence appropriate, or overstated?
4. What critical data is Missing that would change the conclusion?

Be critical. A report that passes your review should be publishable."""

There is a non-obvious design decision here: the Critic receives the raw retrieved data, not only the Synthesizer's report. A critic that reads just the report can check internal consistency — it cannot check whether the report is actually grounded in the evidence the Scout retrieved. Building that ground-truth link explicitly is what makes this a review rather than a consistency check.

In practice, the Critic triggers a revision loop in roughly one research-tier query in four (about 23%). A representative case: a Synthesizer once proposed a target ranked first on dependency evidence, reported with high confidence. The Critic objected that the supporting effect was carried by a small number of lines from a single cell-line panel, and that a second source in the Scout's own retrieval showed the effect failing to replicate. The revised report demoted the candidate, surfaced the replication conflict explicitly, and named the experiment that would settle it.

The 77% of queries where the Critic does not trigger a revision still get its verdict — usually caveats pinned to specific claims rather than a clean pass — and researchers notice and value those caveats.

Contradiction Is Signal, Not Noise

A system built for efficiency converges. It reconciles sources into the consensus view and delivers it faster. That is genuinely useful — and it is structurally incapable of producing a novel hypothesis, because the mainstream answer is the objective.

The Critic inverts this. Once a role is explicitly rewarded for finding fault — and one of its four tasks is to surface refuting evidence — disagreement between sources stops being an error to smooth over and becomes the most interesting output of the run.

Three things this surfaces that retrieval alone cannot:

• Replication conflicts — where a dependency holds in one experimental panel and fails in another. A consensus engine averages these away. An adversarial architecture preserves them.
• Confidence asymmetry — findings that look equally supported but rest on very different evidential weight. One might come from 1,100 cell lines; another from 60. The Analyst quantifies this; the Critic flags it when the report treats them as equivalent.
• Structured absence — the experiment nobody has run. The Critic's fourth question ("what critical data is missing?") turns a gap in the evidence into a testable proposal. That is the closest thing to a hypothesis generator this system has.

Persisting disagreements matters as much as surfacing them. A conflict found in one query that gets written to the knowledge graph is queryable by every future session that touches the same target. The system builds memory of what is contested, not just what is known.

Research Memory Across Sessions

A single-agent system answers questions and forgets them. The finding from Monday's query has no bearing on Thursday's query about the same target unless someone explicitly re-introduces the context. Multi-agent systems create the same problem unless there is a shared memory layer that carries knowledge forward across sessions.

Two layers address this. A knowledge graph accumulates structured relationships — gene essentiality across cancer types, pathway memberships, compound-target associations — queryable by any future agent call. Contradictions are first-class: a disagreement between two sources is stored as a conflict node, not averaged away, so the system knows what is contested rather than only what is known. Amazon Bedrock AgentCore Memory handles conversation context across sessions: both the fast single-agent path and the full multi-agent pipeline read prior context on session entry, so the agent knows what the researcher explored before, what evidence was contested, and what questions were left open.

The combined effect is that the system becomes more useful the longer it is used. Each query enriches both layers. This is not the property of a retrieval tool — it is the property of a research partner.

For a deeper look at how to build cross-session memory for agents, see Cross-Session Agent Memory. For the failure mode where agent memory begins lying to itself, see When Your Agent's Memory Starts Lying.

Deployment: Each Role is a Separate Service

The multi-agent system isn't just logical separation — it's physical separation on Kubernetes. Each role runs as its own Deployment with its own resource profile:

# deployment-data-scout.yaml (excerpt)
spec:
  replicas: 2              # High throughput, parallel queries
  template:
    spec:
      containers:
        - resources:
            requests:
              memory: "1Gi"    # Heaviest role — holds the raw tool results
              cpu: "250m"

# deployment-synthesizer.yaml (excerpt)
spec:
  replicas: 2              # Sequential within a query; replicas serve concurrent users
  template:
    spec:
      containers:
        - resources:
            requests:
              memory: "512Mi"  # Inference runs on Bedrock, not in the pod
              cpu: "500m"

This lets you scale each role independently. Data Scouts scale out for throughput. The Synthesizer is sequential within a query but runs two replicas for concurrent users; size its concurrency limit to your throughput target and expect 429s beyond it. The Critic can be temporarily scaled to 0 in low-priority environments to save cost, but requires a feature flag in the orchestrator to skip the stage — scaling to 0 without one causes connection failures.

One operational trap: create a fresh instance of every role per query. Agents carry conversational state: reuse one and the previous query's conversation leaks into the next, and current Strands releases refuse a second concurrent call on the same instance outright. This is not obvious until you hit it in production.

Lessons Learned

1. Don't start with multi-agent

Run a single agent first, and split only once you can name the failure mode. Splitting is justified only when you can point at where thinking harder stopped helping and thinking differently would. Before that point you are buying latency, cost, and operational complexity with no demonstrated return.

2. Give roles opposing incentives

Four agents with the same prompt and the same goal is one agent with four times the bill. The value comes entirely from objectives that genuinely conflict: breadth versus precision, synthesis versus refutation. If the incentives do not conflict, the architecture is waste.

3. Resolution is harder than retrieval

Getting a model to resolve contradictions rather than list them took the most iteration of anything in this system. What worked: require an explicit confidence score per finding, then weight the synthesis by those scores. Without that instruction, the model produces a tidy summary of the disagreement and leaves the decision to the human — which defeats the purpose.

4. Make every role earn its place

Per-role cost, latency, and revision-rate data is what tells you whether a role is contributing or just adding time. We keep four roles because four survived that test — not because four looked right on an architecture diagram. If a role is not reducing revision rate or catching errors, remove it.

From Answering Questions to Proposing Experiments

The next step for this class of system is not a bigger chatbot. It is a governed, traceable research partner that can reason across evidence, critique uncertainty, and suggest testable next steps.

Three properties define what that looks like in practice:

• Evidence-driven reasoning: agents retrieve, compare, and connect signals from multiple sources instead of returning a single answer. Competing interpretations reach the table, not just the consensus view.
• Built-in scientific critique: a Critic role challenges assumptions, flags contradictions, and makes uncertainty visible before conclusions are accepted — not after, when they are harder to retract.
• Experiment-oriented output: the system moves from summarizing evidence toward proposing hypotheses and follow-up analyses for human review. The Critic's fourth question — "what critical data is missing?" — is the first step toward that.

If you're building multi-agent systems or have questions about orchestrating AI roles in production, feel free to connect.

Disclaimer: The views and opinions expressed in this post are my own and do not represent those of my employer or any organisation I am associated with.