Model Tiering: Matching LLM Cost to Task Complexity in Multi-Agent Systems

What it is

A single multi-agent pipeline rarely needs uniform model capability at every step. Some agents mostly call tools and lightly structure the results; others must synthesize several prior agents' output into a genuine decision. Model tiering matches LLM choice to that difficulty gradient — cheap/fast models where the task is mechanical, expensive/high-reasoning models where it isn't — rather than defaulting to "the best model everywhere," which is simpler but wastes cost on tasks that don't need it.

How ProspectAI implements it

ProspectAI's config/agents.yaml assigns a default model per agent. Data-gathering agents (Market and Technical Analyst) get Haiku; the reasoning-heavy agents (Investor Strategic, Critic) get Sonnet:

# Model strategy:
#   Agents 1-3 (data gathering): claude-haiku-4-5-20251001  -- fast, cost-efficient
#   Agents 4-6 (strategy/critic): claude-sonnet-4-6         -- highest reasoning quality

  market_analyst:
    llm:
      provider: "anthropic"
      model: "claude-haiku-4-5-20251001"
      api_key: null
      base_url: null

  investor_strategic:
    llm:
      provider: "anthropic"
      model: "claude-sonnet-4-6"
      api_key: null
      base_url: null

  critic:
    llm:
      provider: "anthropic"
      model: "claude-sonnet-4-6"
      api_key: null
      base_url: null

Worth being precise about: the header comment states agents 1–3 (Market, Technical, and Fundamental Analyst) all use Haiku, but the live config currently pins fundamental_analyst to claude-sonnet-4-6 — a real discrepancy between the doc comment and the actual configuration at time of writing, not a simplification for this page. It illustrates a genuine risk of this pattern: per-agent model assignment is easy to drift from its own documentation if nothing enforces the mapping.

The per-agent default isn't hardcoded into the assignment logic, though — it's resolved through a layered override chain. config/config.py lets any individual agent's model be overridden via environment variable in production, without touching code or YAML, falling back to a global default if no override is set:

AGENT_MODEL_ENV_KEYS: Dict[str, str] = {
    "market_analyst":      "AGENT_MARKET_ANALYST_MODEL",
    "technical_analyst":   "AGENT_TECHNICAL_ANALYST_MODEL",
    "fundamental_analyst": "AGENT_FUNDAMENTAL_ANALYST_MODEL",
    "investor_strategic":  "AGENT_INVESTOR_STRATEGIC_MODEL",
    "critic":              "AGENT_CRITIC_MODEL",
}

def model_id_for_agent(self, agent_key: str) -> Optional[str]:
    """Resolve model for one agent. AGENT_*_MODEL overrides global MODEL."""
    env_key = AGENT_MODEL_ENV_KEYS.get(agent_key)
    if env_key:
        per = os.getenv(env_key)
        if per and per.strip():
            return _strip_optional_provider_prefix(per)

    return self.default_model_id()

agents/base_agent.py is the single place this resolution actually gets wired into a live CrewAI LLM instance — every agent calls the same _get_llm(), so the tiering logic lives in exactly one place, not duplicated per agent:

def _get_llm(self):
    """Resolve LLM from MODEL_PROVIDER + AGENT_*_MODEL/MODEL (yaml llm.model is fallback)."""
    provider = os.getenv("MODEL_PROVIDER", "anthropic")

    resolved = self.config.model_id_for_agent(self.agent_key)
    model = resolved or self.llm_model
    if provider == "ollama":
        return LLM(
            model=f"ollama/{model}",
            base_url=self.llm_base_url or self.config.OLLAMA_BASE_URL,
            temperature=self.temperature,
            max_tokens=self.max_tokens,
        )

    api_key = self.llm_api_key if self.llm_api_key else self.config.ANTHROPIC_API_KEY
    return make_caching_llm(
        model=f"anthropic/{model}",
        api_key=api_key,
        temperature=self.temperature,
        max_tokens=self.max_tokens or 10000,
    )

Why it matters

Model tiering is a cost/quality optimization that only works if the mapping from agent to model is centralized and operationally tunable — otherwise it either drifts (as the fundamental_analyst discrepancy above shows) or becomes a hardcoded assumption nobody can safely change in production. The env-var override layer means someone running this pipeline can shift an individual agent to a different model tier — say, testing whether the Critic (see adversarial critic) needs its current reasoning tier or could run cheaper — without a code change or redeploy.