LLM Providers

heddle supports multiple LLM providers out of the box. Switch providers by changing a few lines in your spec. No code changes needed.

An LLM config appears as llm_config on an Agent or an LlmNode. All supported providers speak the OpenAI chat completions API.

Every config needs a name and a model_id. These four are the ones heddle implements, and all of them speak the OpenAI chat-completions API underneath.

Any other component_type is rejected at the first model call, unless a loaded plugin provides it. A plugin can supply a custom llm_config type that answers model calls itself, which is how a spec reaches an endpoint heddle ships no client for. It may not claim one of the four above: a flow writing OpenAiConfig reaches heddle's own client whatever is loaded. See Plugins.

OpenAI

llm_config:
  component_type: OpenAiConfig
  name: openai
  model_id: gpt-4o

Set your API key in the environment:

export OPENAI_API_KEY=sk-...

Or reference an environment variable from the spec:

llm_config:
  component_type: OpenAiConfig
  name: openai
  model_id: gpt-4o
  api_key: $OPENAI_API_KEY

OpenAI-compatible

For any endpoint that implements the OpenAI API format:

llm_config:
  component_type: OpenAiCompatibleConfig
  name: my-provider
  model_id: my-model
  url: https://my-endpoint.com/v1
  api_key: $MY_API_KEY

vLLM

For self-hosted models via vLLM:

llm_config:
  component_type: VllmConfig
  name: llama
  model_id: meta-llama/Llama-3.1-8B
  url: http://localhost:8000/v1
  api_key: $VLLM_API_KEY

Ollama

For local models via Ollama:

llm_config:
  component_type: OllamaConfig
  name: ollama
  model_id: llama3.1
  url: http://localhost:11434/v1
  api_key: $OLLAMA_API_KEY

url is required; there is no default host.

API keys

API keys can reference environment variables using the $ prefix:

api_key: $MY_API_KEY

heddle resolves these from your shell environment at the first model call, not at compile time, and fails with a clear error if the variable is unset. A value without a $ prefix is used literally. Because resolution is deferred, heddle validate never reaches a credential: a spec naming $OPENAI_API_KEY validates with nothing in the environment.

heddle run looks ahead of that. Before the run starts it reads every variable the spec's api_key fields name and asks for the ones this shell does not have:

OPENAI_API_KEY is not set, and this flow reads it from the environment.
An answer is used for this run only — it is not echoed, and it is not written anywhere.
Press enter to leave one unset.

  OPENAI_API_KEY:

What you type is used by that process and nothing else: it is not echoed, and nothing is written to disk. Export the variable to stop being asked. Pressing enter leaves it unset and the run goes ahead — a variable named on a branch the run never takes is one the run never needs, and whatever is skipped fails where it always did.

The question needs a terminal on both stdin and stderr, so a pipeline, a CI job, a server or a --protocol stream is never asked and fails at the first model call exactly as before. --no-ask-env turns it off on a terminal too.

export OLLAMA_API_KEY=local