Skip to main content
The Model Provider subsystem in Palyra handles the abstraction and normalization of interactions with various Large Language Model (LLM) backends. It provides a uniform interface for chat completions, embeddings, and audio transcription while managing complex requirements like streaming event normalization, tool call repair, and multi-layered configuration.

​
Provider Adapter Architecture

Palyra uses a trait-based adapter system to support multiple LLM providers. The core runtime traits are ModelProvider (for chat and audio) and EmbeddingsProvider crates/palyra-daemon/src/model_provider.rs#5-12.

​
Supported Adapters

​
Streaming and Normalization

Upstream provider events are normalized into a uniform stream of ProviderEvent::ModelToken events crates/palyra-daemon/src/model_provider.rs#14-18. This ensures the Orchestrator can consume a consistent event format regardless of whether the underlying transport is Server-Sent Events (SSE) or a standard HTTP response body.

​
Tool Repair

The subsystem includes a ToolRepairStreamNormalizer that monitors assistant output for malformed tool calls crates/palyra-daemon/src/model_provider.rs#71-77. It can identify and potentially repair truncated or syntactically incorrect JSON in tool arguments before they reach the execution layer.

​
Provider Data Flow

The following diagram illustrates the flow from a high-level request to the specific code entities handling the adapter logic. Model Request Normalization Flow Sources: crates/palyra-daemon/src/model_provider.rs#10-18, crates/palyra-daemon/src/model_provider.rs#39-43, crates/palyra-daemon/src/model_provider.rs#71-77

​
Configuration Pipeline

Palyra employs a multi-layered configuration loading strategy defined in crates/palyra-daemon/src/config/load.rs.

​
Precedence Layers

  1. Secure Defaults: Hardcoded values in DaemonConfig and related structs crates/palyra-daemon/src/config/schema.rs#1-13.
  2. TOML Configuration: Values loaded from palyra.toml crates/palyra-daemon/src/config/load.rs#1-9.
  3. Environment Variables: PALYRA_* prefixed variables that override both defaults and file settings crates/palyra-daemon/src/config/load.rs#4-5.

​
Secret Management and SecretRefs

Sensitive values like API keys are never stored directly in the primary configuration if a SecretRef is provided. The system supports: Configuration Loading Pipeline Sources: crates/palyra-daemon/src/config/load.rs#1-15, crates/palyra-daemon/src/config/load.rs#91-115

​
CLI and Model Management

The palyra models command group provides tools for inspecting and mutating the provider registry.

​
Key Commands

CommandFunctionCode Entity
statusReports effective model/provider configurationModelsStatusPayload
listShows available models and provider capabilitiesModelsListPayload
setMutates the default chat model with automatic backupsModelsMutationPayload
test-connectionProbes provider endpoints with credential validationrun_console_provider_probe
discoverFetches available models directly from provider APIsconsole_models_discover_handler
Sources: crates/palyra-cli/src/commands/models.rs#41-68, crates/palyra-cli/src/commands/models.rs#121-128, crates/palyra-daemon/src/transport/http/handlers/console/models.rs#116-137

​
Connection Probing and Discovery

The palyra-daemon exposes internal handlers for the CLI and Web Console to verify credentials. These probes resolve credentials from the Vault or AuthProfileRegistry and perform a live request to the provider’s /models or /v1/models endpoint crates/palyra-daemon/src/transport/http/handlers/console/models.rs#3-9.

​
Configuration Redaction

To prevent accidental leakage, the SECRET_CONFIG_PATHS constant defines all sensitive paths (e.g., model_provider.openai_api_key) that must be redacted whenever configuration is exported or displayed in the console crates/palyra-common/src/daemon_config_schema.rs#22-38. Sources: crates/palyra-common/src/daemon_config_schema.rs#22-38, crates/palyra-daemon/src/transport/http/handlers/console/models.rs#145-156