System Overview
The governance and observability architecture is centered around thepalyrad daemon’s ability to track every token, tool call, and system event.
| Component | Role | Code Entity |
|---|---|---|
| Usage Governance | Enforces budget policies and smart routing. | usage_governance.rs |
| Diagnostics | Aggregates system state for troubleshooting. | diagnostics.rs |
| Observability State | Tracks real-time counters and failure classes. | observability.rs |
| Support Bundles | Generates redacted system snapshots. | support_bundle.rs |
Code-to-Entity Mapping (Governance Flow)
The following diagram illustrates how high-level usage concepts map to specific backend implementation structures. Usage Decision Pipeline Sources: crates/palyra-daemon/src/usage_governance.rs#8-21, crates/palyra-daemon/src/transport/http/handlers/console/usage.rs#11-18Usage Governance & Budget Policies
Usage Governance is responsible for ensuring that agent runs stay within defined financial and operational boundaries. It evaluates every request against a set ofUsageBudgetPolicyRecord entries to determine if a run should proceed, be throttled, or require an manual override.
Key features include:
- Hard/Soft Limits: Automated enforcement of token and cost caps crates/palyra-daemon/src/usage_governance.rs#92-109.
- Smart Routing: Selecting models based on complexity scores and provider health crates/palyra-daemon/src/usage_governance.rs#112-130.
- Override Flows: A human-in-the-loop mechanism where users can request budget increases via the console crates/palyra-daemon/src/usage_governance.rs#15-16.
Diagnostics, Self-Healing & Support Bundles
The diagnostics system provides a unified view of the daemon’s internal state, covering everything from memory maintenance to model provider connectivity. Diagnostics Aggregation Map Sources: crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#6-66, crates/palyra-daemon/src/transport/http/handlers/console/support_bundle.rs#1-30Key Observability Components
- Diagnostics Pipeline: Aggregates snapshots from all major subsystems (Skills, Plugins, Webhooks, Memory) into a single JSON payload crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#73-144.
- Self-Healing: Background loops that monitor for common failure classes and attempt automated remediation, such as refreshing stale auth profiles or clearing circuit breakers.
- Support Bundles: Generates a portable archive of logs and diagnostics with automatic redaction of sensitive keys (API keys, secrets) using
redact_console_diagnostics_valuecrates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#28-35.
Real-time Telemetry & Costs
The system tracks usage across several dimensions to provide immediate insights into the “Model Mix” and operational costs.Metric Tracking
TheUsageSection in the Web Console visualizes data derived from the UsageSummaryEnvelope, which includes:
- Total Tokens: Aggregated prompt and completion counts crates/palyra-daemon/src/transport/http/handlers/console/usage.rs#162-168.
- Estimated Cost: Calculated using
PricingEstimatelogic that accounts for specific provider pricing tiers crates/palyra-daemon/src/usage_governance.rs#69-89. - Latency Posture: Average response times across successful runs apps/web/src/console/sections/UsageSection.tsx#116-120.
Model Mix Telemetry
The daemon maintains records of which models are being used and their relative performance.UsageModelMixRecord: Tracks runs and tokens per model/provider pair crates/palyra-daemon/src/usage_governance.rs#169-178.UsageToolMixRecord: Monitors which tools are most frequently proposed by agents crates/palyra-daemon/src/usage_governance.rs#180-183.