Background Queue & Task Orchestration
Thebackground_queue provides a durable execution mechanism for tasks that should not block the main request-response cycle. It is managed by a dedicated loop that polls the JournalStore for pending OrchestratorBackgroundTaskRecord entries.
Task Lifecycle
- Scheduling: Tasks are created via
create_orchestrator_background_taskcrates/palyra-daemon/src/application/learning.rs#90-120. - Polling: The
spawn_background_queue_loopcrates/palyra-daemon/src/background_queue.rs#31-44 runs every 3 seconds (defined byBACKGROUND_QUEUE_IDLE_SLEEPcrates/palyra-daemon/src/background_queue.rs#28) to fetch tasks. - Execution: Tasks are dispatched based on their
task_kind. For example,post_run_reflectiontriggers the AI learning pipeline crates/palyra-daemon/src/background_queue.rs#12-26. - Heartbeating: During execution, the queue records a
WorkHeartbeatUpdatecrates/palyra-daemon/src/background_queue.rs#108-112 to signal to the Self-Healing system that progress is being made.
Data Flow: Background Task Dispatch
| Step | Entity | Action | Code Reference |
|---|---|---|---|
| 1 | GatewayRuntimeState | list_orchestrator_background_tasks | crates/palyra-daemon/src/background_queue.rs#51-60 |
| 2 | BackgroundQueue | process_background_task | crates/palyra-daemon/src/background_queue.rs#95-101 |
| 3 | SelfHealing | record_self_healing_heartbeat | crates/palyra-daemon/src/background_queue.rs#108-112 |
| 4 | JournalStore | update_orchestrator_background_task | crates/palyra-daemon/src/background_queue.rs#71-89 |
Self-Healing & Diagnostics
The Self-Healing system monitors the health of daemon operations and automatically attempts remediation when incidents are detected. It relies on heartbeats from active work units (Runs, Background Tasks, Channel Workers).Implementation Entities
WorkHeartbeatRecord: Tracks the last known activity of a specific object (e.g., a Run ID) crates/palyra-daemon/src/gateway/runtime.rs#28-33.RuntimeIncidentRecord: Represents a detected failure or timeout in a specificIncidentDomaincrates/palyra-daemon/src/gateway/runtime.rs#28-33.RemediationAttempt: A logged action taken by the system to resolve an incident crates/palyra-daemon/src/gateway/runtime.rs#28-33.
The Doctor System
The “Doctor” system provides diagnostic checks and automated recovery. It is exposed via the CLI and the Console Operations section.- Diagnostics: Monitors
self_healing_active_incidentsandself_healing_heartbeat_countcrates/palyra-cli/src/commands/status.rs#51-53. - Recovery: If a background task or run hangs (missing heartbeats), the self-healing loop can transition them to a
failedorcancelledstate to unblock resources crates/palyra-daemon/src/background_queue.rs#102-106.
Node Runtime & Device Pairing
TheNodeRuntimeState manages the lifecycle of external nodes (CLI, Desktop, or other Daemons) connecting to the central gateway. This includes the secure mTLS-backed pairing flow.
Pairing Flow (mTLS)
Palyra uses a multi-step pairing process to establish trust:- Initiation: A node requests pairing via
PairingMethod(Pin or QR) crates/palyra-daemon/src/node_runtime.rs#24-45. - Approval: The daemon creates an
ApprovalPromptRecordfor the operator to review in the Console crates/palyra-daemon/src/node_rpc.rs#187-200. - Verification: Upon approval, the
VerifiedPairingmaterial is generated, and the node receives a client certificate crates/palyra-daemon/src/node_runtime.rs#106-121. - Enforcement: Subsequent gRPC calls require the certificate fingerprint to match the
device_idcrates/palyra-daemon/src/node_rpc.rs#99-129.
Node RPC Entity Mapping
The following diagram bridges the logical pairing process to the internal code structures. Diagram: Node Pairing & RPC Verification Sources: crates/palyra-daemon/src/node_runtime.rs#106-156, crates/palyra-daemon/src/node_rpc.rs#99-129Learning & Reflection Loop
The Learning subsystem is a specialized background service that analyzes completed runs to extract “durable facts,” “preferences,” and “procedures.”Post-Run Reflection
When a run completes,finalize_run_stream_after_provider_response crates/palyra-daemon/src/application/run_stream/orchestration.rs#107-113 triggers the learning pipeline.
- Sampling: The system checks
LearningRuntimeConfigto see if the run should be analyzed based onsampling_percentcrates/palyra-daemon/src/application/learning.rs#47-64. - Task Creation: A background task of kind
post_run_reflectionis queued crates/palyra-daemon/src/application/learning.rs#30-46. - Candidate Generation: The task processes the run transcript to generate
LearningCandidateRecordentries for facts or procedures crates/palyra-daemon/src/application/learning.rs#158-185. - Auto-Application: If a candidate exceeds the
durable_fact_auto_write_threshold_bps, it is automatically committed to the memory store crates/palyra-daemon/src/gateway/runtime.rs#85-96.
Learning Configuration
| Parameter | Description | Code Reference |
|---|---|---|
sampling_percent | Percentage of runs analyzed | crates/palyra-daemon/src/gateway/runtime.rs#87 |
cooldown_ms | Minimum time between reflections | crates/palyra-daemon/src/gateway/runtime.rs#88 |
max_candidates | Max candidates per run | crates/palyra-daemon/src/gateway/runtime.rs#90 |
System Health & Maintenance
The daemon performs periodic maintenance on theMemoryStore and Journal via the memory_maintenance_status crates/palyra-daemon/src/transport/http/handlers/console/memory.rs#11-20.
Maintenance Tasks
- Embeddings Backfill: Ensures all memory items have vector embeddings for semantic search crates/palyra-daemon/src/transport/http/handlers/console/memory.rs#143-150.
- Retention Enforcement: Vacuums old records based on
retention_max_bytesorretention_ttl_dayscrates/palyra-daemon/src/gateway/runtime.rs#72-82. - Self-Healing Cleanup: Clears stale heartbeats for completed work crates/palyra-daemon/src/application/run_stream/cancellation.rs#33-34.