Skip to main content
Palyra implements a multi-layered observability and maintenance strategy designed to ensure daemon reliability, facilitate remote troubleshooting, and provide automated remediation for common failure modes. This system aggregates internal state from disparate subsystems (model providers, skills, node runtime, etc.) into a unified diagnostics pipeline and provides a mechanism for generating encrypted, redacted support bundles.

​
Diagnostics Aggregation Pipeline

The diagnostics system is centered around the console_diagnostics_handler crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#6-144, which serves as the primary aggregation point for the Web Console’s Support section. It collects snapshots from various daemon subsystems to build a comprehensive view of the system’s health.

​
Data Flow & Collection

The handler orchestrates data collection from several internal components:
  1. Runtime State: Fetches model provider status and global counters (e.g., denied_requests) crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#11-15.
  2. Auth Runtime: Captures the state of authentication profiles crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#16-20.
  3. Subsystem Payloads: Collects specific diagnostics for Browserd, Skills, Plugins, Hooks, and Webhooks crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#37-41.
  4. Memory & Learning: Aggregates maintenance status, vacuum schedules, and reflection counters crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#54-57.

​
Redaction & Security

To prevent sensitive information leakage, the diagnostics pipeline applies strict redaction. The redact_console_diagnostics_value function crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#28 is called on payloads before they are returned to the console. Additionally, the system utilizes redact_secret_config_values crates/palyra-daemon/src/lib.rs#120 and redact_auth_error crates/palyra-daemon/src/lib.rs#123 to mask credentials and PII.

​
Entity Mapping: Diagnostics Surface

The following diagram maps the HTTP surface to the underlying collection functions. Diagnostics Collection Map Sources: [crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#6-144](http://crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#6-144), [crates/palyra-daemon/src/transport/http/router.rs#134-176](http://crates/palyra-daemon/src/transport/http/router.rs#134-176)

​
Self-Healing & Background Maintenance

The daemon maintains a background loop to manage long-running maintenance tasks and automated recovery.

​
Memory Maintenance

The spawn_scheduler_loop crates/palyra-daemon/src/lib.rs#75 manages the MEMORY_MAINTENANCE_INTERVAL, which triggers the memory_maintenance_status check crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#54-55. This loop handles:

​
Incident Remediation (Doctor)

The “Doctor” subsystem provides structured recovery flows. In the Web Console, the SupportSection apps/web/src/console/sections/SupportSection.tsx#72-107 allows operators to:
  1. Queue Recovery Previews: Generate a plan of action for a degraded system apps/web/src/console/sections/SupportSection.tsx#56.
  2. Apply Recovery: Execute the planned remediation steps apps/web/src/console/sections/SupportSection.tsx#57.
  3. Rollback: Revert to a specific run_id if a deployment or skill update causes instability apps/web/src/console/sections/SupportSection.tsx#58-59.
Sources: [crates/palyra-daemon/src/lib.rs#75](http://crates/palyra-daemon/src/lib.rs#75), [apps/web/src/console/sections/SupportSection.tsx#56-60](http://apps/web/src/console/sections/SupportSection.tsx#56-60), [crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#100-116](http://crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#100-116)

​
Support Bundles

Support bundles are queue-backed jobs that aggregate logs, configuration (redacted), and state snapshots into a portable ZIP archive for troubleshooting.

​
Support Job Lifecycle

Support jobs are managed via the SupportSection and the useSupportDomain hook.

​
Data Composition

A bundle typically includes: Sources: [apps/web/src/console/sections/SupportSection.tsx#97-101](http://apps/web/src/console/sections/SupportSection.tsx#97-101), [apps/web/src/console/__fixtures__/m56ControlPlane.ts#180-205](http://apps/web/src/console/__fixtures__/m56ControlPlane.ts#180-205)

​
Node Runtime & Device Pairing

The NodeRuntimeState crates/palyra-daemon/src/node_runtime.rs#177-182 manages the lifecycle of external nodes (CLI or Desktop) connecting to the daemon.

​
Pairing Handshake

Pairing is a multi-step process involving:
  1. Code Issuance: Generating a DevicePairingCodeRecord (PIN or QR) crates/palyra-daemon/src/node_runtime.rs#48-54.
  2. Approval Workflow: Creating an ApprovalRecord in the journal for the operator to accept the new device crates/palyra-daemon/src/node_rpc.rs#187-200.
  3. mTLS Material Exchange: Upon approval, the node receives a DevicePairingMaterialRecord containing the client certificate and CA crates/palyra-daemon/src/node_runtime.rs#79-87.

​
Runtime Enforcement

The NodeRpcServiceImpl enforces security for connected nodes:

​
Code-to-Entity Mapping: Node RPC

Node Runtime Logic Sources: [crates/palyra-daemon/src/node_rpc.rs#38-54](http://crates/palyra-daemon/src/node_rpc.rs#38-54), [crates/palyra-daemon/src/node_runtime.rs#141-148](http://crates/palyra-daemon/src/node_runtime.rs#141-148), [crates/palyra-daemon/src/node_runtime.rs#166-170](http://crates/palyra-daemon/src/node_runtime.rs#166-170)

​
Summary of Diagnostic Counters

The system tracks several critical metrics for self-healing and monitoring crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#136-141:
CounterDescription
reflections_scheduledTotal background learning tasks queued.
reflections_completedTotal background learning tasks finished.
candidates_createdPotential facts/procedures identified by the learning engine.
candidates_auto_appliedHigh-confidence items automatically written to memory.
denied_requests_totalRequests blocked by rate limiting or policy.
Sources: [crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#136-141](http://crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#136-141), [crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#77-83](http://crates/palyra-daemon/src/transport/http/handlers/console/diagnostics.rs#77-83)