System Overview
Distributed execution in Palyra is managed by theWorkerFleetManager in the palyra-workerd crate, which maintains an in-memory ledger of available workers and their current lifecycle states crates/palyra-workerd/src/lib.rs#3-6. Workers join the fleet by presenting a WorkerAttestation containing identity, integrity, and compatibility claims crates/palyra-workerd/src/lib.rs#30-35.
The daemon selects between local executors (Tier B/C) and networked workers based on the tool’s required security posture and the WorkerFleetPolicy.
Distributed Execution Flow
The following diagram illustrates the lifecycle of a networked worker from registration to task execution. Worker Lifecycle and Execution Flow Sources: crates/palyra-workerd/src/lib.rs#3-6, crates/palyra-daemon/src/node_runtime.rs#1-9, crates/palyra-egress-proxy/src/lib.rs#3-6Worker Fleet Management
TheWorkerFleetManager is the central authority for networked worker coordination. It tracks worker health via heartbeats and manages the lifecycle states (e.g., Available, Leased, Quarantined, Orphaned) crates/palyra-workerd/src/lib.rs#10-11.
Key components include:
- WorkerAttestation: A structure carrying SHA-256 digests of the worker image, build, and artifacts to ensure code integrity crates/palyra-workerd/src/lib.rs#35-40.
- WorkerFleetPolicy: Defines the expectations for the fleet, such as requiring an attested egress proxy or specific version ranges crates/palyra-workerd/src/lib.rs#69-74.
- Lease Management: Short-lived
WorkerLeaserecords bind a worker to a specificrun_id, preventing cross-session contamination crates/palyra-workerd/src/lib.rs#173-177.
Node Runtime and mTLS
TheNodeRuntimeState in the daemon handles the low-level persistence and dispatching for remote nodes crates/palyra-daemon/src/node_runtime.rs#1-4. Communication is secured via mTLS, where every device_id is bound to a specific client certificate fingerprint crates/palyra-daemon/src/node_rpc.rs#4-6.
- Pairing: Nodes must undergo a pairing handshake (PIN or QR) which creates a
VerifiedPairingcrates/palyra-daemon/src/node_runtime.rs#134-149. - Dispatch: Tools are dispatched as
CapabilityRequestRecorditems over a persistent gRPC event stream crates/palyra-daemon/src/node_runtime.rs#180-181.
Egress Proxy Integration
Networked workers are often required to boot bound to an attested egress proxy crates/palyra-workerd/src/lib.rs#41-42. Thepalyra-egress-proxy crate provides the EgressProxyPolicyService, which enforces:
- Allowlists: Exact-match hosts and DNS-suffix matching crates/palyra-egress-proxy/src/lib.rs#40-43.
- Network Isolation: Blocking of private, loopback, and link-local addresses to prevent SSRF crates/palyra-egress-proxy/src/lib.rs#38-39.
- Credential Injection: Injecting secrets directly from the vault into headers, ensuring the worker never sees the raw secret material crates/palyra-egress-proxy/src/lib.rs#19-24.
Worker Quarantine and Security Gates
Workers that fail heartbeats or present invalid attestations are moved to aQuarantined or Orphaned state crates/palyra-workerd/src/lib.rs#5-6. Security is further hardened through “Critical Attack Scenarios” which are replayed against the attestation logic to ensure it remains fail-closed against digest mismatches or missing proxy bindings crates/palyra-workerd/tests/critical_attack_scenarios.rs#1-6.
Sub-pages
- Worker Fleet Management — Details the WorkerFleetManager in-memory ledger, attestation verification, lease lifecycle, quarantine/orphan handling, the WorkerFleetPolicy, and the node RPC mTLS communication.
- Egress Proxy for Networked Workers — Explains the palyra-egress-proxy crate: EgressProxyPolicyService, scheme/host allowlists, private IP blocking, DNS rebinding prevention, vault-only credential injection, and how workers must boot with an attested egress proxy.