> ## Documentation Index > Fetch the complete documentation index at: https://docs-code.palyra.com/llms.txt > Use this file to discover all available pages before exploring further. # Networked Workers and Distributed Execution
Relevant source files The following files were used as context for generating this wiki page: * apps/web/src/console/sections/LogsSection.tsx * crates/palyra-cli/examples/run\_release\_eval\_gate.rs * crates/palyra-cli/src/args/approvals.rs * crates/palyra-cli/src/args/devices.rs * crates/palyra-cli/src/args/node.rs * crates/palyra-cli/src/args/nodes.rs * crates/palyra-cli/src/args/pairing.rs * crates/palyra-cli/src/commands/devices.rs * crates/palyra-cli/src/commands/node.rs * crates/palyra-cli/src/commands/nodes.rs * crates/palyra-cli/tests/help\_snapshots/memory-learning-promote-procedure-help.txt * crates/palyra-cli/tests/help\_snapshots/node-install-help.txt * crates/palyra-common/src/release\_evals/catalog.rs * crates/palyra-common/src/release\_evals/evaluator.rs * crates/palyra-common/src/release\_evals/mod.rs * crates/palyra-common/src/release\_evals/projections.rs * crates/palyra-common/src/release\_evals/schema.rs * crates/palyra-common/tests/release\_eval\_contract.rs * crates/palyra-daemon/src/node\_rpc.rs * crates/palyra-daemon/src/node\_runtime.rs * crates/palyra-daemon/src/transport/http/handlers/console/approvals.rs * crates/palyra-daemon/src/transport/http/handlers/console/devices.rs * crates/palyra-daemon/src/transport/http/handlers/console/logs.rs * crates/palyra-daemon/src/transport/http/handlers/console/nodes.rs * crates/palyra-daemon/src/transport/http/handlers/console/pairing.rs * crates/palyra-egress-proxy/src/lib.rs * crates/palyra-egress-proxy/tests/critical\_attack\_scenarios.rs * crates/palyra-identity/src/pairing/manager.rs * crates/palyra-safety/tests/critical\_attack\_scenarios.rs * crates/palyra-workerd/src/lib.rs * crates/palyra-workerd/tests/critical\_attack\_scenarios.rs * fixtures/golden/release\_eval\_inventory.json
The Palyra networked worker system extends tool execution beyond the local daemon host. It enables distributed execution of high-risk or resource-intensive tasks across a fleet of remote workers while maintaining a "fail-closed" security posture. The system is built on cryptographically verified attestation, mTLS-secured communication, and strict egress control. ## System Overview Distributed execution in Palyra is managed by the `WorkerFleetManager` in the `palyra-workerd` crate, which maintains an in-memory ledger of available workers and their current lifecycle states [crates/palyra-workerd/src/lib.rs#3-6](http://crates/palyra-workerd/src/lib.rs#3-6). Workers join the fleet by presenting a `WorkerAttestation` containing identity, integrity, and compatibility claims [crates/palyra-workerd/src/lib.rs#30-35](http://crates/palyra-workerd/src/lib.rs#30-35). The daemon selects between local executors (Tier B/C) and networked workers based on the tool's required security posture and the `WorkerFleetPolicy`. ### Distributed Execution Flow The following diagram illustrates the lifecycle of a networked worker from registration to task execution. **Worker Lifecycle and Execution Flow** ```mermaid theme={null} sequenceDiagram participant W as Remote Worker participant WFM as WorkerFleetManager participant NR as NodeRuntimeState participant GRS as GatewayRuntimeState participant EP as EgressProxy W->>WFM: register_worker(WorkerAttestation) WFM->>WFM: validate(WorkerAttestationExpectation) Note over WFM: Check Image/Build/Artifact Digests WFM-->>W: Registration Success (mTLS established) GRS->>WFM: request_lease(run_id, capabilities) WFM-->>GRS: WorkerLease (ULID) GRS->>NR: enqueue_capability_request(device_id, tool_input) NR->>W: CapabilityDispatch (gRPC Stream) W->>EP: evaluate_request(EgressProxyRequest) EP-->>W: EgressPolicyVerdict (Allowed) W->>W: Execute Tool W-->>NR: CapabilityExecutionResult NR-->>GRS: Tool Output ``` Sources: [crates/palyra-workerd/src/lib.rs#3-6](http://crates/palyra-workerd/src/lib.rs#3-6), [crates/palyra-daemon/src/node\_runtime.rs#1-9](http://crates/palyra-daemon/src/node_runtime.rs#1-9), [crates/palyra-egress-proxy/src/lib.rs#3-6](http://crates/palyra-egress-proxy/src/lib.rs#3-6) ## Worker Fleet Management The `WorkerFleetManager` is the central authority for networked worker coordination. It tracks worker health via heartbeats and manages the lifecycle states (e.g., `Available`, `Leased`, `Quarantined`, `Orphaned`) [crates/palyra-workerd/src/lib.rs#10-11](http://crates/palyra-workerd/src/lib.rs#10-11). Key components include: * **WorkerAttestation**: A structure carrying SHA-256 digests of the worker image, build, and artifacts to ensure code integrity [crates/palyra-workerd/src/lib.rs#35-40](http://crates/palyra-workerd/src/lib.rs#35-40). * **WorkerFleetPolicy**: Defines the expectations for the fleet, such as requiring an attested egress proxy or specific version ranges [crates/palyra-workerd/src/lib.rs#69-74](http://crates/palyra-workerd/src/lib.rs#69-74). * **Lease Management**: Short-lived `WorkerLease` records bind a worker to a specific `run_id`, preventing cross-session contamination [crates/palyra-workerd/src/lib.rs#173-177](http://crates/palyra-workerd/src/lib.rs#173-177). For details, see [Worker Fleet Management](/networked_workers_and_distributed_execution/worker_fleet_management). ## Node Runtime and mTLS The `NodeRuntimeState` in the daemon handles the low-level persistence and dispatching for remote nodes [crates/palyra-daemon/src/node\_runtime.rs#1-4](http://crates/palyra-daemon/src/node_runtime.rs#1-4). Communication is secured via mTLS, where every `device_id` is bound to a specific client certificate fingerprint [crates/palyra-daemon/src/node\_rpc.rs#4-6](http://crates/palyra-daemon/src/node_rpc.rs#4-6). * **Pairing**: Nodes must undergo a pairing handshake (PIN or QR) which creates a `VerifiedPairing` [crates/palyra-daemon/src/node\_runtime.rs#134-149](http://crates/palyra-daemon/src/node_runtime.rs#134-149). * **Dispatch**: Tools are dispatched as `CapabilityRequestRecord` items over a persistent gRPC event stream [crates/palyra-daemon/src/node\_runtime.rs#180-181](http://crates/palyra-daemon/src/node_runtime.rs#180-181). **Code Entity Mapping: Node and Worker Communication** ```mermaid theme={null} classDiagram class NodeRpcServiceImpl { +identity_manager: Arc~Mutex~IdentityManager~~ +node_runtime: Arc~NodeRuntimeState~ +enforce_cert_bound_device() +peer_certificate_fingerprint() } class NodeRuntimeState { +nodes: HashMap~String, RegisteredNodeRecord~ +capability_requests: HashMap~String, CapabilityRequestRecord~ +enqueue_capability_request() } class WorkerFleetManager { +ledger: BTreeMap~Ulid, WorkerRecord~ +register_worker() +request_lease() } NodeRpcServiceImpl --> NodeRuntimeState : Drives NodeRuntimeState --> WorkerFleetManager : Orchestrates ``` Sources: [crates/palyra-daemon/src/node\_rpc.rs#55-61](http://crates/palyra-daemon/src/node_rpc.rs#55-61), [crates/palyra-daemon/src/node\_runtime.rs#171-181](http://crates/palyra-daemon/src/node_runtime.rs#171-181), [crates/palyra-workerd/src/lib.rs#3-6](http://crates/palyra-workerd/src/lib.rs#3-6) ## Egress Proxy Integration Networked workers are often required to boot bound to an attested egress proxy [crates/palyra-workerd/src/lib.rs#41-42](http://crates/palyra-workerd/src/lib.rs#41-42). The `palyra-egress-proxy` crate provides the `EgressProxyPolicyService`, which enforces: * **Allowlists**: Exact-match hosts and DNS-suffix matching [crates/palyra-egress-proxy/src/lib.rs#40-43](http://crates/palyra-egress-proxy/src/lib.rs#40-43). * **Network Isolation**: Blocking of private, loopback, and link-local addresses to prevent SSRF [crates/palyra-egress-proxy/src/lib.rs#38-39](http://crates/palyra-egress-proxy/src/lib.rs#38-39). * **Credential Injection**: Injecting secrets directly from the vault into headers, ensuring the worker never sees the raw secret material [crates/palyra-egress-proxy/src/lib.rs#19-24](http://crates/palyra-egress-proxy/src/lib.rs#19-24). For details, see [Egress Proxy for Networked Workers](/networked_workers_and_distributed_execution/egress_proxy_for_networked_workers). ## Worker Quarantine and Security Gates Workers that fail heartbeats or present invalid attestations are moved to a `Quarantined` or `Orphaned` state [crates/palyra-workerd/src/lib.rs#5-6](http://crates/palyra-workerd/src/lib.rs#5-6). Security is further hardened through "Critical Attack Scenarios" which are replayed against the attestation logic to ensure it remains fail-closed against digest mismatches or missing proxy bindings [crates/palyra-workerd/tests/critical\_attack\_scenarios.rs#1-6](http://crates/palyra-workerd/tests/critical_attack_scenarios.rs#1-6). ### Sub-pages * [Worker Fleet Management](/networked_workers_and_distributed_execution/worker_fleet_management) — Details the WorkerFleetManager in-memory ledger, attestation verification, lease lifecycle, quarantine/orphan handling, the WorkerFleetPolicy, and the node RPC mTLS communication. * [Egress Proxy for Networked Workers](/networked_workers_and_distributed_execution/egress_proxy_for_networked_workers) — Explains the palyra-egress-proxy crate: EgressProxyPolicyService, scheme/host allowlists, private IP blocking, DNS rebinding prevention, vault-only credential injection, and how workers must boot with an attested egress proxy. Sources: [crates/palyra-workerd/src/lib.rs#1-6](http://crates/palyra-workerd/src/lib.rs#1-6), [crates/palyra-workerd/tests/critical\_attack\_scenarios.rs#1-6](http://crates/palyra-workerd/tests/critical_attack_scenarios.rs#1-6), [crates/palyra-egress-proxy/src/lib.rs#1-7](http://crates/palyra-egress-proxy/src/lib.rs#1-7) ## Child Pages * [Worker Fleet Management](/networked_workers_and_distributed_execution/worker_fleet_management) * [Egress Proxy for Networked Workers](/networked_workers_and_distributed_execution/egress_proxy_for_networked_workers)