WorkerFleetManager Ledger
TheWorkerFleetManager serves as the in-memory ledger within the daemon. It tracks the availability, health, and current assignments of all registered workers.
- Registration: Workers must present a
WorkerAttestationto join the fleet. The manager validates this against theWorkerFleetPolicycrates/palyra-workerd/src/lib.rs#3-6. - State Tracking: It manages the
WorkerLifecycleState, transitioning workers between states such asAvailable,Busy,Quarantined, orOrphanedcrates/palyra-workerd/src/lib.rs#10-11. - Capacity Management: It enforces limits defined in the policy, such as the maximum number of concurrent workers or leases per worker.
Worker Lifecycle State Machine
The following diagram illustrates the transitions managed by theWorkerFleetManager.
“Worker Lifecycle State Machine”
Sources: crates/palyra-workerd/src/lib.rs#1-11, crates/palyra-workerd/src/lib.rs#73-91
Attestation and Verification
Attestation is the process by which a worker proves its identity and integrity to the daemon. TheWorkerAttestation struct contains claims about the worker’s environment and build crates/palyra-workerd/src/lib.rs#35-62.
Key Attestation Fields
| Field | Description |
|---|---|
worker_id | Unique identifier (ULID) for the worker crates/palyra-workerd/src/lib.rs#37. |
image_digest_sha256 | SHA-256 hash of the container/VM image crates/palyra-workerd/src/lib.rs#38. |
egress_proxy_attested | Boolean indicating if the worker is bound to a verified egress proxy crates/palyra-workerd/src/lib.rs#42. |
wit_abi_version | The Wasm Interface Type ABI version supported by the worker crates/palyra-workerd/src/lib.rs#52-54. |
WorkerAttestation::validate against a WorkerAttestationExpectation crates/palyra-workerd/src/lib.rs#113-117. The system fails closed if any digest (image, build, or artifact) mismatches or if the egress proxy binding is missing when required crates/palyra-workerd/src/lib.rs#127-151.
Sources: crates/palyra-workerd/src/lib.rs#35-153, crates/palyra-workerd/tests/critical_attack_scenarios.rs#73-91
Node RPC and mTLS Communication
Networked workers (and other remote nodes) communicate with the daemon via theNodeService gRPC interface crates/palyra-daemon/src/node_rpc.rs#1-4.
mTLS Enforcement
TheNodeRpcServiceImpl enforces Mutual TLS (mTLS) to ensure only paired devices can communicate.
- Fingerprint Extraction: The daemon extracts the SHA-256 fingerprint of the client certificate from the
TlsConnectInfocrates/palyra-daemon/src/node_rpc.rs#80-109. - Revocation Check: The fingerprint is checked against the
IdentityManagerto ensure it hasn’t been revoked crates/palyra-daemon/src/node_rpc.rs#115-119. - Device Binding: The daemon ensures the
device_idin the request matches the ID bound to that specific certificate fingerprint crates/palyra-daemon/src/node_rpc.rs#125-155.
Lease Lifecycle and Quarantine
When an agent requires a networked tool execution, it requests a lease from theWorkerFleetManager.
- Grant Authorization: An approval grant (
WorkerApprovalGrant) must exist for the specificrun_idcrates/palyra-workerd/src/lib.rs#173-177. - Assignment: The manager selects an
Availableworker and transitions it toBusy. - Workspace Scoping: The lease includes a
WorkerWorkspaceScopedefining the root directory and allowed paths for the worker crates/palyra-workerd/src/lib.rs#157-162. - Quarantine/Orphan Handling:
- Quarantine: If a worker crashes or violates security policy (e.g., unauthorized network access detected by the egress proxy), it is placed in
Quarantinedstate and cannot be reassigned until an operator intervenes crates/palyra-workerd/src/lib.rs#5-6. - Orphan: If a worker stops sending heartbeats, it is marked
Orphaned.
- Quarantine: If a worker crashes or violates security policy (e.g., unauthorized network access detected by the egress proxy), it is placed in
Egress Proxy Integration
Networked workers must boot bound to an attested egress proxy. TheEgressProxyPolicyService provides the logic used to validate outbound requests crates/palyra-egress-proxy/src/lib.rs#3-7.
- Address Blocking: It blocks resolution to private, loopback, or link-local addresses unless specifically opted-in via
allow_private_targetscrates/palyra-egress-proxy/src/lib.rs#38-39. - DNS Rebinding Protection: The service pins resolved socket addresses to prevent DNS rebinding attacks between policy evaluation and request execution crates/palyra-egress-proxy/src/lib.rs#65-67.
- Credential Injection: Only vault-backed secrets can be injected into proxied headers via
CredentialBindingPlancrates/palyra-egress-proxy/src/lib.rs#20-29.
Node Runtime and Pairing
TheNodeRuntimeState manages the persistence of node metadata and pairing requests in node-runtime.v1.json crates/palyra-daemon/src/node_runtime.rs#4-9.
- Pairing Codes: Operators mint
DevicePairingCodeRecord(PIN or QR) with a specific TTL crates/palyra-daemon/src/node_runtime.rs#65-71. - Approval Flow: Pairing requests enter
PendingApprovalstate. An operator must approve the request via the Console (which callsconsole_approval_decision_handler) crates/palyra-daemon/src/transport/http/handlers/console/approvals.rs#79-84. - Capability Inventory: Once registered, nodes report their
DeviceCapabilityView, which the daemon uses to route tool execution requests crates/palyra-daemon/src/node_runtime.rs#152-156.
| System Concept | Code Entity | File |
|---|---|---|
| Node Ledger | PersistedNodeRuntimeState | crates/palyra-daemon/src/node_runtime.rs#171-181 |
| Pairing Request | DevicePairingRequestRecord | crates/palyra-daemon/src/node_runtime.rs#134-149 |
| mTLS Implementation | NodeRpcServiceImpl | crates/palyra-daemon/src/node_rpc.rs#56-61 |
| Capability Dispatch | CapabilityRequestRecord | crates/palyra-daemon/src/node_runtime.rs#180 |
| Console Handler | console_nodes_list_handler | crates/palyra-daemon/src/transport/http/handlers/console/nodes.rs#55-58 |