Workflow Regression Matrix
The workflow regression matrix is the primary integration gate for the Palyra ecosystem. It validates end-to-end scenarios by spawning realpalyrad and browserd processes on dynamic loopback ports, interacting with them via the CLI and gRPC clients.
Implementation and Data Flow
The matrix is driven by a JSON manifestinfra/release/workflow-regression-matrix.json infra/release/workflow-regression-matrix.json#1-121, which defines:
- Profiles:
fast(per-push) andfull(nightly/pre-release) infra/release/workflow-regression-matrix.json#5-51. - Subsystems: Logical groupings of functionality like
session_queue_policy,retrieval_dual_path, andnetworked_workersinfra/release/workflow-regression-matrix.json#52-121. - Scenarios: Specific test cases mapped to CLI commands or unit tests infra/release/workflow-regression-matrix.json#152-190.
run_workflow_regression example crates/palyra-cli/examples/run_workflow_regression.rs#24-180 orchestrates the execution. It loads the manifest, filters scenarios based on the active profile, and executes them, capturing logs and generating a WorkflowRegressionRunReport.
Regression Components
| Component | Responsibility | Source |
|---|---|---|
workflow_regression_matrix.rs | Main test runner for local/remote/session lifecycle. | crates/palyra-cli/tests/workflow_regression_matrix.rs#38-158 |
workflow_regression_contract.rs | Validates consistency between manifest and code. | crates/palyra-cli/tests/workflow_regression_contract.rs#45-107 |
run-workflow-regression.sh | Entry point for CI environments. | scripts/test/run-workflow-regression.sh#40-49 |
CLI Parity and ACP Shim Harness
The CLI infrastructure ensures that thepalyra binary maintains a stable contract for both human operators and automated shims (like the Agent Control Protocol - ACP).
Help Snapshot and Status Tests
The suite validates that CLI output remains consistent across versions. Thestatus_reports_http_grpc_and_admin_health test crates/palyra-cli/tests/cli_v1_acp_shim.rs#26-61 verifies that the status command correctly reports on all gateway transports.
ACP NDJSON Shim
The ACP shim allows external tools to consume agent events as NDJSON. Theagent_acp_shim_emits_ndjson_events test crates/palyra-cli/tests/cli_v1_acp_shim.rs#204-207 ensures that when --output-format json is used, the CLI emits valid, machine-readable events instead of formatted text.
Sources: crates/palyra-cli/tests/cli_v1_acp_shim.rs#1-207.
Golden Fixture and Installed Smoke Tests
To prevent regression in the distribution artifacts, Palyra runs “smoke tests” against fully packaged binaries.Installed Smoke Context
Theinstalled_smoke.rs suite uses environment variables like PALYRA_BIN_UNDER_TEST to locate a production-build binary crates/palyra-cli/tests/installed_smoke.rs#28-40. It performs:
- Baseline Commands:
version,--help, anddoctorcrates/palyra-cli/tests/installed_smoke.rs#86-96. - Configuration Validation:
config validateandsetup --wizardcrates/palyra-cli/tests/installed_smoke.rs#102-109. - Update/Uninstall Dry-runs: Validates the logic for managing the installation root crates/palyra-cli/tests/installed_smoke.rs#119-163.
Scenario Orchestration
The PowerShell scriptrun-cli-install-smoke.ps1 scripts/test/run-cli-install-smoke.ps1#41-83 creates isolated scenario contexts (temporary home directories, config paths, and state roots) to ensure that the binary behaves correctly in clean-room environments.
Sources: crates/palyra-cli/tests/installed_smoke.rs#1-163, scripts/test/run-cli-install-smoke.ps1#1-101.
Worker Fleet and Egress Testing
Testing the worker fleet requires simulating attestation and network isolation.Worker Attestation Harness
Thepalyra-workerd crate defines WorkerAttestation crates/palyra-workerd/src/lib.rs#35-62. Tests validate the validate function crates/palyra-workerd/src/lib.rs#113-153 against WorkerAttestationExpectation to ensure that only workers with valid image digests and attested egress proxies can join the fleet.
Egress Proxy Verification
Thepalyra-egress-proxy suite ensures that the EgressProxyPolicyService crates/palyra-egress-proxy/src/lib.rs#125-127 correctly blocks forbidden schemes and non-allowlisted hosts. The evaluate_request function crates/palyra-egress-proxy/src/lib.rs#140-150 is tested against malicious inputs to confirm fail-closed behavior.
Sources: crates/palyra-workerd/src/lib.rs#35-153, crates/palyra-egress-proxy/src/lib.rs#1-150.
Release Evaluation Gate
The Release Evaluation Gate is the final check before a build is promoted. It uses theCompatReleaseReadinessChecklist crates/palyra-cli/tests/workflow_regression_contract.rs#12-13 to ensure that all “must-pass” scenarios for a given release scope are met.
Readiness Logic
The gate logic is defined ininfra/release/compat-hardening-readiness.json infra/release/compat-hardening-readiness.json#1-37, which tracks:
- Rollout Controls: Feature flags that must be in a specific state (e.g.,
networked_worker_preview_rollout) infra/release/compat-hardening-readiness.json#28-31. - Evidence Requirements: Maps specific evidence IDs (like
workflow_regression_fast) to the subsystems and scenarios that must pass infra/release/compat-hardening-readiness.json#76-123.