Skip to main content
This page documents the multi-layered testing strategy used to ensure the reliability, security, and cross-platform parity of the Palyra ecosystem. The strategy spans from unit-level deterministic core tests to high-level workflow regression matrices and automated fuzzing.

​
Testing Architecture Overview

Palyra employs a “pyramid” testing model with specialized tracks for different failure modes:
  1. Deterministic Core Tests: Fast, hermetic tests for critical logic in palyrad and palyra-cli.
  2. Workflow Regression Matrix: End-to-end integration tests that simulate complex user journeys (e.g., setup, backup, restore).
  3. CLI Install Smoke Tests: Validation of production-ready artifacts across Windows, macOS, and Linux.
  4. Fuzzing: Property-based and mutation-based testing for untrusted input parsers.
  5. Parity Fixtures: Ensuring behavioral consistency across different transport layers and platforms.

​
Test Execution Flow

The following diagram illustrates how different test suites are orchestrated during the CI lifecycle. Diagram: CI Test Orchestration Flow Sources: scripts/test/run-deterministic-core.sh:1-64(), .github/workflows/cli-install-smoke.yml:1-63(), crates/palyra-cli/tests/workflow_regression_matrix.rs:29-190()

​
Deterministic Core Tests

The core tests focus on the daemon’s internal state transitions and the CLI’s command parsing without requiring external network access. These are orchestrated by scripts/test/run-deterministic-core.ps1 (or .sh). Key components tested:

​
Deterministic Fixtures

The system uses crates/palyra-connector-core/tests/fixtures/channel_simulator_expected.json to verify that channel simulators produce byte-for-byte identical output across versions scripts/test/check-deterministic-fixtures.ps1#5-7. The check-deterministic-fixtures.sh script enforces that these fixtures are updated and committed whenever underlying logic changes scripts/test/check-deterministic-fixtures.sh#20-26. Sources: scripts/test/run-deterministic-core.ps1:1-36(), scripts/test/check-deterministic-fixtures.ps1:1-32()

​
Workflow Regression Matrix

The workflow_regression_matrix.rs suite performs high-level integration testing by spawning actual palyrad and palyra processes. It simulates real-world usage patterns in a temporary workspace crates/palyra-cli/tests/workflow_regression_matrix.rs#31-35.

​
Key Regression Scenarios

Diagram: Regression Test Environment Setup Sources: crates/palyra-cli/tests/workflow_regression_matrix.rs:31-55()

​
CLI Install Smoke Tests

Smoke tests verify the integrity of the release artifacts. Unlike core tests which use cargo test, these run against the compiled production binary crates/palyra-cli/tests/installed_smoke.rs#43-56.

​
Smoke Harness Implementation

The harness uses scripts/test/run-cli-install-smoke.ps1 to:
  1. Initialize a New-ScenarioContext with isolated environment variables like PALYRA_CONFIG and PALYRA_VAULT_DIR scripts/test/run-cli-install-smoke.ps1#41-83.
  2. Invoke commands via Invoke-TranscriptCommand, which captures stdout, stderr, and exit_code into a detailed transcript for CI debugging scripts/test/run-cli-install-smoke.ps1#126-200.
  3. Verify the palyra update --dry-run and palyra uninstall flows to ensure maintenance commands work on an installed system crates/palyra-cli/tests/installed_smoke.rs#102-146.
Sources: scripts/test/run-cli-install-smoke.ps1:41-200(), crates/palyra-cli/tests/installed_smoke.rs:1-148()

​
Fuzzing Targets

Palyra uses cargo-fuzz (libFuzzer) to find edge cases in parsers that handle untrusted data. Fuzz targets are located in the fuzz/ directory.
TargetCrate / ModulePurpose
workspace_patch_parserpalyra-commonFuzzes RFC 6902-style JSON patches used in A2UI.
process_runner_input_parserpalyra-commonFuzzes the serialization/deserialization of tool execution payloads crates/palyra-common/src/process_runner_input.rs.
config_path_parserpalyra-cliFuzzes path resolution and URI scheme parsing in CLI configuration.
Sources: fuzz/Cargo.lock:1-258(), fuzz/fuzz_targets/workspace_patch_parser.rs(), fuzz/fuzz_targets/process_runner_input_parser.rs()

​
Performance Smoke Tests

The scripts/test/run-performance-smoke.ps1 script executes high-frequency operations to detect regressions in latency or resource consumption.
  • Journal Append Throughput: Measures the time to commit 1,000 events to the JournalStore.
  • Vault Encryption Latency: Measures the overhead of SensitiveBytes envelope encryption.
  • Startup Latency: Ensures the palyrad daemon reaches a Ready state within defined thresholds.
Sources: scripts/test/run-performance-smoke.ps1(), crates/palyra-cli/tests/workflow_regression_matrix.rs:26-27()