palyra) is designed to operate consistently across Linux, macOS, and Windows. To ensure this consistency, the codebase employs a multi-layered testing strategy including platform-specific help snapshots, a parity matrix for command coverage, and automated “smoke” tests that validate installation and critical workflows (setup, backup, and update) on all supported operating systems.
CLI Parity Framework
TheCliParityMatrix is the central definition of expected CLI behavior across platforms. It ensures that every command is accounted for, categorized, and has an associated help snapshot for regression testing.
CliParityMatrix Implementation
The framework is defined inpalyra-cli/src/cli_parity.rs and utilizes a TOML-based matrix (tests/cli_parity_matrix.toml) to track the status of the CLI surface area crates/palyra-cli/tests/help_snapshots.rs#6-6.
CliParityMatrix: A collection ofCliParityEntryitems representing the entire command tree.CliParityEntry: Contains the command path (e.g.,palyra agents list), its category (e.g.,management), and aCliParitySnapshotSpec.CliParitySnapshotSpec: Defines which file contains the expected help output for a given platform. This handles differences in line endings and file extensions (e.g.,palyravspalyra.exe) crates/palyra-cli/tests/help_snapshots.rs#22-24.
Help Snapshot Testing
Thehelp_snapshots_match_cli_parity_matrix test crates/palyra-cli/tests/help_snapshots.rs#75-126 iterates through the matrix and executes palyra <cmd> --help. It compares the actual output against stored text files in tests/help_snapshots/.
- Unix Snapshots: Stored in
root-help-unix.txtcrates/palyra-cli/tests/help_snapshots/root-help-unix.txt#1-112. - Windows Snapshots: Stored in
root-help-windows.txtcrates/palyra-cli/tests/help_snapshots/root-help-windows.txt#1-112. - Normalization: The
normalize_help_textfunction strips carriage returns and replaces platform-specific binary names to ensure snapshots remain portable where possible crates/palyra-cli/tests/help_snapshots.rs#22-24.
Install Smoke Tests
Install smoke tests validate that the CLI functions correctly after being packaged and installed, rather than just running in a development environment. These tests are executed viascripts/test/run-cli-install-smoke.ps1 (PowerShell) and its shell script equivalent scripts/test/run-cli-install-smoke.ps1#1-46.
Smoke Test Lifecycle
- Context Creation:
New-ScenarioContextgenerates a temporary filesystem layout includingconfig/,state/, andvault/directories to simulate a fresh installation scripts/test/run-cli-install-smoke.ps1#41-83. - Environment Isolation: The script sets
PALYRA_CONFIG,PALYRA_STATE_ROOT, andPALYRA_VAULT_DIRto point to these temporary locations, ensuring no interference with the host system scripts/test/run-cli-install-smoke.ps1#85-101. - Command Execution:
Invoke-TranscriptCommandruns the binary and captures stdout/stderr/exit codes into a transcript log for debugging scripts/test/run-cli-install-smoke.ps1#126-203.
Validated Commands
Theinstalled_binary_runs_baseline_smoke_commands test crates/palyra-cli/tests/installed_smoke.rs#39-148 verifies:
palyra versionandpalyra --helpcrates/palyra-cli/tests/installed_smoke.rs#69-70.palyra doctor --jsonfor diagnostic health crates/palyra-cli/tests/installed_smoke.rs#72-75.palyra config validatefor schema integrity crates/palyra-cli/tests/installed_smoke.rs#85-92.palyra update --dry-runandpalyra uninstall --dry-runcrates/palyra-cli/tests/installed_smoke.rs#102-146.
Workflow Regression Matrix
The workflow regression matrix validates complex, multi-step CLI operations like the setup wizard and configuration migration. This is implemented incrates/palyra-cli/tests/workflow_regression_matrix.rs.
Regression Scenarios
The testlocal_remote_and_lifecycle_workflows_are_regression_tested crates/palyra-cli/tests/workflow_regression_matrix.rs#29-188 covers the following critical paths:
| Workflow | Command | Key Validations |
|---|---|---|
| Local Setup | palyra setup --mode local | Creates palyra.toml, initializes local state crates/palyra-cli/tests/workflow_regression_matrix.rs#57-85. |
| Remote Onboarding | palyra onboarding wizard --flow remote | Validates server certificates and admin tokens crates/palyra-cli/tests/workflow_regression_matrix.rs#87-119. |
| Configuration | palyra configure --section gateway | Modifies ports and TLS profiles in existing config crates/palyra-cli/tests/workflow_regression_matrix.rs#121-162. |
| Backup | palyra backup create | Archives config and workspace into a ZIP crates/palyra-cli/tests/workflow_regression_matrix.rs#173-188. |
Internal Architecture of Workflow Tests
The regression tests use arun_cli helper that wraps std::process::Command. It allows passing standard input (for interactive prompts) and environment variables crates/palyra-cli/tests/support/cli_harness.rs.
Title: Workflow Regression Components
Sources: crates/palyra-cli/tests/workflow_regression_matrix.rs#29-188, crates/palyra-cli/src/args/mod.rs#1-44, crates/palyra-cli/src/commands/mod.rs#1-48
CI Integration
The parity and smoke tests are integrated into the GitHub Actions pipeline to prevent platform-specific regressions.cli-install-smoke.yml: Runs the smoke harness onubuntu-latest,macos-latest, andwindows-latest.github/workflows/cli-install-smoke.yml#21-27.- Artifacts: On failure, the pipeline uploads the transcript logs and the temporary state root created by the smoke harness for inspection .github/workflows/cli-install-smoke.yml#56-62.
- Snapshot Update: Developers can update help snapshots by setting the
PALYRA_UPDATE_HELP_SNAPSHOTSenvironment variable while running tests crates/palyra-cli/tests/help_snapshots.rs#8-8.