JournalStore & SQLite Backend
TheJournalStore is the central authority for persistence in the Palyra daemon crates/palyra-daemon/src/journal.rs#63-71. It manages a SQLite database (defaulting to data/journal.sqlite3) that stores everything from orchestrator tapes to cron job definitions and memory items crates/palyra-daemon/src/config/schema.rs#34-34.
Data Architecture
The system uses a single SQLite connection, typically wrapped in anArc<Mutex<JournalStore>> to coordinate access across asynchronous tasks crates/palyra-daemon/src/journal.rs#5-5.
| Entity | Purpose | Key Code Reference |
|---|---|---|
| JournalEvent | Generic audit log for system-wide events. | JournalEventRecord crates/palyra-daemon/src/journal.rs#63-63 |
| Orchestrator Tape | Append-only log of a specific Run’s execution steps. | OrchestratorTapeRecord crates/palyra-daemon/src/journal.rs#69-69 |
| Memory Item | Vectorized knowledge entries for RAG. | MemoryItemCreateRequest crates/palyra-daemon/src/journal.rs#64-64 |
| Cron Jobs | Scheduled task definitions and run history. | CronJobRecord crates/palyra-daemon/src/journal.rs#61-61 |
Tamper-Evident Hash Chaining
To ensure the integrity of the audit log, Palyra implements a hash-chaining mechanism. Every event appended to the journal includes ahash_chain value, which is a SHA-256 digest of the current event’s content concatenated with the hash_chain of the previous entry crates/palyra-daemon/src/gateway.rs#31-31.
Implementation Logic
- Seed: The chain starts with a constant seed
0000...0000crates/palyra-daemon/src/gateway.rs#114-115. - Append: When
JournalAppendRequestis processed, the daemon calculates the next hash. - Verification: The
palyra doctorCLI command can verify the chain to detect unauthorized database modifications.
Orchestrator Tape
The “Tape” is a specialized subset of the journal that tracks the lifecycle of a singleRun. It acts as the “source of truth” for the RunStateMachine crates/palyra-daemon/src/gateway.rs#77-77.
Tape Event Types
The tape records granular transitions, including:OrchestratorRunStartRequest: Initialization of a run crates/palyra-daemon/src/journal.rs#67-67.OrchestratorTapeAppendRequest: New LLM tokens, tool calls, or tool outputs crates/palyra-daemon/src/journal.rs#69-69.OrchestratorRunStatusSnapshot: Periodic state captures for UI streaming crates/palyra-daemon/src/journal.rs#67-67.
Data Flow: Gateway to Tape
The following diagram illustrates how incoming messages from theGatewayService are persisted into the JournalStore.
Diagram: Inbound Message Persistence Flow
Sources: crates/palyra-daemon/src/gateway.rs#40-82, crates/palyra-daemon/src/journal.rs#63-71
Memory Subsystem
The Memory subsystem provides both lexical (text) and vector (semantic) search capabilities. It is designed for “Learning” where the agent can store information for future recall.Vector Search & Embeddings
Palyra supports multiple embedding providers via theMemoryEmbeddingProvider trait crates/palyra-daemon/src/journal.rs#64-68.
- Hash-based Embeddings: A deterministic
HashMemoryEmbeddingProvideris used by default for local-only, low-latency indexing crates/palyra-daemon/src/journal.rs#71-79. - External Providers: Support for OpenAI-compatible embedding models crates/palyra-daemon/src/model_provider.rs#69-69.
Memory Maintenance & Retention
A background loop managed by theCron service handles memory health crates/palyra-daemon/src/cron.rs#56-58:
- Backfill: Backfills missing embeddings for newly added items crates/palyra-daemon/src/cron.rs#57-57.
- Retention: Enforces
MemoryRetentionPolicy(TTL, max entries, max bytes) crates/palyra-daemon/src/journal.rs#66-66. - Vacuum: Periodic SQLite vacuuming based on a configurable schedule (default: weekly) crates/palyra-daemon/src/config/schema.rs#31-31.
Persistence Configuration
Persistence behavior is governed by theStorageConfig and JournalConfig sections in palyra.toml.
| Parameter | Default | Description |
|---|---|---|
journal_db_path | data/journal.sqlite3 | Path to the SQLite database file crates/palyra-daemon/src/config/schema.rs#34-34. |
hash_chain_enabled | true | Enables tamper-evident logging crates/palyra-daemon/src/config/schema.rs#35-35. |
max_journal_payload_bytes | 256 KiB | Maximum size for a single journal entry payload crates/palyra-daemon/src/config/schema.rs#36-36. |
memory.retention.ttl_days | 30 | Retention period for memory items crates/palyra-daemon/src/config/schema.rs#28-28. |