JournalStore. It covers the event log architecture, the SQLite-backed storage mechanism, the memory ingestion pipeline for long-term context, and the background maintenance tasks that ensure data integrity and relevance.
JournalStore and Event Log
TheJournalStore is the primary persistence engine for the Palyra daemon. It manages a structured event log that records every significant action within the system, including message routing, tool executions, and state transitions.
Implementation and Integrity
The store is backed by SQLite and implements a hash-chain integrity model. Each event record is linked to the previous one via a SHA-256 hash, ensuring that the history of a session or a run cannot be tampered with without detection. Key features of theJournalStore:
- Hash-Chaining: Every
JournalEventRecordcontains a hash of its contents and the hash of the preceding event crates/palyra-daemon/src/gateway.rs#114-115. - ULID Indexing: Events are indexed using Lexicographically Sortable Identifiers (ULID) for efficient time-based querying crates/palyra-daemon/src/journal.rs#17-17.
- Write Latency Budget: The system targets a journal write latency of under 25ms to avoid blocking the execution pipeline crates/palyra-daemon/src/gateway.rs#92-92.
Data Flow: Event Appending
When a component (like the Orchestrator) needs to persist an event, it issues aJournalAppendRequest. The JournalStore handles the transaction, calculates the next hash in the chain, and commits the record to the journal_events table.
Sources: crates/palyra-daemon/src/gateway.rs#63-71, crates/palyra-daemon/src/journal.rs#13-17
Memory Ingestion Pipeline
Memory in Palyra is a multi-tiered system that transitions short-term session context into long-term “semantic” memory.Ingestion and Embeddings
When events are marked for long-term retention, they enter the ingestion pipeline.- Text Extraction: Content is extracted from
OrchestratorTapeRecordentries. - Embedding Generation: Text is sent to a
MemoryEmbeddingProvider. By default, Palyra uses aHashMemoryEmbeddingProviderfor deterministic, local-first processing crates/palyra-daemon/src/journal.rs#71-80, but it can be configured to use external LLM providers like OpenAI crates/palyra-daemon/src/model_provider.rs#20-21. - Vector Storage: The resulting vectors are stored in the SQLite database using a flattened vector representation (64 dimensions by default) crates/palyra-daemon/src/journal.rs#53-53.
Backfill Mechanism
To handle high-throughput scenarios without stalling, embeddings are often processed as a background “backfill” task. TheMEMORY_EMBEDDINGS_BACKFILL_INTERVAL (default 10 minutes) triggers a process that identifies memory items missing vectors and generates them in batches of 64 crates/palyra-daemon/src/cron.rs#57-58.
Sources: crates/palyra-daemon/src/journal.rs#64-100, crates/palyra-daemon/src/cron.rs#56-58, crates/palyra-daemon/src/model_provider.rs#27-30
Persistence Policies and Compaction
To prevent database bloat and maintain performance, Palyra implements several retention and compaction strategies.MemoryRetentionPolicy
TheMemoryRetentionPolicy defines the lifecycle of persisted items. It supports:
- TTL (Time-to-Live): Automatic expiration of items after a set duration (e.g.,
ttl_days) crates/palyra-common/src/daemon_config_schema.rs#201-201. - Capacity Limits: Enforcing maximum entry counts or byte sizes for the memory store crates/palyra-common/src/daemon_config_schema.rs#201-202.
- Vacuuming: Scheduled SQLite vacuuming to reclaim disk space, configured via
vacuum_schedulecrates/palyra-common/src/daemon_config_schema.rs#204-204.
Session Compaction
AsOrchestratorSessionRecord histories grow, they undergo compaction. The session_compaction module crates/palyra-daemon/src/application/mod.rs#10-10 reduces the “tape” of a session by:
- Summarizing old turns.
- Removing redundant tool outputs.
- Archiving the full high-fidelity logs to the
JournalStorewhile keeping a lightweight “active” state for the LLM context.
| Policy Parameter | Default Value | Code Reference |
|---|---|---|
MAX_MEMORY_ITEM_BYTES | 16 KB | crates/palyra-daemon/src/gateway.rs#118-118 |
MAX_MEMORY_ITEM_TOKENS | 2,048 | crates/palyra-daemon/src/gateway.rs#119-119 |
MEMORY_MAINTENANCE_INTERVAL | 5 Minutes | crates/palyra-cron.rs#56-56 |
Code Entity Map: Persistence Layer
The following diagram maps the logical persistence concepts to their specific implementation classes and file locations. Sources: crates/palyra-daemon/src/journal.rs#13-17, crates/palyra-daemon/src/journal.rs#63-71, crates/palyra-daemon/src/journal.rs#71-80Background Maintenance and Cron
Persistence is not a passive activity in Palyra. Thespawn_scheduler_loop manages several maintenance tasks:
- Memory Maintenance Tick: Every 5 minutes, the system checks for expired memory items based on the
MemoryRetentionPolicycrates/palyra-daemon/src/cron.rs#56-56. - Embedding Backfill: Periodically scans for items requiring vector generation crates/palyra-daemon/src/cron.rs#57-57.
- Cron Job Execution: The
CronMatcherevaluatesCronJobRecordschedules to trigger automated runs, which are then logged asCronRunRecordin the journal crates/palyra-daemon/src/cron.rs#129-156.