Add pragmas, hard_reset, and vacuum for tuning disk-backed caches

This commit is contained in:
Jan Doubravský
2026-06-09 17:58:41 +02:00
parent 8744f458cc
commit a21b5a2a04
10 changed files with 359 additions and 6 deletions
+28
View File
@@ -263,17 +263,44 @@ engine = CachingEngine(base_engine, in_memory=False)
The constructor argument wins over the env var; when `in_memory` is omitted it falls back to `SQLMEM_IN_MEMORY`.
#### Tuning the SQLite layer (`pragmas=`)
For a large disk-backed cache, pass SQLite PRAGMAs to tune the read path and on-disk layout without bypassing SQLmem:
```python
engine = CachingEngine(
base_engine,
in_memory=False,
pragmas={
"mmap_size": 32 * 1024**3, # map the DB into the address space (32 GB)
"cache_size": -262144, # 256 MB page cache (negative = KiB)
"temp_store": 2, # ORDER BY / GROUP BY scratch in RAM
"page_size": 8192, # larger pages → fewer reads on range scans
"auto_vacuum": "INCREMENTAL",# reclaim free pages with vacuum() (see below)
},
)
```
- Every entry is applied as `PRAGMA <key> = <value>` when the cache connection opens. **Unknown or inapplicable pragmas are silently ignored** by SQLite, so a bad value degrades gracefully instead of crashing startup.
- **`page_size` and `auto_vacuum` are layout pragmas** — they only take effect on a *fresh* file (set before the first table). On an existing cache, `page_size` is ignored with a one-time warning; use [`hard_reset()`](#manual-cache-control) to rebuild the file with the new value.
## Manual cache control
```python
engine.invalidate("orders") # drop one table from cache; next query re-fetches it from DB
engine.reset() # wipe the whole cache (RAM + cache.db) — full clean slate
engine.hard_reset() # disk mode: delete the file and reopen with current pragmas/page_size
engine.vacuum() # disk mode: incremental VACUUM (reclaim free pages from delta churn)
engine.refresh() # pull deltas for all delta-tracked tables now
engine.close() # flush to disk and shut down background thread
```
Use `reset()` after a **structural change** in the source (columns added/removed, values cleared in bulk without bumping the change timestamp) so the cache rebuilds from scratch. `invalidate(table)` is the targeted version for a single table.
`hard_reset()` goes further than `reset()` in disk mode: it closes every connection, deletes `cache.db` (and its `-wal`/`-shm` sidecars) and reopens from scratch — the only way to change a baked-in `page_size`/`auto_vacuum`. In memory mode it falls back to `reset()`.
`vacuum()` reclaims free pages left behind by delta `INSERT OR REPLACE` churn. Incremental (the default) is cheap and non-blocking but needs `auto_vacuum=INCREMENTAL`; `vacuum(incremental=False)` runs a full VACUUM that rewrites the file (~2× disk, blocks readers) — schedule it in a maintenance window. Both are no-ops in memory mode.
## Runtime statistics
```python
@@ -346,6 +373,7 @@ engine = CachingEngine(
refresh_interval=300, # SQLMEM_REFRESH_INTERVAL
fetch_batch=10000, # SQLMEM_FETCH_BATCH
dialect="tsql", # SQLMEM_SQL_DIALECT
pragmas={"mmap_size": 32 * 1024**3, "page_size": 8192}, # disk-mode SQLite tuning
blocking_startup_refresh=False, # block startup until caught up? (default: no)
)
```