Add pragmas, hard_reset, and vacuum for tuning disk-backed caches
This commit is contained in:
@@ -263,17 +263,44 @@ engine = CachingEngine(base_engine, in_memory=False)
|
||||
|
||||
The constructor argument wins over the env var; when `in_memory` is omitted it falls back to `SQLMEM_IN_MEMORY`.
|
||||
|
||||
#### Tuning the SQLite layer (`pragmas=`)
|
||||
|
||||
For a large disk-backed cache, pass SQLite PRAGMAs to tune the read path and on-disk layout without bypassing SQLmem:
|
||||
|
||||
```python
|
||||
engine = CachingEngine(
|
||||
base_engine,
|
||||
in_memory=False,
|
||||
pragmas={
|
||||
"mmap_size": 32 * 1024**3, # map the DB into the address space (32 GB)
|
||||
"cache_size": -262144, # 256 MB page cache (negative = KiB)
|
||||
"temp_store": 2, # ORDER BY / GROUP BY scratch in RAM
|
||||
"page_size": 8192, # larger pages → fewer reads on range scans
|
||||
"auto_vacuum": "INCREMENTAL",# reclaim free pages with vacuum() (see below)
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
- Every entry is applied as `PRAGMA <key> = <value>` when the cache connection opens. **Unknown or inapplicable pragmas are silently ignored** by SQLite, so a bad value degrades gracefully instead of crashing startup.
|
||||
- **`page_size` and `auto_vacuum` are layout pragmas** — they only take effect on a *fresh* file (set before the first table). On an existing cache, `page_size` is ignored with a one-time warning; use [`hard_reset()`](#manual-cache-control) to rebuild the file with the new value.
|
||||
|
||||
## Manual cache control
|
||||
|
||||
```python
|
||||
engine.invalidate("orders") # drop one table from cache; next query re-fetches it from DB
|
||||
engine.reset() # wipe the whole cache (RAM + cache.db) — full clean slate
|
||||
engine.hard_reset() # disk mode: delete the file and reopen with current pragmas/page_size
|
||||
engine.vacuum() # disk mode: incremental VACUUM (reclaim free pages from delta churn)
|
||||
engine.refresh() # pull deltas for all delta-tracked tables now
|
||||
engine.close() # flush to disk and shut down background thread
|
||||
```
|
||||
|
||||
Use `reset()` after a **structural change** in the source (columns added/removed, values cleared in bulk without bumping the change timestamp) so the cache rebuilds from scratch. `invalidate(table)` is the targeted version for a single table.
|
||||
|
||||
`hard_reset()` goes further than `reset()` in disk mode: it closes every connection, deletes `cache.db` (and its `-wal`/`-shm` sidecars) and reopens from scratch — the only way to change a baked-in `page_size`/`auto_vacuum`. In memory mode it falls back to `reset()`.
|
||||
|
||||
`vacuum()` reclaims free pages left behind by delta `INSERT OR REPLACE` churn. Incremental (the default) is cheap and non-blocking but needs `auto_vacuum=INCREMENTAL`; `vacuum(incremental=False)` runs a full VACUUM that rewrites the file (~2× disk, blocks readers) — schedule it in a maintenance window. Both are no-ops in memory mode.
|
||||
|
||||
## Runtime statistics
|
||||
|
||||
```python
|
||||
@@ -346,6 +373,7 @@ engine = CachingEngine(
|
||||
refresh_interval=300, # SQLMEM_REFRESH_INTERVAL
|
||||
fetch_batch=10000, # SQLMEM_FETCH_BATCH
|
||||
dialect="tsql", # SQLMEM_SQL_DIALECT
|
||||
pragmas={"mmap_size": 32 * 1024**3, "page_size": 8192}, # disk-mode SQLite tuning
|
||||
blocking_startup_refresh=False, # block startup until caught up? (default: no)
|
||||
)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user