Store named datetime columns as INTEGER microseconds (datetime_columns)
This commit is contained in:
@@ -284,6 +284,23 @@ engine = CachingEngine(
|
||||
- Every entry is applied as `PRAGMA <key> = <value>` when the cache connection opens. **Unknown or inapplicable pragmas are silently ignored** by SQLite, so a bad value degrades gracefully instead of crashing startup.
|
||||
- **`page_size` and `auto_vacuum` are layout pragmas** — they only take effect on a *fresh* file (set before the first table). On an existing cache, `page_size` is ignored with a one-time warning; use [`hard_reset()`](#manual-cache-control) to rebuild the file with the new value.
|
||||
|
||||
#### INTEGER datetime columns (`datetime_columns=`)
|
||||
|
||||
A pure datetime column stored as an ISO `TEXT` string costs ~28 bytes per row and compares by string collation. For a large table you can store named datetime columns as **INTEGER microseconds since the Unix epoch** instead — 8 bytes, native integer comparison:
|
||||
|
||||
```python
|
||||
engine = CachingEngine(
|
||||
base_engine,
|
||||
delta={"VW_P_PRATVALUES": DeltaConfig("CHANGE_DATE", ["PRATVALUE_ID"])},
|
||||
datetime_columns={"VW_P_PRATVALUES": ["CHANGE_DATE"]},
|
||||
)
|
||||
```
|
||||
|
||||
- **Opt-in per column.** Only the columns you name change; everything else keeps the default lossless `TEXT` storage.
|
||||
- ⚠️ **It changes the output contract for those columns** — `execute()` returns them as `int` (µs since epoch), not ISO strings, and a `WHERE` on them must compare against integer µs. Don't list a column your callers read as a string.
|
||||
- The delta watermark is handled transparently: it is persisted as the integer and bound back to a real `datetime` for the source query, so incremental refresh keeps working.
|
||||
- ⚠️ This is a **breaking on-disk change** (`SCHEMA_VERSION` 4): an existing cache is wiped and reloaded on first start after enabling it — schedule a maintenance window for a large reload.
|
||||
|
||||
## Manual cache control
|
||||
|
||||
```python
|
||||
@@ -374,6 +391,7 @@ engine = CachingEngine(
|
||||
fetch_batch=10000, # SQLMEM_FETCH_BATCH
|
||||
dialect="tsql", # SQLMEM_SQL_DIALECT
|
||||
pragmas={"mmap_size": 32 * 1024**3, "page_size": 8192}, # disk-mode SQLite tuning
|
||||
datetime_columns={"orders": ["created_at"]}, # store these as INTEGER µs (opt-in)
|
||||
blocking_startup_refresh=False, # block startup until caught up? (default: no)
|
||||
)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user