Honza/SQLmem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add scheduled refresh for tables that change at a fixed time of day

Browse Source
This commit is contained in:
Honza
2026-07-30 11:28:20 +02:00
parent 4a86b2282f
commit 0026a4df22
12 changed files with 652 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+51 -8
View File
@@ -212,12 +212,48 @@ engine = CachingEngine(
- **Read-time guarantee** — when a query touches a TTL table whose cache is older than its TTL, the table is fully reloaded *before* the query is answered, so a stale copy is never returned.
- **Proactive** — the background thread also full-reloads expired TTL tables every `SQLMEM_REFRESH_INTERVAL` seconds, keeping them warm so reads usually don't pay the reload latency.
- TTL age is measured from `last_refresh_at`, which is persisted in `cache.db`, so the guarantee holds across restarts (an expired table is reloaded on first use after start).
- A table may be in **either** `delta` **or** `ttl`, not both (delta already keeps it fresh) — supplying both raises `ValueError`.
- A table has **exactly one** refresh method — `delta`, `ttl` or [`schedule`](#scheduled-refresh-at-a-fixed-time-of-day); listing it under two of them raises `ValueError`.
```python
engine.refresh() # also reloads any expired TTL tables on demand
```
## Scheduled refresh (at a fixed time of day)
When a table only changes at a known moment — an overnight batch, a nightly import, a report generated every morning — a TTL wastes reloads waiting for the next tick to notice. Give it a **schedule** instead: the table is fully reloaded at the times of day you name.
```python
from datetime import time
from sqlmem import CachingEngine
engine = CachingEngine(
base_engine,
schedule={
"VW_NIGHTLY_PRICES": "03:00", # every day at 03:00
"VW_STOCK_LEVELS": ["06:30", "14:30"], # twice a day
"VW_REPORT": time(5, 15), # datetime.time works too
},
)
```
Times are `"HH:MM"` / `"HH:MM:SS"` strings or `datetime.time` objects, one or a list of them, interpreted in the **local timezone** of the process. An unparsable or timezone-aware time raises `ValueError` at construction.
- **Slot-based, not timer-based** — a table is due when it has not been reloaded since its most recent scheduled moment. Consequences:
- **A missed slot is caught up.** If the process was down at 03:00 and starts at 09:00, the table reloads on the first refresh check after start (`last_refresh_at` is persisted in `cache.db`, so this holds across restarts).
- **A slot never fires twice**, however often the check runs.
- **Fires on time** — the background thread shortens its wait so it wakes within about a second of each slot, instead of up to `SQLMEM_REFRESH_INTERVAL` late.
- **Read-time guarantee** — same as TTL: a query touching a table whose slot has passed triggers the reload *before* it is answered, so a pre-batch copy is never returned even if the background thread hasn't run yet.
- Around a daylight-saving change a slot may land an hour off for that one day; the reload still happens.
Also available as a `TableSpec.refresh` strategy in [declarative mode](#declarative-initialization-tables):
```python
from sqlmem import TableSpec, Schedule
TableSpec("VW_NIGHTLY_PRICES", refresh=Schedule("03:00"), preload=True)
```
Pick `schedule` when the source changes on a clock, `ttl` when it changes continuously but a bounded lag is acceptable, and `delta` when the table is large and has a change column.
## Secondary indexes
To accelerate lookups, you can declare **secondary indexes** per table — they are created on the in-memory SQLite copy so `WHERE`/`JOIN` filters on those columns run as indexed searches instead of full scans:
@@ -236,14 +272,14 @@ Each value is a list of index definitions: a string is a single-column index, a
- **Index columns are pulled into the cache automatically** (like delta key columns), so the index exists from the first load even if your queries don't select those columns.
- Indexes are **recreated after every (re)load** — full loads, TTL reloads, and `invalidate()` + re-fetch all rebuild them — so they're always present, and they persist in `cache.db` across restarts.
- Delta-tracked tables already get a unique index on their key columns; secondary indexes are independent and can be combined with `delta` or `ttl`.
- Delta-tracked tables already get a unique index on their key columns; secondary indexes are independent and can be combined with `delta`, `ttl` or `schedule`.
## Declarative initialization (`tables=`)
Instead of the lazy "learn columns from queries" mode, you can **declare every table up front** with `tables=[TableSpec(...)]` — its columns, indexes, refresh strategy and which columns are datetimes — and have the engine preload them and reject anything undeclared:
```python
from sqlmem import CachingEngine, TableSpec, Delta, TTL
from sqlmem import CachingEngine, TableSpec, Delta, TTL, Schedule
engine = CachingEngine(
base_engine,
@@ -263,6 +299,11 @@ engine = CachingEngine(
refresh=TTL(seconds=1800),
preload=True,
),
TableSpec(
name="VW_NIGHTLY_PRICES",
refresh=Schedule("03:00"),
preload=True,
),
],
pragmas={"mmap_size": 32 * 1024**3, "page_size": 8192},
)
@@ -270,8 +311,8 @@ engine = CachingEngine(
- **Preload** — `preload=True` tables are loaded at startup (on the background thread by default, so startup isn't blocked; pass `blocking_startup_refresh=True` to load them synchronously before serving). A copy already fresh in the persistent cache is **skipped**, so a warm restart is instant. During warm-up a table reports `TableState.LOADING` in [`stats`](#runtime-statistics) — handy for gating a `503` until it's `ready`.
- **Fail-fast** — a query for a table without a `TableSpec`, or for a column outside a spec's declared `columns` (including `SELECT *` on a column-restricted table), raises `UndeclaredError` instead of silently kicking off an expensive lazy load. Use `columns=None` to cache the whole table and allow any column.
- `refresh=` takes a `Delta(change_column=…, key_columns=…)` (same as `DeltaConfig`) or `TTL(seconds=…)`, or `None` for a static table.
- **Backward compatible** — omit `tables=` and the legacy `delta=`/`ttl=`/`indexes=`/`datetime_columns=` kwargs work exactly as before (lazy mode, no fail-fast). Passing both raises `ValueError`.
- `refresh=` takes a `Delta(change_column=…, key_columns=…)` (same as `DeltaConfig`), `TTL(seconds=…)` or [`Schedule(times)`](#scheduled-refresh-at-a-fixed-time-of-day), or `None` for a static table.
- **Backward compatible** — omit `tables=` and the legacy `delta=`/`ttl=`/`schedule=`/`indexes=`/`datetime_columns=` kwargs work exactly as before (lazy mode, no fail-fast). Passing both raises `ValueError`.
## Persistence
@@ -350,7 +391,7 @@ engine.invalidate("orders") # drop one table from cache; next query re-fetches
engine.reset() # wipe the whole cache (RAM + cache.db) — full clean slate
engine.hard_reset() # disk mode: delete the file and reopen with current pragmas/page_size
engine.vacuum() # disk mode: incremental VACUUM (reclaim free pages from delta churn)
engine.refresh() # pull deltas for all delta-tracked tables now
engine.refresh() # pull deltas + reload any expired ttl/schedule tables now
engine.close() # flush to disk and shut down background thread
```
@@ -389,13 +430,14 @@ Each `TableStats` reports a live processing **state** and how the table is kept
| `loading` | a full load is in progress |
| `refreshing` | an incremental (delta) refresh is in progress |
| `ready` | cached and idle (up to date) |
| `stale` | a TTL table whose cache has expired; reloads on next access |
| `stale` | a TTL table past its max age, or a scheduled table whose slot has passed; reloads on next access |
| `error` | the last load failed |
| `tracking` | Meaning |
|---|---|
| `delta` | kept in sync incrementally via a change column |
| `ttl` | full-reloaded when older than its TTL |
| `schedule` | full-reloaded at fixed times of day |
| `static` | loaded on demand, never auto-refreshed |
## Memory and very large tables
@@ -419,7 +461,7 @@ Set via environment variables or a `.env` file:
| `SQLMEM_IN_MEMORY` | `true` | `false` queries `cache.db` on disk directly (no RAM copy); overridden by the `in_memory` constructor arg |
| `SQLMEM_BACKUP_INTERVAL` | `3600` | Disk backup interval in seconds (in-memory mode only) |
| `SQLMEM_SQL_DIALECT` | `tsql` | sqlglot dialect used to parse incoming SQL (e.g. `tsql`, `postgres`, `mysql`) |
| `SQLMEM_REFRESH_INTERVAL` | `300` | background refresh tick (seconds) — delta pulls and proactive TTL reloads |
| `SQLMEM_REFRESH_INTERVAL` | `300` | background refresh tick (seconds) — delta pulls and proactive TTL/schedule reloads; shortened automatically when a `schedule` slot falls sooner |
| `SQLMEM_FETCH_BATCH` | `10000` | rows fetched per batch when loading a table — caps peak memory for huge tables |
Most of these can also be passed **per engine** to the constructor, overriding the env default — handy for running two engines (with separate cache files) in one process, and for tests:
@@ -483,6 +525,7 @@ Set `SQLMEM_DEBUG=true` in `.env` to make the default level DEBUG when no explic
- [x] **Primary-key auto-discovery** from the source DB (`inspect(engine).get_pk_constraint`) so `key_columns` is only needed for views.
- [x] **`engine.reset()`** — wipe RAM + `cache.db` for a clean rebuild after structural changes.
- [x] **Per-table TTL** (time-to-live) — bounded-staleness full refresh for tables without a change column.
- [x] **Scheduled refresh** — full reload at fixed times of day for tables that change on a clock (nightly batch, morning import).
## Dependencies