Split last_upsert (persisted write) and last_refresh (run liveness) in stats

This commit is contained in:
Jan Doubravský
2026-06-09 08:48:29 +02:00
parent 6dc85e4f3c
commit 8744f458cc
10 changed files with 108 additions and 11 deletions
+10 -1
View File
@@ -280,13 +280,22 @@ Use `reset()` after a **structural change** in the source (columns added/removed
stats = engine.stats # Stats snapshot
print(stats.hits, stats.misses, stats.refetches, stats.errors)
for name, t in stats.tables.items():
print(name, t.rows, t.state, t.tracking, t.last_refresh)
print(name, t.rows, t.state, t.tracking, t.last_upsert, t.last_refresh)
if t.consecutive_failures:
print(f" {name} failing ×{t.consecutive_failures}: {t.last_error} ({t.last_error_at})")
```
`Stats.errors` is the total number of load/refresh failures since start. Each `TableStats` also carries `last_error`, `last_error_at` and `consecutive_failures` (reset to 0 on the next success) — so a delta that fails *before* streaming (which otherwise leaves `state` looking `ready`) is still visible, and the table is marked `error`.
Two timestamps distinguish *data freshness* from *liveness*:
| field | meaning |
|---|---|
| `last_upsert` | wall-clock (UTC) of the last actual **data write** — full load or a delta cycle that wrote rows. Persisted, survives restarts. Answers *"when did the data last change?"* |
| `last_refresh` | wall-clock (UTC) of the last time a **refresh cycle ran** for the table — bumped **even when it wrote nothing**. In-memory per process (`None` until the first cycle runs after start). Answers *"is the refresh loop alive?"* |
A delta table that runs every cycle but finds no new rows keeps `last_refresh` ticking while `last_upsert` stays put — that's healthy, not stuck. (Both are UTC ISO strings; the default log timestamps are local time, so expect an offset.)
Each `TableStats` reports a live processing **state** and how the table is kept fresh (**tracking**):
| `state` | Meaning |