Transparent in-memory cache layer between SQLAlchemy and your database. Drop it in front of any SQLAlchemy engine — SELECT queries are served from a fast in-memory SQLite cache, writes are rejected (read-only cache).

Goals

SQLmem sits between your application and the database and behaves like a normal SQLAlchemy connection. It transparently:

Intercepts every query that passes through it and learns, from the SQL itself, which tables and which columns the application actually uses.
Holds exactly those tables/columns locally in SQLite — primarily in RAM, secondarily persisted to disk (cache.db) at regular intervals and on shutdown.
Serves repeated queries from RAM with no database round-trip.
Stays in sync incrementally (see Incremental refresh): for large tables you declare a change-timestamp column, and SQLmem only re-fetches rows that changed in the last few minutes (or since the last shutdown) instead of reloading tens of millions of rows on every start.

The application keeps calling SQL as usual — the cache is an implementation detail behind the interface.

How it works

flowchart TB
    App["Application (SQLAlchemy code)"]
    DB[("Source database")]

    subgraph SM["SQLmem - transparent cache layer"]
        direction TB
        P["SQL Parser (sqlglot)<br/>detect SELECT vs write<br/>extract tables + columns"]
        R["Column Registry<br/>tracks tables + columns in cache"]
        QE["Query Executor<br/>cache hit / miss / refetch"]
        MEM[("In-memory SQLite - PRIMARY")]
        DISK[("cache.db on disk - SECONDARY")]
        P --> R --> QE --> MEM
        MEM -->|"backup every N s + on shutdown"| DISK
        DISK -->|"load on startup"| MEM
    end

    App -->|"execute(sql, params)"| P
    QE -->|"cache miss / delta refresh only"| DB
    DB -->|"rows"| MEM
    MEM -->|"list of dicts"| App

On the first SELECT touching a table, SQLmem fetches the required rows from the database and stores them in the in-memory SQLite. Subsequent queries for the same columns hit RAM with no database round-trip. When a query requests a column not yet cached, SQLmem re-fetches the table with the expanded column set. Parametrized queries, JOINs and SELECT * are all supported; each table in a JOIN is cached independently and the JOIN runs inside the in-memory SQLite.

Query lifecycle

sequenceDiagram
    participant App
    participant SQLmem
    participant Mem as In-memory SQLite
    participant DB as Source DB

    App->>SQLmem: execute(SELECT a, b FROM t WHERE id = ?, params)
    SQLmem->>SQLmem: parse -> table = t, columns = {a, b, id}
    alt columns already cached
        SQLmem->>Mem: run query in RAM (with params)
        Mem-->>SQLmem: rows
    else cache miss or new column
        SQLmem->>DB: SELECT a, b, id FROM t   (whole table, no WHERE)
        DB-->>SQLmem: rows
        SQLmem->>Mem: store / expand table
        SQLmem->>Mem: run query in RAM (with params)
        Mem-->>SQLmem: rows
    end
    SQLmem-->>App: list[dict]

Note: query parameters are applied only to the in-memory query, never to the source fetch — a cache load always pulls the full table so the cache can answer any later WHERE on those columns.

Installation

pip install sqlmem
# or with Poetry
poetry add sqlmem

Requires Python 3.14.

Quick start

from sqlmem import CachingEngine
from sqlalchemy import create_engine

base_engine = create_engine("postgresql://user:pass@host/db")
engine = CachingEngine(base_engine)

# Use exactly like a regular SQLAlchemy engine:
results = engine.execute("SELECT id, name FROM users WHERE status = 'active'")
for row in results:
    print(row["id"], row["name"])

# Positional parameters (?):
engine.execute("SELECT id, name FROM users WHERE id = ?", ("42",))

# Named parameters (:name):
engine.execute("SELECT id, name FROM users WHERE id = :id", {"id": "42"})

# JOINs — each table is cached independently:
engine.execute(
    "SELECT u.name, o.total FROM users u "
    "JOIN orders o ON o.user_id = u.id WHERE u.id = ?",
    ("42",),
)

# SELECT * — loads and caches the whole table:
engine.execute("SELECT * FROM users")

execute() returns a list of dicts. Parameters are passed straight through to SQLite, so positional (?) and named (:name) styles both work.

Cache behaviour

Column accumulation — SQLmem learns which columns your app needs at runtime, no upfront configuration required:

Query 1: SELECT a, b FROM orders   → cache miss → fetch orders(a, b) from DB
Query 2: SELECT a, d FROM orders   → new column d → re-fetch orders(a, b, d)
Query 3: SELECT b FROM orders      → cache hit, no DB query
Query 4: SELECT * FROM orders      → fetches all columns, marks the table fully cached
Query 5: SELECT a FROM orders      → cache hit (table already full)

SELECT * loads every column and marks the table as fully cached, so any later column query is a guaranteed cache hit with no re-fetch.

Writes are blocked — INSERT, UPDATE, and DELETE raise ReadOnlyError. SQLmem is a read-only cache.

Incremental (delta) refresh

Reloading a table with tens of millions of rows on every startup is unacceptable. To avoid it, SQLmem keeps the cache in sync by pulling only changed rows. For each delta-tracked table you declare its last-change timestamp column and the key column(s) that identify a row:

from sqlmem import CachingEngine, DeltaConfig

engine = CachingEngine(
    base_engine,
    delta={
        "VW_P_PRATVALUES": DeltaConfig(
            change_column="LAST_CHANGE_DATE",   # required — the row's change timestamp
            key_columns=["PRODUCT_PRODUCTNR"],  # optional for base tables (auto-discovered)
        ),
    },
)

What you must configure, and what is automatic:

Item	Source
which tables / columns to cache	automatic — learned from the queries that pass through
`change_column` (timestamp)	manual, always — its meaning can't be inferred from the column type^*
`key_columns` (primary key)	auto-discovered for real tables (`inspect(engine).get_pk_constraint`); manual for views, which carry no key in the DB catalog

^* The one exception is a true MSSQL rowversion/timestamp-typed column, which is unique per table and auto-maintained — that could be detected automatically. A plain DATETIME like LAST_CHANGE_DATE cannot.

If key_columns is omitted, SQLmem tries to read the primary key from the source DB on startup and raises a clear error if it can't (e.g. for a view) so you can supply it explicitly.

How sync works

The boundary of "what changed since last time" is a data-driven watermark, not a wall-clock window. SQLmem persists, per delta-tracked table, last_synced_at = the maximum change_column value actually present in the cache after the previous sync (stored in cache.db, so it survives restarts). The next sync pulls WHERE change_column >= last_synced_at.

Why a watermark and not now − 5 min:

No clock dependency — it compares DB values to DB values, so app-server vs database clock skew is irrelevant.
Survives downtime for free — after hours offline, >= watermark pulls everything since then; "catch up since last shutdown" needs no special case.
Never misses late commits — a wall-clock window can drop a row whose timestamp falls outside the window by the time it commits.

The filter is >= (not >) so rows sharing the exact boundary timestamp are re-read; combined with idempotent upsert by key_columns, re-reading a handful of boundary rows each tick is harmless (they overwrite themselves), and no row is ever skipped. The 5-minute interval is only the polling cadence, never the filter boundary.

sequenceDiagram
    participant Trigger as Startup / every 5 min
    participant SQLmem
    participant Mem as In-memory SQLite
    participant DB as Source DB

    Trigger->>SQLmem: refresh delta-tracked tables
    SQLmem->>Mem: read last_synced_at for table
    SQLmem->>DB: SELECT * FROM t WHERE LAST_CHANGE_DATE >= last_synced_at
    DB-->>SQLmem: only rows changed since the watermark
    SQLmem->>Mem: upsert rows by key_columns (INSERT OR REPLACE)
    SQLmem->>Mem: last_synced_at = max(LAST_CHANGE_DATE)

First use of a delta table → full load; the watermark is set to the table's current max(change_column).
On startup → for each delta table restored from disk, a single catch-up query pulls everything changed since the last shutdown and upserts it, bringing the cache back in sync without a full reload.
While running → a background thread repeats the delta pull every SQLMEM_REFRESH_INTERVAL seconds (default 5 minutes), so the cache trails the source DB by at most that interval.
Tables without a DeltaConfig keep the default behaviour: full load on miss, never auto-refreshed — unless they are given a TTL.

Requirements and limits of delta sync

The change_column must be set by the source DB on every insert/update and be non-decreasing (e.g. a DATETIME/rowversion/timestamp maintained by a trigger or the application).
key_columns must uniquely identify a row — they are used to upsert changed rows in place.
Updates, including "deletes by nulling" (a row that keeps its identity but has values cleared), are handled automatically: the change timestamp bumps, the row is re-pulled and overwritten in place.
Structural changes are not covered by delta sync — adding/removing attributes, or clearing values without bumping change_column, won't be picked up. For those, force a clean reload with engine.reset() (or invalidate() for a single table).
Hard DELETEs of whole rows are not detected by a change-timestamp; this workload doesn't delete rows, but if yours does, use a soft-delete flag column or reset().

Time-based refresh (tables without a change column)

Some tables can't be delta-synced because they have no change timestamp. For those you can set a TTL (max age in seconds): SQLmem keeps serving from cache and guarantees the cached copy is never older than the TTL by doing a full reload when it expires.

engine = CachingEngine(
    base_engine,
    ttl={
        "VW_LOOKUP_CODES": 300,   # full-reload if the cache is older than 5 minutes
        "VW_SETTINGS": 3600,
    },
)

Read-time guarantee — when a query touches a TTL table whose cache is older than its TTL, the table is fully reloaded before the query is answered, so a stale copy is never returned.
Proactive — the background thread also full-reloads expired TTL tables every SQLMEM_REFRESH_INTERVAL seconds, keeping them warm so reads usually don't pay the reload latency.
TTL age is measured from last_refresh_at, which is persisted in cache.db, so the guarantee holds across restarts (an expired table is reloaded on first use after start).
A table may be in either delta or ttl, not both (delta already keeps it fresh) — supplying both raises ValueError.

engine.refresh()   # also reloads any expired TTL tables on demand

Secondary indexes

To accelerate lookups, you can declare secondary indexes per table — they are created on the in-memory SQLite copy so WHERE/JOIN filters on those columns run as indexed searches instead of full scans:

engine = CachingEngine(
    base_engine,
    indexes={
        "VW_P_PRATVALUES": ["PRODUCT_PRODUCTNR"],   # single-column index
        "VW_ELEMENTS": [["ELEMENT_ID", "ELEMENTVARIANT_ID"], "ELEMENTVARIANT_NAME"],
    },
)

Each value is a list of index definitions: a string is a single-column index, a nested list is a composite index.

Index columns are pulled into the cache automatically (like delta key columns), so the index exists from the first load even if your queries don't select those columns.
Indexes are recreated after every (re)load — full loads, TTL reloads, and invalidate() + re-fetch all rebuild them — so they're always present, and they persist in cache.db across restarts.
Delta-tracked tables already get a unique index on their key columns; secondary indexes are independent and can be combined with delta or ttl.

Persistence

By default the cache lives in an in-memory SQLite and is persisted to cache.db on disk:

On startup: if cache.db exists, it is loaded into memory.
Periodically: a background thread writes a snapshot to disk every SQLMEM_BACKUP_INTERVAL seconds.
On shutdown: a final flush via atexit and SIGTERM handler.

The schema version is checked on load — if it does not match, the stale file is discarded and the cache is rebuilt from the database.

Disk-backed cache (no RAM copy)

Set in_memory=False (or SQLMEM_IN_MEMORY=false) to query the on-disk cache.db directly instead of mirroring it in RAM:

engine = CachingEngine(base_engine, in_memory=False)

The cache can exceed available memory — nothing is held in RAM beyond SQLite's page cache.
Every write persists immediately (WAL + synchronous=NORMAL), so there is no hourly backup thread, no load-into-memory step on startup, and no shutdown flush to lose.
Reads run concurrently — each thread reads through its own read-only WAL connection, so a slow SELECT doesn't block writers (loads/upserts) or other readers.
On open, a cache file with a mismatched schema version is wiped in place and rebuilt; engine.reset() drops the cached tables and VACUUMs the file (it does not delete the open file).

The constructor argument wins over the env var; when in_memory is omitted it falls back to SQLMEM_IN_MEMORY.

Manual cache control

engine.invalidate("orders")   # drop one table from cache; next query re-fetches it from DB
engine.reset()                # wipe the whole cache (RAM + cache.db) — full clean slate
engine.refresh()              # pull deltas for all delta-tracked tables now
engine.close()                # flush to disk and shut down background thread

Use reset() after a structural change in the source (columns added/removed, values cleared in bulk without bumping the change timestamp) so the cache rebuilds from scratch. invalidate(table) is the targeted version for a single table.

Runtime statistics

stats = engine.stats          # Stats snapshot
print(stats.hits, stats.misses, stats.refetches, stats.errors)
for name, t in stats.tables.items():
    print(name, t.rows, t.state, t.tracking, t.last_upsert, t.last_refresh)
    if t.consecutive_failures:
        print(f"  {name} failing ×{t.consecutive_failures}: {t.last_error} ({t.last_error_at})")

Stats.errors is the total number of load/refresh failures since start. Each TableStats also carries last_error, last_error_at and consecutive_failures (reset to 0 on the next success) — so a delta that fails before streaming (which otherwise leaves state looking ready) is still visible, and the table is marked error.

Two timestamps distinguish data freshness from liveness:

field	meaning
`last_upsert`	wall-clock (UTC) of the last actual data write — full load or a delta cycle that wrote rows. Persisted, survives restarts. Answers "when did the data last change?"
`last_refresh`	wall-clock (UTC) of the last time a refresh cycle ran for the table — bumped even when it wrote nothing. In-memory per process (`None` until the first cycle runs after start). Answers "is the refresh loop alive?"

A delta table that runs every cycle but finds no new rows keeps last_refresh ticking while last_upsert stays put — that's healthy, not stuck. (Both are UTC ISO strings; the default log timestamps are local time, so expect an offset.)

Each TableStats reports a live processing state and how the table is kept fresh (tracking):

`state`	Meaning
`loading`	a full load is in progress
`refreshing`	an incremental (delta) refresh is in progress
`ready`	cached and idle (up to date)
`stale`	a TTL table whose cache has expired; reloads on next access
`error`	the last load failed

`tracking`	Meaning
`delta`	kept in sync incrementally via a change column
`ttl`	full-reloaded when older than its TTL
`static`	loaded on demand, never auto-refreshed

Memory and very large tables

By default the cache is in-memory SQLite, so a cached table lives in RAM — it must fit in available memory. To keep huge tables manageable:

Use disk-backed mode (in_memory=False) when the working set simply doesn't fit in RAM — queries then run against cache.db on disk instead of a memory copy.
Loads are streamed in batches (SQLMEM_FETCH_BATCH rows at a time, default 10 000) into a staging table and swapped in atomically. A multi-million-row table never gets fully materialized in Python at once, so the load doesn't spike memory or crash the process, and readers keep seeing the previous copy until the swap completes.
Use delta refresh for large tables that have a change column — after the first load only changed rows are pulled, so restarts and refreshes don't re-read the whole table.
A single query that returns a huge result set (e.g. SELECT * over a multi-million-row cached table) still materializes that result as a list of dicts; bound it with a WHERE/LIMIT rather than selecting everything.

Configuration

Set via environment variables or a .env file:

Variable	Default	Description
`SQLMEM_DEBUG`	`false`	`true` enables DEBUG-level logging
`SQLMEM_CACHE_DB`	`cache.db`	Path to the on-disk persistence file
`SQLMEM_IN_MEMORY`	`true`	`false` queries `cache.db` on disk directly (no RAM copy); overridden by the `in_memory` constructor arg
`SQLMEM_BACKUP_INTERVAL`	`3600`	Disk backup interval in seconds (in-memory mode only)
`SQLMEM_SQL_DIALECT`	`tsql`	sqlglot dialect used to parse incoming SQL (e.g. `tsql`, `postgres`, `mysql`)
`SQLMEM_REFRESH_INTERVAL`	`300`	background refresh tick (seconds) — delta pulls and proactive TTL reloads
`SQLMEM_FETCH_BATCH`	`10000`	rows fetched per batch when loading a table — caps peak memory for huge tables

Most of these can also be passed per engine to the constructor, overriding the env default — handy for running two engines (with separate cache files) in one process, and for tests:

engine = CachingEngine(
    base_engine,
    cache_db_path="orders_cache.db",   # SQLMEM_CACHE_DB
    in_memory=False,                   # SQLMEM_IN_MEMORY
    backup_interval=3600,              # SQLMEM_BACKUP_INTERVAL
    refresh_interval=300,              # SQLMEM_REFRESH_INTERVAL
    fetch_batch=10000,                 # SQLMEM_FETCH_BATCH
    dialect="tsql",                    # SQLMEM_SQL_DIALECT
    blocking_startup_refresh=False,    # block startup until caught up? (default: no)
)

By default the startup catch-up (delta pulls and TTL reloads for tables restored from disk) runs on the background thread so it never blocks application startup; the cache may serve slightly stale data until the first refresh completes. Set blocking_startup_refresh=True to catch up synchronously before the engine starts serving.

Exceptions

Exception	When raised
`ReadOnlyError`	INSERT, UPDATE, or DELETE statement
`UnsupportedQueryError`	non-SELECT statement, `SELECT` without `FROM`, or an unqualified column in a multi-table query

from sqlmem import ReadOnlyError, UnsupportedQueryError

Logging

SQLmem is silent by default. Call add_sink() to opt in:

import sys
from sqlmem import add_sink

add_sink(sys.stderr)                      # INFO by default
add_sink(sys.stderr, level="DEBUG")       # verbose: every query, cache hit/miss, backup
add_sink("sqlmem.log", rotation="10 MB")  # to a file

Set SQLMEM_DEBUG=true in .env to make the default level DEBUG when no explicit level is passed to add_sink().

Limitations

In a multi-table (JOIN) query, every column must be qualified with its table or alias; unqualified columns raise UnsupportedQueryError.
Tables are keyed by their base name — two tables with the same name in different schemas share one cache entry.
No distributed cache backend (Redis etc.).
No transactional consistency guarantees; the cache trails the source DB.
Write operations (INSERT/UPDATE/DELETE) are always blocked.

Roadmap

Incremental (delta) refresh via per-table change-timestamp + key columns (see above) — the key feature for large tables.
Primary-key auto-discovery from the source DB (inspect(engine).get_pk_constraint) so key_columns is only needed for views.
engine.reset() — wipe RAM + cache.db for a clean rebuild after structural changes.
Per-table TTL (time-to-live) — bounded-staleness full refresh for tables without a change column.

Dependencies

Layer	Library
SQL parsing	`sqlglot`
Cache storage	`sqlite3` (stdlib)
Integration	SQLAlchemy 2.x
Logging	`loguru`, `python-dotenv`

License

MIT

README.md Unescape Escape

SQLmem