SQLmem/README.md

SQLmem

Transparent in-memory cache layer between SQLAlchemy and your database. Drop it in front of any SQLAlchemy engine — SELECT queries are served from a fast in-memory SQLite cache, writes pass through unchanged.

How it works

Application (SQLAlchemy)
        │
        ▼
  [ SQLmem Proxy ]
  ┌──────────────────────────────┐
  │  SQL Parser                  │  → detects SELECT vs. write
  │  Column Registry             │  → tracks which columns are cached per table
  │  Cache Manager (SQLite RAM)  │  → stores data in memory
  │  Query Executor              │  → cache hit / miss logic
  └──────────────────────────────┘
        │
        ▼
  Database (via original SQLAlchemy engine)

On the first SELECT for a table, SQLmem fetches the required rows from the database and stores them in an in-memory SQLite instance. Subsequent queries for the same columns hit the in-memory cache with no database round-trip. When a query requests a column not yet in cache, SQLmem re-fetches the table with the expanded column set.

Parametrized queries, JOINs and SELECT * are all supported. Each table referenced in a JOIN is cached independently; the JOIN itself runs in the in-memory SQLite. Query parameters are applied during in-memory filtering, so cache loads always fetch the full table regardless of the WHERE values.

Installation

pip install sqlmem
# or with Poetry
poetry add sqlmem

Requires Python 3.14.

Quick start

from sqlmem import CachingEngine
from sqlalchemy import create_engine, text

base_engine = create_engine("postgresql://user:pass@host/db")
engine = CachingEngine(base_engine)

# Use exactly like a regular SQLAlchemy engine:
results = engine.execute("SELECT id, name FROM users WHERE status = 'active'")
for row in results:
    print(row["id"], row["name"])

# Positional parameters (?):
engine.execute("SELECT id, name FROM users WHERE id = ?", ("42",))

# Named parameters (:name):
engine.execute("SELECT id, name FROM users WHERE id = :id", {"id": "42"})

# JOINs — each table is cached independently:
engine.execute(
    "SELECT u.name, o.total FROM users u "
    "JOIN orders o ON o.user_id = u.id WHERE u.id = ?",
    ("42",),
)

# SELECT * — loads and caches the whole table:
engine.execute("SELECT * FROM users")

execute() returns a list of dicts. Parameters are passed straight through to SQLite, so positional (?) and named (:name) styles both work. Results are compatible with standard iteration patterns.

Cache behaviour

Column accumulation — SQLmem learns which columns your app needs at runtime, no upfront configuration required:

Query 1: SELECT a, b FROM orders   → cache miss → fetch orders(a, b) from DB
Query 2: SELECT a, d FROM orders   → new column d → re-fetch orders(a, b, d)
Query 3: SELECT b FROM orders      → cache hit, no DB query
Query 4: SELECT * FROM orders      → fetches all columns, marks the table fully cached
Query 5: SELECT a FROM orders      → cache hit (table already full)

SELECT * loads every column and marks the table as fully cached, so any later column query is a guaranteed cache hit with no re-fetch.

Writes are blocked — INSERT, UPDATE, and DELETE raise ReadOnlyError. SQLmem is a read-only cache.

Persistence

The in-memory cache is optionally persisted to cache.db on disk:

• On startup: if cache.db exists, it is loaded into memory.
• Hourly: a background thread writes a snapshot to disk.
• On shutdown: a final flush via atexit and SIGTERM handler.

Schema version is checked on load — if it does not match, the stale file is discarded and the cache is rebuilt from the database.

Manual cache invalidation

engine.invalidate("orders")   # drops the table from cache; next query re-fetches from DB
engine.close()                # flush to disk and shut down background thread

Configuration

Set via environment variables or a .env file:

Variable	Default	Description
SQLMEM_DEBUG	false	true enables DEBUG-level logging
SQLMEM_CACHE_DB	cache.db	Path to the on-disk persistence file
SQLMEM_BACKUP_INTERVAL	3600	Backup interval in seconds
SQLMEM_SQL_DIALECT	tsql	sqlglot dialect used to parse incoming SQL (e.g. tsql, postgres, mysql)

Exceptions

Exception	When raised
ReadOnlyError	INSERT, UPDATE, or DELETE statement
UnsupportedQueryError	non-SELECT statement, SELECT without FROM, or an unqualified column in a multi-table query

from sqlmem import ReadOnlyError, UnsupportedQueryError

Logging

SQLmem is silent by default. Call add_sink() to opt in:

import sys
from sqlmem import add_sink

add_sink(sys.stderr)                      # INFO by default
add_sink(sys.stderr, level="DEBUG")       # verbose: every query, cache hit/miss, backup
add_sink("sqlmem.log", rotation="10 MB") # to a file

Set SQLMEM_DEBUG=true in .env to make the default level DEBUG when no explicit level is passed to add_sink().

Limitations

• In a multi-table (JOIN) query, every column must be qualified with its table or alias; unqualified columns raise UnsupportedQueryError.
• Tables are keyed by their base name — two tables with the same name in different schemas share one cache entry.
• No distributed cache backend (Redis etc.).
• No transactional consistency guarantees.
• Write operations (INSERT/UPDATE/DELETE) are always blocked.

Dependencies

Layer	Library
SQL parsing	sqlglot
Cache storage	sqlite3 (stdlib)
Integration	SQLAlchemy 2.x
Logging	loguru, python-dotenv

License

MIT