Jan Doubravský
160 lines
5.8 KiB
Markdown
160 lines
5.8 KiB
Markdown
# SQLmem
|
|
|
|
Transparent in-memory cache layer between SQLAlchemy and your database. Drop it in front of any SQLAlchemy engine — SELECT queries are served from a fast in-memory SQLite cache, writes pass through unchanged.
|
|
|
|
## How it works
|
|
|
|
```
|
|
Application (SQLAlchemy)
|
|
│
|
|
▼
|
|
[ SQLmem Proxy ]
|
|
┌──────────────────────────────┐
|
|
│ SQL Parser │ → detects SELECT vs. write
|
|
│ Column Registry │ → tracks which columns are cached per table
|
|
│ Cache Manager (SQLite RAM) │ → stores data in memory
|
|
│ Query Executor │ → cache hit / miss logic
|
|
└──────────────────────────────┘
|
|
│
|
|
▼
|
|
Database (via original SQLAlchemy engine)
|
|
```
|
|
|
|
On the first SELECT for a table, SQLmem fetches the required rows from the database and stores them in an in-memory SQLite instance. Subsequent queries for the same columns hit the in-memory cache with no database round-trip. When a query requests a column not yet in cache, SQLmem re-fetches the table with the expanded column set.
|
|
|
|
Parametrized queries, JOINs and `SELECT *` are all supported. Each table referenced in a JOIN is cached independently; the JOIN itself runs in the in-memory SQLite. Query parameters are applied during in-memory filtering, so cache loads always fetch the full table regardless of the `WHERE` values.
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
pip install sqlmem
|
|
# or with Poetry
|
|
poetry add sqlmem
|
|
```
|
|
|
|
Requires Python 3.14.
|
|
|
|
## Quick start
|
|
|
|
```python
|
|
from sqlmem import CachingEngine
|
|
from sqlalchemy import create_engine, text
|
|
|
|
base_engine = create_engine("postgresql://user:pass@host/db")
|
|
engine = CachingEngine(base_engine)
|
|
|
|
# Use exactly like a regular SQLAlchemy engine:
|
|
results = engine.execute("SELECT id, name FROM users WHERE status = 'active'")
|
|
for row in results:
|
|
print(row["id"], row["name"])
|
|
|
|
# Positional parameters (?):
|
|
engine.execute("SELECT id, name FROM users WHERE id = ?", ("42",))
|
|
|
|
# Named parameters (:name):
|
|
engine.execute("SELECT id, name FROM users WHERE id = :id", {"id": "42"})
|
|
|
|
# JOINs — each table is cached independently:
|
|
engine.execute(
|
|
"SELECT u.name, o.total FROM users u "
|
|
"JOIN orders o ON o.user_id = u.id WHERE u.id = ?",
|
|
("42",),
|
|
)
|
|
|
|
# SELECT * — loads and caches the whole table:
|
|
engine.execute("SELECT * FROM users")
|
|
```
|
|
|
|
`execute()` returns a list of dicts. Parameters are passed straight through to SQLite, so positional (`?`) and named (`:name`) styles both work. Results are compatible with standard iteration patterns.
|
|
|
|
## Cache behaviour
|
|
|
|
**Column accumulation** — SQLmem learns which columns your app needs at runtime, no upfront configuration required:
|
|
|
|
```
|
|
Query 1: SELECT a, b FROM orders → cache miss → fetch orders(a, b) from DB
|
|
Query 2: SELECT a, d FROM orders → new column d → re-fetch orders(a, b, d)
|
|
Query 3: SELECT b FROM orders → cache hit, no DB query
|
|
Query 4: SELECT * FROM orders → fetches all columns, marks the table fully cached
|
|
Query 5: SELECT a FROM orders → cache hit (table already full)
|
|
```
|
|
|
|
**`SELECT *`** loads every column and marks the table as fully cached, so any later column query is a guaranteed cache hit with no re-fetch.
|
|
|
|
**Writes are blocked** — INSERT, UPDATE, and DELETE raise `ReadOnlyError`. SQLmem is a read-only cache.
|
|
|
|
## Persistence
|
|
|
|
The in-memory cache is optionally persisted to `cache.db` on disk:
|
|
|
|
- **On startup**: if `cache.db` exists, it is loaded into memory.
|
|
- **Hourly**: a background thread writes a snapshot to disk.
|
|
- **On shutdown**: a final flush via `atexit` and SIGTERM handler.
|
|
|
|
Schema version is checked on load — if it does not match, the stale file is discarded and the cache is rebuilt from the database.
|
|
|
|
## Manual cache invalidation
|
|
|
|
```python
|
|
engine.invalidate("orders") # drops the table from cache; next query re-fetches from DB
|
|
engine.close() # flush to disk and shut down background thread
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Set via environment variables or a `.env` file:
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `SQLMEM_DEBUG` | `false` | `true` enables DEBUG-level logging |
|
|
| `SQLMEM_CACHE_DB` | `cache.db` | Path to the on-disk persistence file |
|
|
| `SQLMEM_BACKUP_INTERVAL` | `3600` | Backup interval in seconds |
|
|
| `SQLMEM_SQL_DIALECT` | `tsql` | sqlglot dialect used to parse incoming SQL (e.g. `tsql`, `postgres`, `mysql`) |
|
|
|
|
## Exceptions
|
|
|
|
| Exception | When raised |
|
|
|---|---|
|
|
| `ReadOnlyError` | INSERT, UPDATE, or DELETE statement |
|
|
| `UnsupportedQueryError` | non-SELECT statement, `SELECT` without `FROM`, or an unqualified column in a multi-table query |
|
|
|
|
```python
|
|
from sqlmem import ReadOnlyError, UnsupportedQueryError
|
|
```
|
|
|
|
## Logging
|
|
|
|
SQLmem is silent by default. Call `add_sink()` to opt in:
|
|
|
|
```python
|
|
import sys
|
|
from sqlmem import add_sink
|
|
|
|
add_sink(sys.stderr) # INFO by default
|
|
add_sink(sys.stderr, level="DEBUG") # verbose: every query, cache hit/miss, backup
|
|
add_sink("sqlmem.log", rotation="10 MB") # to a file
|
|
```
|
|
|
|
Set `SQLMEM_DEBUG=true` in `.env` to make the default level DEBUG when no explicit `level` is passed to `add_sink()`.
|
|
|
|
## Limitations
|
|
|
|
- In a multi-table (JOIN) query, every column must be qualified with its table or alias; unqualified columns raise `UnsupportedQueryError`.
|
|
- Tables are keyed by their base name — two tables with the same name in different schemas share one cache entry.
|
|
- No distributed cache backend (Redis etc.).
|
|
- No transactional consistency guarantees.
|
|
- Write operations (INSERT/UPDATE/DELETE) are always blocked.
|
|
|
|
## Dependencies
|
|
|
|
| Layer | Library |
|
|
|---|---|
|
|
| SQL parsing | `sqlglot` |
|
|
| Cache storage | `sqlite3` (stdlib) |
|
|
| Integration | SQLAlchemy 2.x |
|
|
| Logging | `loguru`, `python-dotenv` |
|
|
|
|
## License
|
|
|
|
MIT
|