Add README, fix column deduplication in parser, remove .env from git
This commit is contained in:
@@ -1,2 +1,126 @@
|
||||
# SQLmem
|
||||
|
||||
Transparent in-memory cache layer between SQLAlchemy and your database. Drop it in front of any SQLAlchemy engine — SELECT queries are served from a fast in-memory SQLite cache, writes pass through unchanged.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
Application (SQLAlchemy)
|
||||
│
|
||||
▼
|
||||
[ SQLmem Proxy ]
|
||||
┌──────────────────────────────┐
|
||||
│ SQL Parser │ → detects SELECT vs. write
|
||||
│ Column Registry │ → tracks which columns are cached per table
|
||||
│ Cache Manager (SQLite RAM) │ → stores data in memory
|
||||
│ Query Executor │ → cache hit / miss logic
|
||||
└──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Database (via original SQLAlchemy engine)
|
||||
```
|
||||
|
||||
On the first SELECT for a table, SQLmem fetches the required rows from the database and stores them in an in-memory SQLite instance. Subsequent queries for the same columns hit the in-memory cache with no database round-trip. When a query requests a column not yet in cache, SQLmem re-fetches the table with the expanded column set.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install sqlmem
|
||||
# or with Poetry
|
||||
poetry add sqlmem
|
||||
```
|
||||
|
||||
Requires Python 3.14.
|
||||
|
||||
## Quick start
|
||||
|
||||
```python
|
||||
from sqlmem import CachingEngine
|
||||
from sqlalchemy import create_engine, text
|
||||
|
||||
base_engine = create_engine("postgresql://user:pass@host/db")
|
||||
engine = CachingEngine(base_engine)
|
||||
|
||||
# Use exactly like a regular SQLAlchemy engine:
|
||||
results = engine.execute("SELECT id, name FROM users WHERE status = 'active'")
|
||||
for row in results:
|
||||
print(row["id"], row["name"])
|
||||
```
|
||||
|
||||
`execute()` returns a list of dicts. Results are compatible with standard iteration patterns.
|
||||
|
||||
## Cache behaviour
|
||||
|
||||
**Column accumulation** — SQLmem learns which columns your app needs at runtime, no upfront configuration required:
|
||||
|
||||
```
|
||||
Query 1: SELECT a, b FROM orders → cache miss → fetch orders(a, b) from DB
|
||||
Query 2: SELECT a, d FROM orders → new column d → re-fetch orders(a, b, d)
|
||||
Query 3: SELECT b FROM orders → cache hit, no DB query
|
||||
Query 4: SELECT * FROM orders → UnsupportedQueryError (wildcard not supported)
|
||||
Query 5: SELECT a FROM orders JOIN … → UnsupportedQueryError (JOIN not supported)
|
||||
```
|
||||
|
||||
**Writes are blocked** — INSERT, UPDATE, and DELETE raise `ReadOnlyError`. SQLmem is a read-only cache.
|
||||
|
||||
## Persistence
|
||||
|
||||
The in-memory cache is optionally persisted to `cache.db` on disk:
|
||||
|
||||
- **On startup**: if `cache.db` exists, it is loaded into memory.
|
||||
- **Hourly**: a background thread writes a snapshot to disk.
|
||||
- **On shutdown**: a final flush via `atexit` and SIGTERM handler.
|
||||
|
||||
Schema version is checked on load — if it does not match, the stale file is discarded and the cache is rebuilt from the database.
|
||||
|
||||
## Manual cache invalidation
|
||||
|
||||
```python
|
||||
engine.invalidate("orders") # drops the table from cache; next query re-fetches from DB
|
||||
engine.close() # flush to disk and shut down background thread
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Set via environment variables or a `.env` file:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `SQLMEM_DEBUG` | `false` | `true` enables DEBUG-level logging |
|
||||
| `SQLMEM_CACHE_DB` | `cache.db` | Path to the on-disk persistence file |
|
||||
| `SQLMEM_BACKUP_INTERVAL` | `3600` | Backup interval in seconds |
|
||||
|
||||
## Exceptions
|
||||
|
||||
| Exception | When raised |
|
||||
|---|---|
|
||||
| `ReadOnlyError` | INSERT, UPDATE, or DELETE statement |
|
||||
| `UnsupportedQueryError` | `SELECT *` or any JOIN |
|
||||
|
||||
```python
|
||||
from sqlmem import ReadOnlyError, UnsupportedQueryError
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
SQLmem uses [loguru](https://github.com/Delgan/loguru). Set `SQLMEM_DEBUG=true` for verbose output (every query, cache hit/miss, backup events). Default level is INFO.
|
||||
|
||||
## Limitations
|
||||
|
||||
- `SELECT *` and JOIN queries are not supported.
|
||||
- No distributed cache backend (Redis etc.).
|
||||
- No transactional consistency guarantees.
|
||||
- Write operations (INSERT/UPDATE/DELETE) are always blocked.
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Layer | Library |
|
||||
|---|---|
|
||||
| SQL parsing | `sqlglot` |
|
||||
| Cache storage | `sqlite3` (stdlib) |
|
||||
| Integration | SQLAlchemy 2.x |
|
||||
| Logging | `loguru`, `python-dotenv` |
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
Reference in New Issue
Block a user