Infrastructure / In Progress
Containerised Analytics Stack
Database, scheduler and BI layer running as containers with reproducible configuration — an analytics environment that can be destroyed and rebuilt on demand.
Started 2024-11-02 / Updated 2025-09-28
Analytics environments rot. Someone installs a package, someone edits a config, and six months later nobody can rebuild the thing that produces the numbers the business trusts.
This experiment treats the whole stack as disposable: every component defined in code, every rebuild identical to the last one.
The stack
Four services, one compose definition, no manual steps after clone.
- Postgres as the warehouse layer with schema managed by migrations.
- A scheduler running extract and transform jobs on a fixed cadence.
- A BI service reading from modelled views, never from raw tables.
- Object storage for raw exports so any load can be replayed.
The rule that made it work
Nothing is configured through a UI. If a dashboard, a connection or a job only exists because somebody clicked, it does not survive the next rebuild — so it does not count as part of the stack.
Where it gets hard
State. Reproducible compute is straightforward; reproducible data is not. Keeping raw exports immutable and replaying transformations turned out to be cheaper than trying to back up a mutable warehouse.
Setup
- Domain
- Infrastructure
- Definition
- Single compose file
- Rebuild time
- Minutes, from clone
- Config policy
- Code only, no UI state
Related notes
Want the detail behind this experiment?
If this overlaps with something you're building, I'm happy to share what worked and what didn't.