Rendered at 19:59:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
thdaiminh 6 hours ago [-]
Any plans for ClickHouse? We run Postgres plus ClickHouse, and Postgres is the easy part to clone for a test env. The ClickHouse side is where it hurts, since the parts are large and immutable.
nu_nu11 6 hours ago [-]
[dead]
pranavs98 8 hours ago [-]
Is this like neon DB?
I tried using neon db for branching with agents, but its not that great. Ended up using different sqlite files per branch, and finally throwing away the ones not needed and consolidating all migrations eventually
Works like a charm. Claude can set it all up for you in 20m, including branch tags and management/gc scripts. It is very familiar with this pattern and uses it well. Clones are fast and cheap, and you can basically forget about it all once it’s set up.
SwtCyber 8 hours ago [-]
[flagged]
AlexMC 23 hours ago [-]
Super interesting, I had my own dumb script that clones the DB whenever I needed to work in a feature that requires some sort of DB mutation, some times with a lot of them going, so this does sound useful.
Having said that lately with the speed of AI assisted development I do this less and less...
t0mas88 1 days ago [-]
Postgres 18 has file_copy_method = clone for this, is that used underneath or did you implement your own own?
funcDropShadow 5 hours ago [-]
You can use the same ZFS snapshot cloning with PostgreSQL. I've been using for a few years to test tricky db migrations with production data or to debug difficult production issues we couldn't reproduce otherwise. You take a zfs snapshot, create a zfs clone, copy the postgres config file and postgres in single user mode on the clone. It is practically instantaneous to get a clone of a prod db.
killingtime74 24 hours ago [-]
This is at the file system level. It says it uses the ZFS. Also, this is for MySQL as well, not just Postgres.
nu_nu11 21 hours ago [-]
Thanks for asking! It doesn't use that. db-branch works at the filesystem level: it takes a ZFS snapshot of a cleanly shut down instance, clones it into a new dataset, and starts a separate DB process on top of it. So it doesn't depend on the Postgres version, and the same mechanism works for MySQL. Since everything happens at the filesystem layer, I think the same approach could extend to other databases and stateful services. (Thanks to killingtime74 for answering earlier, too.)
Database Lab is a great project, and since it's also ZFS-based, it's good validation that the approach works. My motivation was that most database branching tools seem to be built around Postgres, and I wanted something that works with MySQL too.
rzzzt 1 days ago [-]
Is it possible to merge changes from a clone/branch back to the "official copy"?
nu_nu11 21 hours ago [-]
Right now, branches are meant to be disposable: you can snapshot and restore, but there's no way to merge changes back into the source. I'd like to try it someday, but there are real challenges to solve, conflict resolution being the main one.
n_u 1 days ago [-]
This is interesting. Does it hook into Postgres and MySQL’s MVCC?
nu_nu11 11 hours ago [-]
While it currently supports MySQL and Postgres, the real strength of this architecture is that it doesn't rely on the internals of any specific database engine. Everything happens one layer down at the filesystem level, utilizing ZFS block-level CoW.
In theory, this means it could act as a universal branching engine for almost any stateful data process. You could instantly spin up disposable clones of massive Redis caches, Elasticsearch nodes, or even Kafka clusters for safe, isolated testing, and throw them away seconds later.
Of course, there are real-world constraints—running dozens of isolated stateful processes on a single host means you hit memory and I/O limits pretty fast. But assuming you have the hardware to handle the overhead, I'm really curious: what other stateful services or software would you want to instantly branch like this? I'd love to hear your ideas!
SwtCyber 8 hours ago [-]
[flagged]
cirrela_founder 1 days ago [-]
[flagged]
WanderZil 5 hours ago [-]
[dead]
hurtri 23 hours ago [-]
[flagged]
nu_nu11 1 days ago [-]
Hey HN! I've read HN for years but this is my first post.
db-branch uses ZFS copy-on-write to create, destroy, and reset MySQL and PostgreSQL clones in seconds, even for terabyte-scale databases. It's self-hosted and doesn't depend on any cloud vendor.
Background
Per-PR preview environments are common now, but the database is usually still a shared one. PlanetScale, Neon, and Aurora's fast clone solved this with Git-like branching, even for huge datasets: cut a branch in an instant, throw it away when you're done.
The catch is that these are managed services tied to specific cloud vendors. Cost can be a barrier, fully air-gapped (on-prem) environments have constraints, and sometimes you just want the same workflow on your laptop. db-branch is meant for those cases.
There's one more use case I care about: letting an AI agent such as Claude Code freely experiment with SQL against a database that looks like real data. Pointing an agent at production or staging directly is dangerous. With a branch, even if the agent runs something destructive (a DROP, a massive UPDATE), you just throw the branch away or roll back to a snapshot. Being a disposable DB sandbox for AI agents is one of the goals of this project.
Architecture
Storage and processes: Using ZFS snapshot/clone, db-branch instantly creates a dataset from a snapshot of a cleanly shut down MySQL/PostgreSQL instance, then starts a process for it using a systemd template unit, switching ports per branch. Each branch runs as its own independent DB process.
Dynamic routing: ProxySQL sits in front of the branch processes, so clients don't have to change port numbers. Branches are routed by connecting user or by a query comment (/db_branch=.../). This works for PostgreSQL as well as MySQL, using ProxySQL 3.x's PostgreSQL protocol support.
Clustering: db-branch clusters are built with Raft and ZFS send/recv, and routing inside the cluster is also done by ProxySQL.
Limitations and trade-offs
- How the base snapshot is taken: For consistency, snapshots are taken from a cleanly shut down DB. This doesn't mean stopping your production DB. The idea is to stop a read replica and snapshot that, or to use an instance restored from a regular dump/backup as the base.
- Memory is the real cost: CoW keeps disk usage low, but every branch is a separate DB process, so per-process memory grows with the number of branches. So we recommend keeping the DB-side caches (InnoDB buffer pool, PostgreSQL shared_buffers) small and giving most of the memory to the ZFS ARC. Clones share unmodified blocks, so data cached in the ARC is shared across branches, instead of every branch holding its own copy of the same data in its own DB cache. Even so, if you run dozens of branches at the same time, you need to plan the host's RAM accordingly.
- Heavy writes still cost disk: CoW only shares blocks that haven't been modified, so a branch that writes a lot consumes that much space.
Features
- Create and destroy MySQL and PostgreSQL branches
- Take snapshots of a branch as save points, and restore from them
- Upgrade verification for MySQL / PostgreSQL (upgrade version feature)
- SQL routing via ProxySQL
- Cluster setup
- [Commercial module] Recording of executed queries and restore to any point (Lite Plan / History Module)
To keep the project sustainable, the core branching engine and the clustering feature are released as fully open source, free of charge.
More advanced features, namely the per-query recording and restore (History Module), plus enterprise user management, SSO, and auditing, are planned as modules of a commercial plan. Only some of these are available so far, and you can use them for a one-month trial period.
(Whispering: if you're only playing around in the local Multipass environment, you can just tear down the VM and recreate it every month and keep using it basically forever... shhh )
yjftsjthsd-h 11 hours ago [-]
> For consistency, snapshots are taken from a cleanly shut down DB. This doesn't mean stopping your production DB. The idea is to stop a read replica and snapshot that, or to use an instance restored from a regular dump/backup
Is that necessary? I'm like 95% sure postgres is crash consistent, so you should be fine to snapshot it while running.
Works like a charm. Claude can set it all up for you in 20m, including branch tags and management/gc scripts. It is very familiar with this pattern and uses it well. Clones are fast and cheap, and you can basically forget about it all once it’s set up.
Having said that lately with the speed of AI assisted development I do this less and less...
In theory, this means it could act as a universal branching engine for almost any stateful data process. You could instantly spin up disposable clones of massive Redis caches, Elasticsearch nodes, or even Kafka clusters for safe, isolated testing, and throw them away seconds later.
Of course, there are real-world constraints—running dozens of isolated stateful processes on a single host means you hit memory and I/O limits pretty fast. But assuming you have the hardware to handle the overhead, I'm really curious: what other stateful services or software would you want to instantly branch like this? I'd love to hear your ideas!
db-branch uses ZFS copy-on-write to create, destroy, and reset MySQL and PostgreSQL clones in seconds, even for terabyte-scale databases. It's self-hosted and doesn't depend on any cloud vendor.
Background
Per-PR preview environments are common now, but the database is usually still a shared one. PlanetScale, Neon, and Aurora's fast clone solved this with Git-like branching, even for huge datasets: cut a branch in an instant, throw it away when you're done.
The catch is that these are managed services tied to specific cloud vendors. Cost can be a barrier, fully air-gapped (on-prem) environments have constraints, and sometimes you just want the same workflow on your laptop. db-branch is meant for those cases.
There's one more use case I care about: letting an AI agent such as Claude Code freely experiment with SQL against a database that looks like real data. Pointing an agent at production or staging directly is dangerous. With a branch, even if the agent runs something destructive (a DROP, a massive UPDATE), you just throw the branch away or roll back to a snapshot. Being a disposable DB sandbox for AI agents is one of the goals of this project.
Architecture
Storage and processes: Using ZFS snapshot/clone, db-branch instantly creates a dataset from a snapshot of a cleanly shut down MySQL/PostgreSQL instance, then starts a process for it using a systemd template unit, switching ports per branch. Each branch runs as its own independent DB process.
Dynamic routing: ProxySQL sits in front of the branch processes, so clients don't have to change port numbers. Branches are routed by connecting user or by a query comment (/db_branch=.../). This works for PostgreSQL as well as MySQL, using ProxySQL 3.x's PostgreSQL protocol support.
Clustering: db-branch clusters are built with Raft and ZFS send/recv, and routing inside the cluster is also done by ProxySQL.
Limitations and trade-offs
- How the base snapshot is taken: For consistency, snapshots are taken from a cleanly shut down DB. This doesn't mean stopping your production DB. The idea is to stop a read replica and snapshot that, or to use an instance restored from a regular dump/backup as the base. - Memory is the real cost: CoW keeps disk usage low, but every branch is a separate DB process, so per-process memory grows with the number of branches. So we recommend keeping the DB-side caches (InnoDB buffer pool, PostgreSQL shared_buffers) small and giving most of the memory to the ZFS ARC. Clones share unmodified blocks, so data cached in the ARC is shared across branches, instead of every branch holding its own copy of the same data in its own DB cache. Even so, if you run dozens of branches at the same time, you need to plan the host's RAM accordingly. - Heavy writes still cost disk: CoW only shares blocks that haven't been modified, so a branch that writes a lot consumes that much space.
Features
- Create and destroy MySQL and PostgreSQL branches - Take snapshots of a branch as save points, and restore from them - Upgrade verification for MySQL / PostgreSQL (upgrade version feature) - SQL routing via ProxySQL - Cluster setup - [Commercial module] Recording of executed queries and restore to any point (Lite Plan / History Module)
Try it
You can try it on a local Mac or Windows machine using Multipass: https://github.com/db-branch/db-branch#evaluate-locally-maco...
Monetization and license
To keep the project sustainable, the core branching engine and the clustering feature are released as fully open source, free of charge. More advanced features, namely the per-query recording and restore (History Module), plus enterprise user management, SSO, and auditing, are planned as modules of a commercial plan. Only some of these are available so far, and you can use them for a one-month trial period.
(Whispering: if you're only playing around in the local Multipass environment, you can just tear down the VM and recreate it every month and keep using it basically forever... shhh )
Is that necessary? I'm like 95% sure postgres is crash consistent, so you should be fine to snapshot it while running.