samlq/tidewater
PublicBack-pressure for job queues - a queue that says no before the database does.
- queues
- postgres
- Go
- 5.1k stars
- 290 forks
Infrastructure engineer. I make deploys dull.
Back-pressure for job queues - a queue that says no before the database does.
Change data capture for SQLite - stream every write to Kafka, NATS or a file.
zsh, tmux and neovim - set up in one command on a fresh machine.
Globex
- today
Initech
-
KTH Stockholm
-
Back-pressure explained with a coffee shop and a very long line.
What we changed in on-call, and what we deleted.
Notes from building sqlite-cdc.
Back-pressure for job queues - a queue that says no before the database does.
go get github.com/samlq/tidewater
go: added github.com/samlq/tidewater v2.4.0
tidewater bench --rps 5000
accepted 4,812/s · refused 188/s · p99 41 ms
At Initech our job queue accepted everything, and the database paid for it: every spike became a backlog, every backlog a page at 3 a.m. Retries made it worse.
I wrote it on weekends, then at work: a queue that measures how long the database takes to answer and stops taking jobs before it drowns. Producers get a clear "not now" with a retry time instead of a timeout.
Five years and 290 forks later it runs in front of Postgres at companies of every size. At Globex it took queue incidents from eleven a quarter to none.
Back-pressure explained with a coffee shop and a very long line.