06 / UNDER THE HOOD
One Durable Object Per Site
I recently built a module that lets users write their own monitoring rules. Pick some devices, pick a reading, set a range, and raise an alarm if the value stays outside it for long enough.
I nearly called it Sentry. It only watches, though. It doesn’t detect anything clever, so it became Observer (the naming story).
The interesting part isn’t the rules. It’s where they run. Every site gets one Durable Object on Cloudflare Workers, and that object checks every rule for that site.
Why one object per site
A rule is simple: is this value below the minimum, above the maximum, or outside the range? Then, has it been like that for 0, 15 or 60 minutes? Users choose how often it’s checked, anywhere from every 15 minutes to once a day.
The hard part is the state around the rule. You need the last value seen, when the breach started, whether an alarm is already open, and when the last check ran. If that state lives in a shared database, every check from every site competes for it.
A Durable Object gives you a single place for each site’s state, with its own storage, and only one thing changing it at a time. One site never waits for another. A site with many rules doesn’t slow down a site with a few. The object can also wake itself up for the next check, without anything outside having to schedule it. More on that below.
The site is the right unit because that’s how users think. Their rules, devices and alarms all belong to a site, so the access check and the object line up.
How you’d normally build it
The usual way is a cron job and a database. Every 15 minutes, cron wakes a worker. The worker loads every site’s rules, loops through them, reads the latest values, and writes the state and any alarms back to the same database.
That works, and it’s what most monitoring looked like before serverless. A Durable Object does the same job in a different shape.
What stays the same:
- A timer starts each check.
- The rules and the state are stored somewhere durable.
- The check itself is identical: compare the value to the range, then count how long it’s been out.
What’s different:
| Cron + database | One Durable Object per site | |
|---|---|---|
| Who keeps the time | One global schedule | Each object sets its own alarm |
| Where state lives | Shared rows for every site | Each object’s own storage, next to the code |
| Concurrency | Locks, or hoping two runs don’t overlap | Each object is single-threaded, so no shared lock |
| A slow site | Delays the whole loop | Only delays itself |
| Scaling | A bigger box, or sharding the loop | More objects, one per new site |
| Idle cost | The server runs whether or not anything is due | Nothing runs until an alarm fires |
| Looking across sites | One SQL query | Hard: you ask each object, or keep a summary elsewhere |
That last row is the real trade-off. Durable Objects are great at “this site, right now” and bad at “every site, as a report.” That’s one more reason the database stays the source of truth for the rules. Reporting goes there, and checking happens in the objects.
Serverless horror
Serverless removes the server you’d otherwise babysit. It doesn’t remove the ways things can go wrong. It just moves them to the invoice.
- Loops become bills. On a server, a polling loop costs nothing extra. On serverless, every wake-up is billed. I learned that the hard way with a Workflow that polled Dagster.
- Retries multiply. A failing check that’s retried, on every site, every interval, is a lot of invocations doing nothing useful. That’s why the sync here backs off, up to an hour.
- Unbounded work is unbounded cost. Let users add as many devices as they like and the bill grows with them. That’s what the series limits below are for.
- Nothing remembers anything. A plain Worker forgets everything between requests, so it’s tempting to read and write the database on every call. A Durable Object is the fix for state that belongs to one thing.
- You can’t SSH in. When something misbehaves, logs and the runtime view are all you have. Build that view early.
One alarm, many schedules
This is my favourite part of Durable Objects, and it’s not obvious at first.
Each object gets one alarm. Not one per rule, just one. Calling setAlarm() again replaces it. But a site has many rules, and each has its own interval: one every 15 minutes, another once a day.
So the object keeps its own timetable, and the timetable is just a table. Durable Objects come with their own SQLite storage, so every rule is a row with a next_check_at column. The single alarm is always set to the earliest next_check_at. When it fires, the object:
- selects the rows that are due now,
- checks those rules,
- writes each one’s new
next_check_atback to its row, - sets the alarm to the smallest
next_check_atleft in the table, and goes back to sleep.
A simplified version:
async alarm() {
const now = Date.now();
const due = this.ctx.storage.sql
.exec('SELECT id, definition FROM rules WHERE next_check_at <= ?', now)
.toArray();
try {
for (const rule of due) await this.check(rule);
} finally {
// Move every due rule forward, even if a check failed, so one bad read
// can't stop the schedule.
for (const rule of due) {
this.ctx.storage.sql.exec(
'UPDATE rules SET next_check_at = ? WHERE id = ?',
nextDue(rule, Date.now()), rule.id,
);
}
const { next } = this.ctx.storage.sql
.exec('SELECT MIN(next_check_at) AS next FROM rules').one();
if (next === null) await this.ctx.storage.deleteAlarm();
else await this.ctx.storage.setAlarm(next);
}
}
Two small details made a real difference:
- Stagger the rules.
nextDuegives each rule a fixed offset inside its interval, worked out from a hash of its ID. Twenty rules every 15 minutes don’t all fire on the same minute and hit the data source at once. - Share the check budget fairly. A single wake-up can only do so much work. The object remembers where it stopped and starts from the next series on the following alarm, so one noisy device can’t starve the rest.
It’s a tiny scheduler inside each site. Adding or changing a rule means writing its row and setting the alarm again. When a site has no rules left, the object deletes its alarm and its storage, and costs nothing.
The interval doesn’t have to be fixed, either. setAlarm() takes any timestamp, so each wake-up can decide when the next one should be. A rule could check more often while a breach is open, to confirm it sooner. It could back off when nothing has changed for hours, or retry a failed read in a minute instead of waiting for the next slot. Cron gives you one fixed beat. The alarm lets every site set its own.
Compare that with cron, which wakes up every 15 minutes whether anything is due or not, then asks a shared database what to do. Here the timetable sits next to the code, in the object’s own table.
The database owns the rules
There are two copies of every rule. The API’s database holds the one the user saved. The Durable Object holds the one it’s checking.
I made the database the source of truth. Saving a rule writes it to the database first, with a new version number and a pending sync state. Then the API pushes that version to the site’s object. If the push works, the state becomes synced. If it fails, it stays pending, and a background sync retries it, backing off up to an hour so a broken site doesn’t get hammered every minute.
Deletes work the same way. A deleted rule stays pending until the object confirms it has stopped checking it.
Each rule also keeps a revision history: who changed what, and when. When an alarm fires at 3am, the first question is always “did someone change the rule?”
A version number keeps them honest
The runtime view shows users what the object is actually seeing: last values, last check, and any open breach. That data comes from the Durable Object, not the database.
That creates a trap. Say a user removes a device from a rule. The database has version 5. The object, mid-sync, is still on version 4 and still has readings for the removed device. Show that, and the user sees data they just removed.
So every runtime read compares versions. If the object’s version doesn’t match the saved one, the API returns a 409 and tells the user to refresh after sync. It also checks that every series in the response belongs to the current rule. If the object is unreachable, the API returns a 503 instead of making something up.
Reading runtime state never syncs, evaluates or changes alarms. A page view shouldn’t be able to change what’s being monitored.
Bound the work, because every series costs
Every device and reading pair is a series the object reads, evaluates and may write on every check. That makes cost easy to predict, and easy to blow up.
So there are hard limits: 200 series per rule and 1,000 per site. They’re checked when the rule is saved, not discovered later on the bill. If you’ve read my Workflows post, you know why I now care about bounding things up front.
Test before you trust it
Users can dry run a rule against up to six hours of past data before saving it. The object runs the same check logic over history and shows what would have fired. It’s capped at six hours and a 60-second timeout, and a too-big test gets a clear “try a shorter range or fewer devices” message.
Nobody knows the right threshold until they see how often it would have fired.
What I’d keep
- Make one store the source of truth, and let the other one catch up.
- Put a version on everything that syncs, and check it on every read.
- Pick the partition users already think in.
- Set limits on the work when the rule is saved.
- Let people test against the past before it pages them in the future.