All technical writing

06 / UNDER THE HOOD

One Durable Object Per Site

I recently built a module that lets users write their own monitoring rules. Pick some devices, pick a reading, set a range, and raise an alarm if the value stays outside it for long enough.

I nearly called it Sentry. It only watches, though. It doesn’t detect anything clever, so it became Observer (the naming story).

The interesting part isn’t the rules. It’s where they run. Every site gets one Durable Object on Cloudflare Workers, and that object checks every rule for that site.

Why one object per site

A rule is simple: is this value below the minimum, above the maximum, or outside the range? Then, has it been like that for 0, 15 or 60 minutes? Users choose how often it’s checked, anywhere from every 15 minutes to once a day.

The hard part is the state around the rule. You need the last value seen, when the breach started, whether an alarm is already open, and when the last check ran. If that state lives in a shared database, every check from every site competes for it.

A Durable Object gives you a single place for each site’s state, with its own storage, and only one thing changing it at a time. One site never waits for another. A site with many rules doesn’t slow down a site with a few. The object can also wake itself up for the next check, without anything outside having to schedule it. More on that below.

The site is the right unit because that’s how users think. Their rules, devices and alarms all belong to a site, so the access check and the object line up.

How you’d normally build it

The usual way is a cron job and a database. Every 15 minutes, cron wakes a worker. The worker loads every site’s rules, loops through them, reads the latest values, and writes the state and any alarms back to the same database.

That works, and it’s what most monitoring looked like before serverless. A Durable Object does the same job in a different shape.

Cron and a shared database compared with one Durable Object per siteTraditional: one cron job wakes one worker, which loops over every site and reads and writes rules and state in one shared database. Durable Objects: each site has its own object with its own storage and its own alarm, which wakes it, runs the check, saves state and sets the next alarm. The database still owns the rules and syncs them to each object.TRADITIONAL: CRON + SHARED DATABASEcron: every 15 minone workerfor each site → for each rule → checkSite ASite BSite Cone shared databaserules + state + alarms for every sitelocks, contention, one slow site delays the restDURABLE OBJECTS: ONE PER SITEdatabase: the saved rulessource of truth, pushes each new versionsync v5Site A objectown alarmcheck its rulesown storageset next alarmsync v5Site B objectown alarmcheck its rulesown storageset next alarmsync v5Site C objectown alarmcheck its rulesown storageset next alarmeach site wakes, checks and saves aloneno shared lock, no global loop, no server idling
Same check, same rules. What changes is who keeps the time and where the state lives.

What stays the same:

  • A timer starts each check.
  • The rules and the state are stored somewhere durable.
  • The check itself is identical: compare the value to the range, then count how long it’s been out.

What’s different:

Cron + database One Durable Object per site
Who keeps the time One global schedule Each object sets its own alarm
Where state lives Shared rows for every site Each object’s own storage, next to the code
Concurrency Locks, or hoping two runs don’t overlap Each object is single-threaded, so no shared lock
A slow site Delays the whole loop Only delays itself
Scaling A bigger box, or sharding the loop More objects, one per new site
Idle cost The server runs whether or not anything is due Nothing runs until an alarm fires
Looking across sites One SQL query Hard: you ask each object, or keep a summary elsewhere

That last row is the real trade-off. Durable Objects are great at “this site, right now” and bad at “every site, as a report.” That’s one more reason the database stays the source of truth for the rules. Reporting goes there, and checking happens in the objects.

Serverless horror

Serverless removes the server you’d otherwise babysit. It doesn’t remove the ways things can go wrong. It just moves them to the invoice.

  • Loops become bills. On a server, a polling loop costs nothing extra. On serverless, every wake-up is billed. I learned that the hard way with a Workflow that polled Dagster.
  • Retries multiply. A failing check that’s retried, on every site, every interval, is a lot of invocations doing nothing useful. That’s why the sync here backs off, up to an hour.
  • Unbounded work is unbounded cost. Let users add as many devices as they like and the bill grows with them. That’s what the series limits below are for.
  • Nothing remembers anything. A plain Worker forgets everything between requests, so it’s tempting to read and write the database on every call. A Durable Object is the fix for state that belongs to one thing.
  • You can’t SSH in. When something misbehaves, logs and the runtime view are all you have. Build that view early.

One alarm, many schedules

This is my favourite part of Durable Objects, and it’s not obvious at first.

Each object gets one alarm. Not one per rule, just one. Calling setAlarm() again replaces it. But a site has many rules, and each has its own interval: one every 15 minutes, another once a day.

So the object keeps its own timetable, and the timetable is just a table. Durable Objects come with their own SQLite storage, so every rule is a row with a next_check_at column. The single alarm is always set to the earliest next_check_at. When it fires, the object:

  1. selects the rows that are due now,
  2. checks those rules,
  3. writes each one’s new next_check_at back to its row,
  4. sets the alarm to the smallest next_check_at left in the table, and goes back to sleep.

A simplified version:

async alarm() {
  const now = Date.now();
  const due = this.ctx.storage.sql
    .exec('SELECT id, definition FROM rules WHERE next_check_at <= ?', now)
    .toArray();

  try {
    for (const rule of due) await this.check(rule);
  } finally {
    // Move every due rule forward, even if a check failed, so one bad read
    // can't stop the schedule.
    for (const rule of due) {
      this.ctx.storage.sql.exec(
        'UPDATE rules SET next_check_at = ? WHERE id = ?',
        nextDue(rule, Date.now()), rule.id,
      );
    }
    const { next } = this.ctx.storage.sql
      .exec('SELECT MIN(next_check_at) AS next FROM rules').one();
    if (next === null) await this.ctx.storage.deleteAlarm();
    else await this.ctx.storage.setAlarm(next);
  }
}

Two small details made a real difference:

  • Stagger the rules. nextDue gives each rule a fixed offset inside its interval, worked out from a hash of its ID. Twenty rules every 15 minutes don’t all fire on the same minute and hit the data source at once.
  • Share the check budget fairly. A single wake-up can only do so much work. The object remembers where it stopped and starts from the next series on the following alarm, so one noisy device can’t starve the rest.

It’s a tiny scheduler inside each site. Adding or changing a rule means writing its row and setting the alarm again. When a site has no rules left, the object deletes its alarm and its storage, and costs nothing.

The interval doesn’t have to be fixed, either. setAlarm() takes any timestamp, so each wake-up can decide when the next one should be. A rule could check more often while a breach is open, to confirm it sooner. It could back off when nothing has changed for hours, or retry a failed read in a minute instead of waiting for the next slot. Cron gives you one fixed beat. The alarm lets every site set its own.

Compare that with cron, which wakes up every 15 minutes whether anything is due or not, then asks a shared database what to do. Here the timetable sits next to the code, in the object’s own table.

Cron ticks compared with a Durable Object alarm over one hourA site has rule A every 30 minutes and rule B every 60 minutes. Cron wakes five times in an hour, and twice nothing is due. The Durable Object alarm wakes three times, only when a rule is due, and each time sets the next alarm to the earliest next due time.One site, two rules: A every 30 min, B every 60 minCRON: EVERY 15 MINUTES0mA B15mnothing due30mA45mnothing due60mA B5 wake-ups, 2 for nothingeach one asks the database what's dueDURABLE OBJECT: ONE ALARMsetAlarm(MIN)setAlarm(MIN)A B0mA30mA B60m3 wake-ups, all usefulTHE TIMETABLE IS A TABLEState after the wake-up at 0m:ruleeverynext_check_atA30 min30mB60 min60mMIN(next_check_at) = 30m → setAlarm(30m)each wake-up: SELECT due rows, check, UPDATE their next_check_at
Cron asks "is anything due?" on a fixed beat. The alarm already knows.

The database owns the rules

There are two copies of every rule. The API’s database holds the one the user saved. The Durable Object holds the one it’s checking.

I made the database the source of truth. Saving a rule writes it to the database first, with a new version number and a pending sync state. Then the API pushes that version to the site’s object. If the push works, the state becomes synced. If it fails, it stays pending, and a background sync retries it, backing off up to an hour so a broken site doesn’t get hammered every minute.

Deletes work the same way. A deleted rule stays pending until the object confirms it has stopped checking it.

Each rule also keeps a revision history: who changed what, and when. When an alarm fires at 3am, the first question is always “did someone change the rule?”

A version number keeps them honest

The runtime view shows users what the object is actually seeing: last values, last check, and any open breach. That data comes from the Durable Object, not the database.

That creates a trap. Say a user removes a device from a rule. The database has version 5. The object, mid-sync, is still on version 4 and still has readings for the removed device. Show that, and the user sees data they just removed.

So every runtime read compares versions. If the object’s version doesn’t match the saved one, the API returns a 409 and tells the user to refresh after sync. It also checks that every series in the response belongs to the current rule. If the object is unreachable, the API returns a 503 instead of making something up.

Reading runtime state never syncs, evaluates or changes alarms. A page view shouldn’t be able to change what’s being monitored.

Bound the work, because every series costs

Every device and reading pair is a series the object reads, evaluates and may write on every check. That makes cost easy to predict, and easy to blow up.

So there are hard limits: 200 series per rule and 1,000 per site. They’re checked when the rule is saved, not discovered later on the bill. If you’ve read my Workflows post, you know why I now care about bounding things up front.

Test before you trust it

Users can dry run a rule against up to six hours of past data before saving it. The object runs the same check logic over history and shows what would have fired. It’s capped at six hours and a 60-second timeout, and a too-big test gets a clear “try a shorter range or fewer devices” message.

Nobody knows the right threshold until they see how often it would have fired.

What I’d keep

  • Make one store the source of truth, and let the other one catch up.
  • Put a version on everything that syncs, and check it on every read.
  • Pick the partition users already think in.
  • Set limits on the work when the rule is saved.
  • Let people test against the past before it pages them in the future.