All technical writing

06 / UNDER THE HOOD

Stop Polling Dagster From Cloudflare Workflows

We use a Cloudflare Workflow to kick off Dagster jobs. The Workflow handles the “something happened, now run the pipeline” part. Dagster does the heavy data work on our own machine.

The first version worked. Then the bill came.

Reaching a private Dagster

Our Dagster machine isn’t on the public internet, and we wanted it to stay that way. The Workflow reaches it through Workers VPC, so the Worker can talk to a private service without us opening a port.

Starting a job is a single call to Dagster’s GraphQL API: launch this job with this config, and get a run ID back.

The hard part is what comes after. A Dagster run can take seconds or a long time. The Workflow has to know when it’s done before it can carry on.

Version one: sleep and poll

The obvious approach is a loop. Sleep for a bit, ask Dagster whether the run has finished, and repeat until it has.

In a Workflow, each of those checks is a step, and Cloudflare bills Workflows per step as of August 2026. Sleeping costs no CPU, but every wake-up to check is another step. A short run takes a few steps. A long run, or a lot of runs every day, takes a lot of steps. Nobody notices until the invoice arrives. We didn’t.

The polling was doing nothing useful. Almost every check got back “still running”.

Version two: let Dagster call back

Cloudflare Workflows can wait for an event. Instead of polling, the Workflow launches the run and then calls step.waitForEvent. While it waits, it’s paused and doesn’t use any steps.

When the run finishes, Dagster sends a callback to a small Worker endpoint. Dagster’s run status sensors are built for reacting to a run succeeding or failing. That Worker finds the Workflow instance and calls sendEvent with the run’s result. The Workflow wakes up and continues.

Polling versus callback between a Cloudflare Workflow and DagsterBefore: the workflow launches a Dagster run over Workers VPC, then sleeps and checks the run status again and again, and every check is another step. After: the workflow launches the run, waits for an event, and Dagster calls back when the run finishes, which wakes the workflow.CF WorkflowDagster (private)CF WorkflowDagster (private)BEFORE: SLEEP AND POLLlaunchRun via GraphQLover Workers VPCstep.sleepstep.do: finished yet?"still running"step.sleepstep.do: finished yet?"still running"step.sleepstep.do: finished yet?"still running"… every check is another steplong run = many steps = the billAFTER: WAIT FOR THE CALLBACKlaunchRun via GraphQLover Workers VPCstep.waitForEventpaused, no pollingrun executesas long as it needscallbackWorker → sendEvent()workflow resumes
Polling paid for every check. The callback pays for one wake-up.

Roughly, the Workflow side looks like this:

const runId = await step.do('launch dagster run', () =>
  launchRun(env, { job: 'customer_report', config }),
);

const done = await step.waitForEvent('dagster run finished', {
  type: 'dagster-run-finished',
  timeout: '6 hours',
});

if (done.payload.status !== 'SUCCESS') {
  throw new Error(`Dagster run ${runId} ended as ${done.payload.status}`);
}

And the callback Worker just forwards what Dagster tells it:

const instance = await env.REPORT_WORKFLOW.get(instanceId);
await instance.sendEvent({ type: 'dagster-run-finished', payload: { runId, status } });

Things to get right

  • Pass the Workflow instance ID into the run. Dagster needs to know who to call back. We send it as part of the run config.
  • Set a timeout on the wait. The default is 24 hours, and it can be up to a year. If the callback never arrives, because a run crashed hard or the machine went down, the Workflow should fail loudly, not wait a day.
  • One event type per run. A Workflow that waits on several Dagster runs should give each its own event type, so a callback for one can’t wake the wait for another.
  • Check the run belongs to the Workflow. Before calling sendEvent, confirm the run ID in the callback is one this Workflow actually launched.
  • Call back on failure too. A finished run is a finished run, whether it succeeded or not. The Workflow decides what failure means.
  • Protect the callback endpoint. Anyone who can reach it can wake your Workflows. Check a shared secret or signature before calling sendEvent.

The lesson

If one system is waiting on another, have the slow side say when it’s done. Polling feels simpler, but every check costs something, even when nothing has changed.