06 / UNDER THE HOOD
Stop Polling Dagster From Cloudflare Workflows
We use a Cloudflare Workflow to kick off Dagster jobs. The Workflow handles the “something happened, now run the pipeline” part. Dagster does the heavy data work on our own machine.
The first version worked. Then the bill came.
Reaching a private Dagster
Our Dagster machine isn’t on the public internet, and we wanted it to stay that way. The Workflow reaches it through Workers VPC, so the Worker can talk to a private service without us opening a port.
Starting a job is a single call to Dagster’s GraphQL API: launch this job with this config, and get a run ID back.
The hard part is what comes after. A Dagster run can take seconds or a long time. The Workflow has to know when it’s done before it can carry on.
Version one: sleep and poll
The obvious approach is a loop. Sleep for a bit, ask Dagster whether the run has finished, and repeat until it has.
In a Workflow, each of those checks is a step, and Cloudflare bills Workflows per step as of August 2026. Sleeping costs no CPU, but every wake-up to check is another step. A short run takes a few steps. A long run, or a lot of runs every day, takes a lot of steps. Nobody notices until the invoice arrives. We didn’t.
The polling was doing nothing useful. Almost every check got back “still running”.
Version two: let Dagster call back
Cloudflare Workflows can wait for an event. Instead of polling, the Workflow launches the run and then calls step.waitForEvent. While it waits, it’s paused and doesn’t use any steps.
When the run finishes, Dagster sends a callback to a small Worker endpoint. Dagster’s run status sensors are built for reacting to a run succeeding or failing. That Worker finds the Workflow instance and calls sendEvent with the run’s result. The Workflow wakes up and continues.
Roughly, the Workflow side looks like this:
const runId = await step.do('launch dagster run', () =>
launchRun(env, { job: 'customer_report', config }),
);
const done = await step.waitForEvent('dagster run finished', {
type: 'dagster-run-finished',
timeout: '6 hours',
});
if (done.payload.status !== 'SUCCESS') {
throw new Error(`Dagster run ${runId} ended as ${done.payload.status}`);
}
And the callback Worker just forwards what Dagster tells it:
const instance = await env.REPORT_WORKFLOW.get(instanceId);
await instance.sendEvent({ type: 'dagster-run-finished', payload: { runId, status } });
Things to get right
- Pass the Workflow instance ID into the run. Dagster needs to know who to call back. We send it as part of the run config.
- Set a timeout on the wait. The default is 24 hours, and it can be up to a year. If the callback never arrives, because a run crashed hard or the machine went down, the Workflow should fail loudly, not wait a day.
- One event type per run. A Workflow that waits on several Dagster runs should give each its own event type, so a callback for one can’t wake the wait for another.
- Check the run belongs to the Workflow. Before calling
sendEvent, confirm the run ID in the callback is one this Workflow actually launched. - Call back on failure too. A finished run is a finished run, whether it succeeded or not. The Workflow decides what failure means.
- Protect the callback endpoint. Anyone who can reach it can wake your Workflows. Check a shared secret or signature before calling
sendEvent.
The lesson
If one system is waiting on another, have the slow side say when it’s done. Polling feels simpler, but every check costs something, even when nothing has changed.