A Workflow Orchestrator for Push and Email

Last year I got a challenge that looked small and was not. The agency where I worked needed any developer to be able to integrate the whole notification layer, push and email, in an easy and simple way.

If you have done this integration before, you know how it goes. It is repetitive and annoying, and it comes back in every project with small differences nobody remembers. I had worked with integrations like this in the past, and I knew the problem was not sending a notification. It was doing the same integration again and again.

So we set out to build a graphical interface where a developer could set up the fixed parts and the dynamic ones, like the tags that describe each user, and also build automated workflows. What mattered was who would change them later. The client, not us.

That weighed heavily because of the kind of product we built. The agency worked on mobile apps, and shipping an update for an app is far more bureaucratic than shipping one on the web. A new version, a store review, the wait. If every tweak to a campaign needed that, the platform was useless. The workflow had to live outside the app and change without a new integration or a review.

With that requirement, my first design idea was the obvious one. Take a workflow, walk the whole graph once, and create every message that would ever be sent. It broke as soon as the graph had a wait and a branch. A wait can last days, and a branch decides its path from data about the user, such as their tags or attributes. Those values can change at any moment, so a message planned on day one could reach someone who no longer belonged on that path.

I dropped the precomputed plan and built an engine that executes each node when its turn comes. Every node is a task. When a task runs, it reads the subscriber's current state, applies only its own logic, and creates the next task. That meant the engine could never be one big loop. It had to be a chain of small, independent steps, each one trusting only what it reads at that moment.

The chain is made of events inside the database. Starting an execution inserts a row, and a Postgres trigger expands it into one row per subscriber, resolving segments into people. Another trigger reads the immutable version of the graph and creates the first task. Every task insert fires one more trigger that maps the node type to an Edge Function and calls it over HTTP, with a ten second timeout and a token kept in the database vault. The function reads its task, does its one job, marks it, and inserts the next task, which fires the next trigger. Statuses roll upward on their own, from the task to the subscriber to the execution, so a stopped subscriber stops its own tasks and an execution closes when its last subscriber does. No process holds the whole workflow in memory, and nothing loops.

The graph supports the nodes a real campaign needs. Start and end, email and push, a condition, a multi-condition that picks the first matching case or falls back to a default, joins to merge branches, a stop node that ends one subscriber's path early, and waits. Each type has its own function and its own small contract, so a new type arrives as a new function in the same shape.

Sending is where the engine meets the outside world, and the outside world fails. Email and push go through queues, one pair per project, created when the project is. A consumer reads up to five hundred messages at a time and processes them in parallel. Each message gets three attempts. After a failure the task is marked as retrying and the message reappears after thirty seconds, and after the last attempt it is marked as failed and archived. Errors that retrying cannot fix, such as a device with no token, fail at once. For email, a final failure still moves the journey to the next node, because one undeliverable message should not freeze a person halfway through a campaign.

Waits are the other exception, because nothing happens at the moment a wait ends. A scheduled job in the database runs every minute, finds the waits whose time has passed and completes them, which hands control back to the chain. Push tasks have a validation function of their own that closes a task once every device it targeted has either succeeded or failed.

Workflow definitions are stored as immutable versions of the graph. An execution points to one version, so editing a workflow in the editor never changes a run that is already in flight.

A wait looks like the simplest node. It hides a surprising amount of date arithmetic. A duration is easy, since days and weeks are plain multiples of twenty-four hours. The harder kind is the wait until, such as the next time it is nine in the morning, the next Tuesday, a day of the month or a fixed date. Nine in the morning is not one moment. It is a different moment for each subscriber.

So every subscriber and every project carries an IANA timezone, and the engine uses the subscriber's first and the project's second. The editor refuses to add a wait until to a project that has none. Getting there meant migrating the database from naive timestamps to timestamps with a timezone, treating the existing rows as UTC. The conversion from a local clock time to an instant probes the possible offsets, and it has to settle two awkward cases. If daylight saving skips the hour you asked for, the wait resolves to the next valid minute of the same day. If the hour happens twice, it takes the first one. The editor needed the same logic to preview when a wait would resume, so a copy of it lives in the console, and a test suite pins the cases in the engine.

A condition reads tags, the labels that describe each subscriber and that the app or the workflow itself can add and remove. Several predicates on one node must all hold, and the schema rejects duplicates. The multi-condition node covers what a single condition cannot. Its cases are evaluated in order, the first match wins, and anything else takes a default branch, so a campaign can split an audience into three groups and still bring them back together. Because the check runs when the subscriber reaches the node, a tag added by an earlier node, or by the app an hour ago, is already counted. Conditions on segments I left for a second version.

Everything belongs to exactly one project. Subscribers, tags, segments, workflows, connectors, layouts and the queues themselves carry the project they belong to, which is what lets many teams share one platform without seeing each other.

Each project connects its own email provider, Resend or a custom SMTP server, and the credentials are resolved through the database at send time. Push goes out through Firebase Cloud Messaging with the project's own service account, stored encrypted. SMTP was the least glamorous part and the most delicate, because hosts block some ports, port 465 requires TLS and the sender has to belong to the mailbox domain or the provider rejects the message.

The graph is not the only way in. Apps talk to a public HTTP API to register devices, identify subscribers, tag them and start workflows, and servers can send push or email in batches. Broadcasts reach a segment, a tag or a list of subscribers. A subscriber can be anonymous until they log in, and the platform replaces the anonymous record with the identified one. Segments are computed from tag rules, so membership is recalculated when tags change instead of being uploaded. Everything lands in the same tables, so one history shows what a person received, whatever the channel or the origin.

Every endpoint validates its input with a schema, and a single middleware checks the key and its scope before anything else runs. The contract is published as an OpenAPI document, so clients and tools can read it.

The console is a Next.js app where the workflow is drawn on a graph. I wrote the layout code myself, in lanes, so that the branches of a multi-condition split and converge on screen. Each node has its own form and validation state, nodes can be copied through the system clipboard, and execution views show how many subscribers sit on each node, with previews of the push and the email as they were rendered. There is also an email layout editor that rejects scripts, iframes and inline event handlers, and a message history per subscriber.

Around the product sits the platform's own plumbing. Authentication emails go out through a hook that verifies the request and sends branded templates. Billing comes from Stripe, synced into Postgres so the console and the API read subscription state with plain queries, and features are limited by plan. Lifecycle emails and alerts to a Discord channel run on the same machinery. A support form in the console opens an issue in the team's repository with the user and the project attached.

The public face is three more pieces. A documentation and marketing site fed from a content studio, which also publishes the OpenAPI document, an llms.txt file generated from the content and the structured data for search. A status page that combines our own delivery metrics with the incident feeds of the providers we depend on. And the content studio itself, with schemas for the blog, the docs and the legal pages.

Once other teams would use the platform, access control stopped being a detail. I built a governance layer with roles, permissions and plans. Each permission says whether it applies to the console, the API or both. Plans carry capabilities, either limits or features, and the console reads them to gate screens and to show the upgrade path.

API keys came in types. A public key can only identify devices and subscribers and manage their tags, so it is safe to ship inside an app. A secret key belongs to a server and can send messages and start workflows. A restricted key carries an explicit list of scopes inside the ceiling of one of those two. Keys are stored hashed, shown once, and rotated or revoked without touching the others.

The largest part was the database. The first version ran its functions with elevated privileges and trusted the code around them. I moved them in waves to run as the caller, under row level security, and revoked execution privileges that nobody should have. Each wave came with smoke tests that run against a real local database, so the policies are exercised and not only read. During the transition the old membership tables and the new ones were kept in sync by a trigger. A legacy token scheme for older clients still exists, and removing it is a documented, planned job.

Everything lives in five repositories split out of an older monolith. The backend, the console, the marketing and docs site, the status page and the content studio. I standardized how they ship. The database is described as declarative schemas, and migrations are generated from the diff, reviewed and applied in CI, never edited by hand in the dashboard. Git hooks enforce commit conventions and run the type check, dead code detection, tests and lint before a push, and they block production pushes outside Monday to Thursday. Releases are generated from the commits, and the main branch is merged back into develop automatically.

A developer connects a project, uses the API or an SDK, and from then on the campaigns live in the console. The people who run them can change a workflow, publish a new version and have it apply to new executions without anyone shipping a release or waiting for a store review. Each message goes out with the data the subscriber has at that moment, through the channel and the provider the team chose. New node types, new key types and new providers arrived as small pieces that did not rewrite the rest.

Event-driven steps are easy to reason about one at a time, and hard to reason about as a whole. The cost shows up in observability, so I ended up building per-node metrics and an execution view to see where each subscriber actually was.

The second lesson is that rules which depend on memory drift. The security that lived in the database and the conventions enforced by hooks and tests held up better than anything that relied on people remembering.