Airflow setup and data pipelines
Apache Airflow installed, configured, and running readable DAGs with dependency-aware scheduling, retries, alerting, and documented backfills.
Airflow setup means installing and configuring Apache Airflow, then writing the DAGs that run your recurring data work on a schedule you can see and trust. It replaces the pile of cron jobs, one-off scripts, and manual Monday-morning refreshes that most growing businesses accumulate. It is for teams whose data jobs have dependencies, and who currently find out about failures from someone asking why a number looks wrong.
The problem
Recurring jobs start simple. One script, one schedule. The trouble arrives when jobs start depending on each other and nobody can see the order they run in or which one broke.
- Cron runs the transform at 2am whether or not the extract that feeds it finished.
- A job fails silently and the dashboard quietly serves last week numbers.
- Retrying means someone SSHing into a box and running a command from memory.
- The schedule lives in three places: crontab, a cloud scheduler, and a colleague head.
- Backfilling a week of data is a manual afternoon nobody wants to volunteer for.
What we build
A working Airflow environment
Managed (MWAA, Cloud Composer, Astronomer) or self-hosted, chosen for what your team can actually operate rather than what looks impressive.
DAGs a human can read
Tasks named after what they do, dependencies that reflect reality, and no clever metaprogramming that makes the graph impossible to follow six months later.
Dependency-aware scheduling
Transforms wait for extracts. Dashboard refreshes wait for models. Nothing runs on a hopeful timer.
Retries and alerting
Sensible retry policy for transient failures, and alerts that fire to where your team already looks. The goal is that failures are visible without being noise.
Backfill and re-run paths
Documented, tested ways to re-run a date range without hand-editing anything or duplicating rows.
Connections to the rest of the stack
Fivetran syncs, dbt runs, warehouse loads, notifications, and downstream jobs coordinated in one place instead of scattered across tools.
What you get at handoff
- DAGs in your repository, deployed through your own workflow.
- A runbook covering each failure mode and the recovery step for it.
- Access and credential setup documented, not held by us.
- Training for whoever will add the next DAG.
- Notes on what we deliberately did not automate, and why.
Stack
Orchestration
Apache Airflow, MWAA, Cloud Composer, Astronomer
Transformation
dbt, SQL, Python
Warehouses
Snowflake, BigQuery, Postgres, Redshift
Sources
Fivetran, Custom connectors, APIs, S3 and GCS
Alerting
Slack, Microsoft Teams, Email, Jira
Related services
FAQ
Do we actually need Airflow, or is cron enough?
Cron is genuinely fine for a handful of independent jobs that nobody depends on. Airflow earns its keep when jobs have dependencies, when you need to see run history, when failures need to be visible, or when backfilling is something you do more than once a year. If none of those apply, we will tell you to keep cron.
Managed or self-hosted?
Managed unless there is a specific reason not to. MWAA, Cloud Composer, and Astronomer all remove the part of Airflow that eats time: upgrades, scaling, and the scheduler. Self-hosting makes sense when cost at your volume is genuinely lower or when data residency rules require it.
How does this fit with Fivetran and dbt?
Airflow is the coordinator, not a replacement. A typical DAG triggers or waits on a Fivetran sync, runs the dbt models that depend on it, checks the results, then refreshes dashboards and notifies the team. Each tool keeps doing what it is good at.
What does alerting actually look like?
Failures post to wherever your team already works, with enough context to act: which task, which run, which date range, and the relevant log excerpt. Repeated failures should not spam the channel, and recoveries should be announced. We have written about that pattern in detail in the pipeline alerting journal post.
Can our team maintain this after you leave?
That is the point of writing DAGs plainly. The handoff includes a walkthrough, a runbook, and at least one DAG added by your team with us watching, so the first one they write alone is not the first one they have ever seen.
Ready to scope Airflow Setup?
Send the workflow, tool stack, or reporting problem. We will tell you what should be automated, what should stay manual, and what is worth building first.