Spark, dbt, and Airflow: How to Use All Three, Plus the Advanced Patterns That Keep Pipelines Alive

Every data engineer eventually lands on the same question: “When do I use Spark vs dbt vs Airflow?”

If you’ve asked yourself this, you’re not alone. These three tools form the backbone of a modern data stack — but the confusion about when to use which one leads to some of the messiest pipeline architectures I’ve ever seen.

In this post, I’m going to break down each tool’s role, show you where they overlap (and where they absolutely don’t), and walk you through a practical architecture pattern that actually scales. Then we’ll go a level deeper: the advanced patterns that keep Spark, dbt, and Airflow pipelines alive in production.

The Short Answer (Before We Dive In)

• Spark = distributed compute for large-scale data processing
• dbt = SQL-based transformation layer inside your data warehouse
• Airflow = orchestrator that schedules and monitors jobs

They’re not competitors. They’re teammates. The trick is giving each one the right job.

Apache Spark: Your Heavy-Lifting Engine

Apache Spark is a distributed computing framework designed to process massive amounts of data fast. We’re talking terabytes or petabytes, spread across a cluster of machines working in parallel.

When should you reach for Spark? Use it when you have raw, unstructured data coming from Kafka, S3, or HDFS. Use it when your data volume makes single-machine processing impractical. Use it when you need complex transformations before data hits your warehouse, or when you’re doing streaming ingestion alongside batch processing.

Spark is excellent at the ingestion and raw processing phase. It can read from almost any source, apply heavy transformations in PySpark or Scala, and write results to your data lake or warehouse.

What Spark is NOT: a scheduler, an orchestrator, or a transformation layer inside your warehouse. Using Spark to run light transformations on structured warehouse data is overkill — that’s dbt’s territory.

dbt: The Transformation Layer Your SQL Deserves

dbt (data build tool) changed how data engineers think about transformations. Instead of scattered SQL scripts with names like final_v3_FINAL.sql, dbt gives you a structured, version-controlled, testable transformation framework.

Here’s what makes dbt powerful: Modularity lets you write reusable SQL models that reference each other. Testing lets you define schema tests (not null, unique, accepted values) that run automatically. Documentation auto-generates a data catalog from your models. Lineage lets you visualize how data flows from source to final table.

dbt runs inside your warehouse — Snowflake, BigQuery, Redshift, Databricks. It doesn’t move data; it transforms data that’s already there.

A Quick dbt Example

-- models/marts/fact_orders.sql
WITH orders AS (
    SELECT * FROM {{ ref('stg_orders') }}
),
customers AS (
    SELECT * FROM {{ ref('stg_customers') }}
)
SELECT
    o.order_id,
    o.order_date,
    c.customer_name,
    o.total_amount
FROM orders o
LEFT JOIN customers c ON o.customer_id = c.customer_id

That ref() function is dbt magic — it builds the dependency graph automatically, so dbt knows to run stg_orders and stg_customers before fact_orders.

What dbt is NOT: a job scheduler, a data ingestion tool, or a substitute for Spark on large raw datasets.

Apache Airflow: The Conductor of Your Pipeline

Airflow is a workflow orchestration platform. Its job is simple but critical: run the right jobs, in the right order, at the right time — and tell you when something goes wrong.

You define workflows as DAGs (Directed Acyclic Graphs) in Python. A typical daily DAG looks like this: Spark ingests raw data → dbt transforms it → dbt tests validate it. Clean, readable, and version-controlled.

The #1 Airflow mistake I see: Running heavy data processing logic inside Airflow operators. PythonOperators with 10,000-row Pandas loops, inline SQL queries that run for hours — this kills your Airflow workers. Airflow schedules work. It doesn’t do the heavy work itself.

The Architecture Pattern That Works

Here’s the pattern I’ve used on production pipelines handling hundreds of millions of rows daily: Airflow triggers a daily DAG → Spark ingests raw data to the data lake → Airflow triggers dbt → dbt transforms inside the warehouse → dbt tests validate data quality → BI tools and downstream consumers read clean data.

Each layer has one responsibility. Airflow handles scheduling and monitoring, Spark handles scale, dbt handles structured transformations. When something breaks, you know exactly where to look.

Common Mistakes to Avoid

1. Running Pandas in Airflow operators. Heavy compute belongs in Spark, not inside Airflow. If your DAG tasks take more than a few minutes, move the logic to a Spark job and trigger it from Airflow.

2. Using dbt for raw data ingestion. dbt reads from what’s already in your warehouse. It doesn’t pull from APIs, Kafka, or flat files. Use Spark, Fivetran, or a custom ingestion job for that.

3. Treating Spark as a scheduler. Spark has no built-in job scheduling or dependency management. Airflow is always needed to coordinate when Spark jobs run.

4. No dbt tests. If you’re not running dbt test, you’re flying blind. Schema tests catch broken pipelines before your stakeholders do.

Advanced Patterns That Keep Data Pipelines Alive

Let me paint you a picture.

It’s 3am. Your on-call alert fires. A critical dashboard that the executive team reviews every morning is showing stale data. You log in, trace the issue, and find that your Spark job silently died four hours ago because a single executor ran out of memory on a skewed join.

The fix takes ten minutes. The damage — missed SLAs, a frantic Slack thread, a post-mortem doc — takes much longer to clean up.

This is not a story about bad engineers. It’s a story about what happens when good engineers build for the happy path and forget to design for the real world. Most data pipelines fail not because of bad code, but because of bad assumptions.

Advanced data engineering with Spark, dbt, and Airflow isn’t about knowing more syntax. It’s about knowing where each tool breaks — and building systems that survive anyway.

Spark: The Hidden Cost of Data Skew

Spark is incredibly powerful, but it has one Achilles heel that catches even experienced engineers off guard: data skew.

When you’re joining two large datasets and one key appears disproportionately often (say, a NULL value, or a single mega-customer ID), Spark will assign all that work to one partition — and one executor. While other executors finish quickly and sit idle, the overloaded one struggles. Your job hangs. Eventually it times out or runs out of memory.

How to diagnose it: Open the Spark UI and look at the task duration for your shuffle stages. If one task is taking 10x longer than the median, you have skew.

How to fix it — Option 1, repartition before the join:

df = df.repartition(200, "join_key")

Option 2 — Salting for severely skewed keys: Add a random salt integer to the skewed dataset’s join key, then explode the same range of salts on the other side, and join on the salted key. This distributes the hot key across multiple partitions artificially — not elegant, but effective.

dbt: Test First, Transform Second

Most engineers treat dbt tests as an afterthought. Senior engineers do the opposite: they write the tests first.

Before you write a single line of SQL in a dbt model, ask yourself: What should always be true about the output of this model? Which columns should never be null? Which columns should always be unique? What referential integrity should exist between tables?

Then codify those assumptions in your schema.yml before you write the model. When your dbt tests run in CI before every merge, bad data assumptions get caught in development — not at 3am in production.

Advanced — custom generic tests: For business-logic validation that goes beyond the built-in tests, write custom generic tests in your tests/generic/ folder. For example, an assert_positive_value test that selects any rows where a column is zero or negative — then apply it to any revenue or quantity column across all your models.

Airflow: Build for Late Data, Not Just Scheduled Data

Airflow’s default mental model is simple: a DAG runs at a scheduled time, does its work, and completes. But production data is rarely that cooperative. APIs go down. Upstream jobs run late. Files arrive hours after they should.

Use sensors instead of assuming. Instead of hardcoding a start time and hoping the upstream data exists, use an Airflow sensor to wait for it — an S3KeySensor, a SqlSensor, or a custom ExternalTaskSensor. Set mode='reschedule' so the sensor releases its worker slot between pokes, avoiding worker pool exhaustion.

Set SLAs on your critical DAGs. Wire up an sla_miss_callback on your DAG that posts to Slack. Set an sla timedelta on your default_args so that if a task hasn’t finished within the expected window, you get alerted — before your stakeholders notice stale dashboards.

Make every DAG idempotent. Every DAG run should produce the same result whether it runs once or ten times. Use INSERT OVERWRITE instead of INSERT INTO. Use date-partitioned tables and overwrite the partition. Use MERGE statements for slowly changing dimensions. Build idempotency in from day one, not as a fix after your first duplicate incident.

The Senior Data Engineer Mindset

The tools — Spark, dbt, Airflow — are just tools. What makes a senior data engineer is the mindset:

• Design for failure. Ask “what happens when this breaks?” before “will this work?”
• Validate at every layer. dbt tests on transformations. Spark checks on ingestion. Airflow sensors on orchestration.
• Instrument everything. Logs, metrics, SLA alerts. If you can’t observe it, you can’t fix it at 3am.
• Build for re-runs. Idempotent pipelines mean recovery is a single button press.

Wrapping Up

Spark, dbt, and Airflow are genuinely complementary. Once you understand each tool’s lane, using them together feels natural — and your pipelines become dramatically more maintainable.

The key mental model: Airflow is the conductor. Spark is the muscle. dbt is the translator.

Give each tool its role and stay disciplined about not crossing the lanes. And remember: the pipelines that survive in production aren’t the cleverest ones. They’re the ones built by engineers who assumed things would go wrong — and planned accordingly.

What’s one pattern from this post that you’re going to implement this week? Drop a comment below — I read every one of them.

— Pushpjeet Cholkar, Data Engineer