Author: pushpjeet

There’s a quiet frustration that many data engineers share but rarely talk about.

You’ve spent years mastering Spark, dbt, Airflow, and Python. You’ve built pipelines that process millions of rows. You’ve solved complex data quality issues that nobody else could crack. And yet — the career opportunities you want aren’t coming fast enough.

Here’s the uncomfortable truth: technical skills get you in the door, but your personal brand is what keeps it open.

In 2026, the data engineering job market is competitive. Companies aren’t just looking for engineers who can write great code — they’re looking for engineers who can communicate, influence, and lead. And your personal brand is the signal they use to evaluate that.

What Is a Personal Brand (And Why Should a Data Engineer Care)?

A personal brand is simply the impression people have of you professionally. It’s what comes up when someone Googles your name. It’s the posts you share on LinkedIn. It’s the articles you write, the problems you solve publicly, and the way you explain complex concepts to others.

• Get you noticed by recruiters who aren’t posting jobs publicly
• Position you as a thought leader in your niche (Kafka? dbt? streaming pipelines?)
• Help you command higher salaries because you’re seen as an expert, not just a candidate
• Build a network that sends opportunities your way

The best part? You don’t need to be an influencer. You just need to be consistent.

Step 1: Pick Your 3 Core Topics

Pick 3 topics that sit at the intersection of what you know deeply, what you enjoy talking about, and what your target audience cares about. For me, those are: Data Engineering (pipelines, architecture, tools), Python for data workflows, and AI tools for engineers.

Step 2: Show Your Work, Not Just Your Results

Instead of waiting until you’ve built the perfect data pipeline to talk about it, share your journey. Instead of posting “Just shipped a new ETL pipeline!”, try: “We had 3-hour data latency. Here’s how I rebuilt it using Kafka + Spark Streaming to get it under 5 minutes — and the mistake I almost made.” That version teaches something. It builds trust.

Step 3: Choose Your Platform and Post Consistently

LinkedIn is the most powerful platform for data engineers right now. A personal blog gives you a home base you own — and helps with SEO. One post per week, every week, for six months will do more for your career than ten posts in a burst followed by three months of silence.

Step 4: Engage, Don’t Just Broadcast

Leave thoughtful comments on posts from engineers you admire. Reply to every comment on your own posts. Share other people’s insights with your own take added. The engineers who grow fastest are the ones genuinely engaging — not just broadcasting.

Step 5: Be Patient and Track What Works

Building a personal brand is a long game. Expect the first 90 days to feel slow. Track what resonates — which posts get comments, which topics drive profile views. As a data engineer, you’re uniquely positioned to be analytical about your content strategy. Use that superpower.

Your Action Plan for This Week

1. Update your LinkedIn headline to reflect your specialty
2. Write one post about a problem you solved recently
3. Leave five genuine comments on posts by engineers you respect
4. Start a notes document to capture ideas for future posts

The engineers who will thrive in the next five years aren’t just the ones with the most technical skills. They’re the ones who can communicate their value, build trust at scale, and show up consistently. Your code already speaks for itself. Now it’s time to let your voice do the same.

— Pushpjeet Cholkar, Data Engineer

1. dbt + LLM Macros: AI-Powered Transformations

What it is: dbt (data build tool) integrates with large language models to auto-generate column descriptions, test cases, and documentation from your SQL models.

Why it matters: Writing documentation and tests is the least glamorous part of data engineering — and the most skipped. LLM-powered macros change that equation. You can prompt dbt to generate YAML documentation for an entire model in seconds.

-- Run dbt's AI doc generation on your model
dbt docs generate --select my_model

With tools like dbt-osmosis and emerging LLM integrations, you can now propagate column descriptions automatically across your DAG.

2. MLflow: The Experiment Tracker You Already Need

What it is: MLflow is an open-source platform for managing the ML lifecycle — experiment tracking, model registry, and model serving.

Why data engineers need it: Even if you never train a model yourself, your pipelines feed them. When a model degrades, the first question is: “Did the training data change?” MLflow gives you the audit trail to answer that.

import mlflow

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_metric("rmse", 0.84)
    mlflow.log_artifact("model.pkl")

3. Feast: The Feature Store That Stops Pipeline Duplication

What it is: Feast is an open-source feature store that lets you define, store, and serve ML features consistently across training and production environments.

The problem it solves: Without a feature store, teams rebuild the same feature logic in multiple places. Models trained on one version of a feature get served predictions from a slightly different version — this is called training-serving skew, and it silently kills model accuracy.

from feast import FeatureStore

store = FeatureStore(repo_path=".")
features = store.get_online_features(
    features=["user_stats:purchase_7d_avg"],
    entity_rows=[{"user_id": 1001}]
).to_dict()

4. Great Expectations + Anomaly Detection: Quality Beyond Rules

Why the combination matters: Rule-based expectations catch what you already know to look for. Anomaly detection catches distribution shifts, sudden value spikes, or gradual drift that no rule anticipated.

import great_expectations as gx

context = gx.get_context()
validator = context.sources.pandas_default.read_csv("orders.csv")
validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_between("order_amount", min_value=0, max_value=50000)
results = validator.validate()

5. Vertex AI / SageMaker Pipelines: Orchestrating ML Like a Data Engineer

If you’ve used Apache Airflow, these tools map cleanly to concepts you already know: DAGs, steps, inputs/outputs, artifacts — just applied to ML workflows.

from kfp import dsl

@dsl.pipeline(name="training-pipeline")
def training_pipeline(data_path: str):
    preprocess_task = preprocess_op(data_path=data_path)
    train_task = train_op(data=preprocess_task.outputs["processed_data"])
    evaluate_task = evaluate_op(model=train_task.outputs["model"])

The Big Picture

In 2026, a pipeline doesn’t just move data from A to B. It moves data from raw sources to clean features to trained models to reliable predictions to business outcomes. Every link in that chain needs engineering.

Build both sides of the stack. That’s where the real leverage is.

— Pushpjeet Cholkar, Data Engineer

Every data engineer has a story.

It usually starts the same way: someone needed a quick data pull, so you wrote a Python script. It worked. Then it got scheduled. Then it fed a dashboard. Then the VP of Sales started refreshing that dashboard every morning before their 9am standup.

Your “quick script” just became critical infrastructure — and nobody updated the README.

This is one of the most common patterns in data engineering, and it’s also one of the most dangerous. When pipelines are built like throwaway scripts, they become time bombs. They break at the worst moments, they’re impossible to debug, and they’re terrifying to hand off to someone else.

The fix? Start treating your data pipelines like products.

1. Version Control Everything — Not Just the Code

Most engineers version-control their Python files. But your pipeline is more than just code. Version control your SQL transformations, dbt models, schema definitions, DAG definitions, and infrastructure configs. When a schema changes without a Git commit, you lose traceability.

Practical tip: Use a monorepo structure for your data platform. Tools like dbt make this natural — every model, test, and doc block lives in version control.

2. Write Data Tests, Not Just Code Tests

Unit tests catch bugs in your logic. Data tests catch bugs in your data — and in data engineering, the data is usually where the real surprises hide. Most production data issues aren’t caused by broken code — they’re caused by an upstream source sending nulls, a date field switching formats, or a join key returning duplicate rows.

Test for not-null checks on critical columns, uniqueness constraints on primary keys, accepted values for categorical columns, referential integrity between tables, and row count anomalies. dbt has built-in generic tests for all of the above.

3. Build Observability From Day One

If your pipeline fails silently, does it even matter that it failed? The answer is yes — and your stakeholders will make it very clear when they figure out their dashboard is two days stale.

Observability means alerting on failures, data freshness monitoring, row-level audit logs, and lineage tracking. The rule of thumb: you should know your pipeline is broken before your stakeholders do. Always.

4. Document the WHY, Not Just the WHAT

Code explains what it does. Documentation should explain why it does it. Six months from now, when someone needs to modify a complex transformation, they don’t need to know that this column is a LEFT JOIN — they need to know why it’s a LEFT JOIN and what business logic it encodes.

Write dbt model descriptions that explain business context, keep an Architecture Decision Record file in your repo for major design choices, and update docs as part of your PR review process — not as an afterthought.

5. Treat Pipeline Failures as Incidents

When your production pipeline breaks, it’s not just a bug — it’s a business incident. Log it with full error context. Alert the right people — not just the on-call engineer, but the data consumers who are affected. Fix it with a proper root cause analysis, not just a git revert. Then post-mortem it.

Teams that run post-mortems on data incidents ship more reliable pipelines over time, because they learn from failures instead of repeating them.

The Product Mindset Shift

All of these practices come down to one mental shift: think about the people downstream from your pipeline. Ask yourself before every pipeline you build: Who depends on this data? What breaks if this pipeline fails at 2am? How will I know it’s working correctly tomorrow? Would a new engineer understand this in 6 months?

If you can answer those questions confidently, you’re not just writing scripts anymore. You’re building data infrastructure that lasts.

Have a “temporary” pipeline that’s been running for years? Share your story in the comments 👇

— Pushpjeet Cholkar, Data Engineer

Most data engineers can write a Spark job. But writing one that’s actually fast? That’s where things get interesting.

I’ve spent years working on large-scale data pipelines, and time and again I see the same performance mistakes show up in Spark jobs — even from experienced engineers. The good news: most of them are easy to fix once you know what to look for.

Here are 5 Spark optimization tricks that have saved me hours of compute time and a lot of frustrated debugging.

1. Stop Triggering Unnecessary Shuffles

Shuffles are Spark’s most expensive operation. When a shuffle happens, data is redistributed across the network — taking time, memory, and I/O. Operations that trigger shuffles: groupBy(), wide join(), distinct(), repartition().

The fix: use broadcast joins for small tables.

from pyspark.sql.functions import broadcast
result = large_df.join(broadcast(small_lookup_df), on="id", how="left")

2. Get Your Partition Count Right

Target ~128MB per partition. Too few = idle cores. Too many = scheduling overhead.

df.rdd.getNumPartitions()
df = df.repartition(200)
df = df.coalesce(50)

Use coalesce() when reducing, repartition() when increasing or needing even distribution.

3. Cache DataFrames You Use More Than Once

Spark’s lazy evaluation means it recomputes DataFrames from scratch each time an action is triggered — unless you cache them.

df = spark.read.parquet("s3://my-bucket/data/")
df.cache()
count = df.count()
filtered = df.filter(...)  # uses cache

4. Switch to Kryo Serialization

Kryo can be 2–10x faster than Java’s default serializer. One config line:

spark = SparkSession.builder     .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")     .getOrCreate()

5. Actually Read the Spark UI

The Spark UI (at http://localhost:4040) shows exactly where time is wasted — skewed partitions, slow stages, missed caches. Check the Stages tab, DAG visualization, and Executors tab. A task taking 5 minutes while 199 others take 10 seconds? That’s partition skew — fix it by salting your join key.

Putting It All Together

Checklist for every new Spark job: avoid shuffle-heavy operations where possible, check partition count, cache reused DataFrames, enable Kryo, and inspect the Spark UI after a test run. Optimizing Spark is systematic — find the bottleneck, understand it, fix it.

— Pushpjeet Cholkar, Data Engineer

Let me paint you a picture.

Two data engineers join the same company on the same day. Same skills. Same stack. Same team.

Eighteen months later, one is a senior engineer with inbound recruiter messages and a growing online following. The other is still waiting for their “turn” at the next performance cycle.

What separated them? It wasn’t the code. It was visibility.

The Visibility Problem in Data Engineering

Data engineering is one of the most impactful roles in a modern tech company. You build the infrastructure that powers product decisions, revenue models, and machine learning systems. Without you, data scientists are staring at empty Jupyter notebooks.

And yet — data engineers are often the least visible people on the technical team. Our output lives in Airflow DAGs and dbt models that only your team appreciates. This invisibility has a career cost. A personal brand solves this.

What Personal Branding Actually Means for Engineers

First, let’s kill the cringe. Personal branding doesn’t mean becoming a LinkedIn influencer. For a data engineer, personal branding means one thing: making your expertise legible to the right people.

The 5 High-ROI Moves for Building Your Brand as a Data Engineer

1. Write About What You Just Solved

Every week, you solve at least one problem that took you longer than it should have. A tricky dbt macro. A Spark memory tuning issue. A confusing Airflow dependency. Write 300 words about how you solved it and post it on LinkedIn or your blog. You will help dozens of engineers who are Googling the exact same problem.

2. Narrate Your Architecture Decisions

Most engineers document the what. Almost nobody documents the why. Why did you choose Kafka over Kinesis? Why did you pick Iceberg over Delta Lake? These decisions are gold. Write them up — share the best ones externally. This positions you as someone who thinks about engineering, not just implements it.

3. Teach One Thing Every Week

You know something that would be useful to someone at an earlier stage of their career. Teaching doesn’t require a YouTube channel. It can be a 5-minute Loom walkthrough, a reply to someone’s LinkedIn question, or a short “Today I Learned” post. Every time you teach, you reinforce your own learning and your reputation simultaneously.

4. Be Consistent Over Being Viral

The first 10 posts feel pointless. By post 50, you start getting DMs. By post 100, recruiters are finding you. Set a sustainable cadence — one LinkedIn post per week, one blog post per month. Two years of that consistency will transform your career.

5. Engage, Don’t Just Broadcast

The fastest way to grow your network is to add value in other people’s conversations first. Find the data engineers you respect. Comment thoughtfully on their posts. Share their work with your own take. This is how you get on people’s radars before you have a big following.

A Week-1 Challenge

This week, do one of these: write a LinkedIn post about a technical problem you solved recently, reply thoughtfully to three posts from data engineers you admire, or write up a short internal doc explaining an architecture decision you made. That’s it. No newsletter or podcast required yet.

Final Thought

You spent years learning SQL, Python, Spark, dbt, Airflow, and a dozen other tools. You’ve built systems that process millions of rows of data. Don’t let that expertise stay invisible. Your career is also a product. Build it with the same intention you bring to your pipelines.

Start this week. One post. One doc. One comment. You’ve got this. 🚀

— Pushpjeet Cholkar, Data Engineer