Every Sunday I sit down and ask myself the same four questions: What went well? What would I do differently? What did I actually learn? What am I carrying into next week?
It’s a simple habit. But it’s changed how I grow as a data engineer more than any course or tutorial ever has. Here’s this week’s honest answer to each.
What Went Well: Automating the Unglamorous Stuff
Mid-week I finally automated a data ingestion job that had been running manually for months — a Python script wrapped in an Airflow DAG, pulling from an SFTP server, transforming data, and landing it in BigQuery. It saved 2 hours of manual work per day. That’s 500+ hours a year given back to the team.
The lesson: the boring automations often have the highest ROI. If you have a manual task that happens daily — that’s your first automation target.
What I’d Do Differently: Ask Sooner
I spent four hours debugging a dbt model. After four hours, I sent a Slack message. My teammate replied in five minutes with the fix. Four hours. Five minutes. Same result.
Next week: if I’m stuck for more than 30 minutes on something a teammate might know — I’m asking. Full stop.
What I Learned: Airflow Dynamic Task Mapping
I finally made peace with Airflow’s dynamic task mapping. Instead of defining a fixed number of tasks at DAG write time, you let the count be determined at runtime using .expand().
@dag(start_date=datetime(2026, 1, 1), schedule="@daily")
def dynamic_pipeline():
@task
def get_files() -> list:
return ["file_a.csv", "file_b.csv", "file_c.csv"]
@task
def process_file(filename: str):
print(f"Processing {filename}")
process_file.expand(filename=get_files())
dynamic_pipeline()What I’m Carrying Into Next Week: Collaborate First
The best moment of the week was a 15-minute call with a teammate who’d seen the problem before. In 15 minutes we covered what would’ve taken me 2 hours alone. My intention for next week: collaborate first, solo-debug second.
The Habit Behind the Reflection
If you don’t do a weekly reflection yet, try it — just 10 minutes on a Sunday. Four questions: What went well? What would I do differently? What did I learn? What’s my intention for next week? Growth as an engineer isn’t just about learning new tools. It’s about getting better at learning itself.
What’s one thing you learned this week? Drop a comment below.
— Pushpjeet Cholkar, Data Engineer