Data pipelines rarely fail because the logic was wrong when it was written. They fail because a source changed and nothing was built to notice. Pipelines that survive change validate data at the boundary, stop loudly rather than continuing quietly, are safe to rerun, and keep movement separate from transformation.
Fail loudly rather than continue quietly
The instinct to make a pipeline resilient often produces the opposite of what is wanted: exceptions swallowed, missing values defaulted, bad rows skipped. Each of those decisions converts a visible failure into an invisible one.
The better default is to stop. A job that halts with a clear message gets fixed within the hour. A job that completes with a quarter of the rows dropped gets discovered when someone questions a report next month.
Validate at the boundary
Check data as it arrives, before any transformation. Expected columns present, types as declared, row counts within a plausible range, key fields unique, reference values recognised. These checks are cheap and they catch the majority of upstream surprises at the point where the cause is still obvious.
Record the results over time. A field whose null rate climbed steadily for three weeks is a different problem from one that broke overnight, and you can only see the difference if you kept the history.
Make reruns safe
Every pipeline gets rerun — after a fix, after a late-arriving file, after an outage. If a rerun duplicates rows or double-counts totals, people become reluctant to rerun, and problems stay unfixed longer than they need to.
Design for it explicitly: process a defined window, write in a way that replaces rather than appends, and make the operation safe to repeat. It is far less work to build this in at the start than to retrofit it during an incident.
Keep transformation separate from movement
Pipelines that reshape data while moving it are hard to debug, because a wrong number could have come from either half. Land the source data as it arrived, then transform in a separate, testable step. The storage costs little and it means you can reprocess without going back to the source system.
Write down what the output promises
Downstream consumers build on assumptions whether or not you documented them — the grain of the table, how late data is handled, when it refreshes. Stating those explicitly turns a silent dependency into an agreement, and gives you something concrete to check against when a change is proposed.
