Data engineering builds the systems that collect, move, clean and store data so that others can use it. Analysts and models are only as good as the data reaching them; the engineer makes it arrive on time, in a known shape, and with a record of where it came from.
Key concepts
- Pipelines: steps that extract data from sources, transform it and load it to a destination.
- Batch and streaming: process data in periodic chunks, or continuously as events arrive.
- Data quality: checks for missing values, duplicates, wrong types and unexpected changes.
- Orchestration: scheduling steps with dependencies, retries and alerts when one fails.
- Storage layers: raw data kept as it arrived, then cleaned and modelled tables built from it.
Vocabulary
- ETL and ELT: transform before loading, or load first and transform inside the warehouse.
- Data warehouse: a database organised for analysis over large histories.
- Idempotent job: one that can be rerun safely and gives the same result.
- Lineage: the record of where each dataset came from and what changed it.
What to study, in order
- SQL and a scripting language, such as Python, comfortably.
- Write a script that reads a file or an interface, cleans it and loads it into a database.
- Make it idempotent, so running it twice does not duplicate rows.
- Learn dimensional modelling: facts, dimensions and why analysts want them.
- Schedule the job with an orchestrator, with dependencies, retries and a failure alert.
- Add data quality checks, and try a small streaming version with a message queue.