Skip to the content ForoHackers
Log inJoin

Data engineering: start here

Data engineering
#401 · Ophuscado · · Report

Data engineering builds the systems that collect, move, clean and store data so that others can use it. Analysts and models are only as good as the data reaching them; the engineer makes it arrive on time, in a known shape, and with a record of where it came from.

Key concepts

  • Pipelines: steps that extract data from sources, transform it and load it to a destination.
  • Batch and streaming: process data in periodic chunks, or continuously as events arrive.
  • Data quality: checks for missing values, duplicates, wrong types and unexpected changes.
  • Orchestration: scheduling steps with dependencies, retries and alerts when one fails.
  • Storage layers: raw data kept as it arrived, then cleaned and modelled tables built from it.

Vocabulary

  • ETL and ELT: transform before loading, or load first and transform inside the warehouse.
  • Data warehouse: a database organised for analysis over large histories.
  • Idempotent job: one that can be rerun safely and gives the same result.
  • Lineage: the record of where each dataset came from and what changed it.

What to study, in order

  1. SQL and a scripting language, such as Python, comfortably.
  2. Write a script that reads a file or an interface, cleans it and loads it into a database.
  3. Make it idempotent, so running it twice does not duplicate rows.
  4. Learn dimensional modelling: facts, dimensions and why analysts want them.
  5. Schedule the job with an orchestrator, with dependencies, retries and a failure alert.
  6. Add data quality checks, and try a small streaming version with a message queue.

Related tags

#databases #sql #big-data #messaging

Log in or join to reply.