Saltar al contenido ForoHackers

Data engineering: start here

Ingeniería de datos
#2907 · · Citar · Denunciar

Data engineering builds the systems that collect, move, clean and store data so that others can use it. Analysts and models are only as good as the data reaching them; the engineer makes it arrive on time, in a known shape, and with a record of where it came from.

Key concepts

  • Pipelines: steps that extract data from sources, transform it and load it to a destination.
  • Batch and streaming: process data in periodic chunks, or continuously as events arrive.
  • Data quality: checks for missing values, duplicates, wrong types and unexpected changes.
  • Orchestration: scheduling steps with dependencies, retries and alerts when one fails.
  • Storage layers: raw data kept as it arrived, then cleaned and modelled tables built from it.

Vocabulary

  • ETL and ELT: transform before loading, or load first and transform inside the warehouse.
  • Data warehouse: a database organised for analysis over large histories.
  • Idempotent job: one that can be rerun safely and gives the same result.
  • Lineage: the record of where each dataset came from and what changed it.

What to study, in order

  1. SQL and a scripting language, such as Python, comfortably.
  2. Write a script that reads a file or an interface, cleans it and loads it into a database.
  3. Make it idempotent, so running it twice does not duplicate rows.
  4. Learn dimensional modelling: facts, dimensions and why analysts want them.
  5. Schedule the job with an orchestrator, with dependencies, retries and a failure alert.
  6. Add data quality checks, and try a small streaming version with a message queue.

Related tags

#databases #sql #big-data #messaging

No has iniciado sesión, así que esto se publica como anónimo: no se guarda con ello ninguna cuenta, nombre ni dirección, y no puede llevar enlaces ni imágenes. Inicia sesión o únete para publicar con tu nombre. Los registros de nuestro servidor web guardan la dirección y la hora de cada petición hasta 14 días; para que la tuya no quede en ellos, usa Tor. Para escribir al equipo de forma confidencial, usa el buzón. Firmar como fuente habitual (una clave guardada solo en este navegador)

Enviar tarda unos segundos aquí: tu navegador resuelve antes una prueba de trabajo contra el spam.

Responder