← WorkMSc DataScale · data integration and quality
Measuring quality before promising it: 38 indicators on energy data
Consumption, population and territorial reference data reconciled by IRIS zone: daily Airflow orchestration, Talend and Apache Hop transforms, 38 quality metrics tracked before and after cleaning.
- Role
- Pipeline design and build
- Period
- 2025 — 2026
- Status
- Academic project
- Tools
- PythonAirflowTalendApache HopSQL
01Overview
Building energy-consumption indicators by IRIS zone and socio-professional category from heterogeneous sources: inconsistent formats, postal codes, units and missing values.
MSc DataScale — data integration and quality.
What I personally did
- 01Designed the daily Airflow orchestration and the Python control scripts.
- 02Computed 38 quality metrics over five tables, keeping before/after cleaning metadata.
- 03Transformed the data in Talend and Apache Hop (mapping, unions, joins, aggregations, conformance rules).
02Constraints
- Heterogeneous public sources, territorial reference data to reconcile.
03Architecture
- 01
Ingestion
Consumption, population, IRIS reference data.
- 02
Profiling
38 metrics measured before any correction.
- 03
Transformation
Talend and Apache Hop, conformance rules.
- 04
Reporting
A before / after comparison dashboard.
Principle architecture.
04Implementation decisions
- 01
Measure first
Metrics are computed before and after each step: quality improvement becomes an observable result, not a claim.
05Challenges
Codes and units
Postal codes and units differing between sources: explicit, traceable normalisation.
06Evaluation
Comparison of the 38 metrics before and after cleaning, per table.
07Outcomes
- A reproducible pipeline separating raw, cleaned and aggregated data.
08What I took from it
- 01Data quality is managed with indicators, like the rest of the system.
09Evidence