← WorkData engineering project · simulated scenario
An open-source, AWS-style data lake, from raw object to partitioned table
S3-compatible MinIO, distributed PySpark processing, Parquet, partitioned PostgreSQL loading and Great Expectations checks — sized for about a million simulated transactions a day.
- Role
- Design and build
- Period
- 2025
- Status
- Simulated volume — not a measured production throughput
- Tools
- PySparkMinIOParquetPostgreSQLGreat Expectations
01Overview
Building, from open-source parts, the equivalent of an AWS data lake for banking transactions: object storage, distributed processing, columnar formats, partitioned loading and quality checks.
Data engineering project on a simulated scenario.
What I personally did
- 01Designed and built the pipeline end to end.
02Constraints
- Everything local and open source, compatible with the equivalent managed services.
03Architecture
- 01
Raw
MinIO, S3-compatible.
- 02
Processing
PySpark, Parquet output.
- 03
Serving
Partitioned PostgreSQL.
- 04
Checks
Great Expectations at every stage.
Data lake layers.
04Implementation decisions
- 01
Separate the layers
Raw, cleaned and aggregated stay distinct: any stage can be replayed.
05Challenges
Saying “simulated”
The pipeline is sized for about a million transactions a day: a design target, not a measured production throughput.
06Evaluation
Great Expectations checks on schemas and values, on generated data.
07Outcomes
- A complete data lake architecture, reproducible locally.
08What I took from it
- 01Explicit data contracts make a pipeline maintainable.
09Evidence
