Production-style Data Engineering Pipeline Built Using AWS, Snowflake, and dbt to Process Airbnb Data
This project demonstrates a production-style data engineering pipeline built using AWS, Snowflake, and dbt to process Airbnb data from raw ingestion to analytics-ready datasets.The pipeline is designed with scalability and maintainability in mind, leveraging a medallion architecture (Bronze - Silver - Gold) to progressively refine data for business use. Tech Stack Cloud Storage - AWS S3 Data Warehouse - Snowflake Transformation - dbt Language - Python 3.12, SQL Version Control - Git Key Features • Incremental Processing for scalable data pipelines • SCD Type 2 Snapshots for historical tracking • Modular dbt Models for maintainability • Reusable Macros for dynamic transformations • Data Quality Tests for reliability • Analytics-Ready Tables for reporting What This Project Demonstrates • End-to-end data pipeline design • Medallion architecture implementation • Incremental processing at scale • Historical tracking with SCD Type 2 • Production-style dbt structure • Strong data modeling practices More details can be found in the repo below https://github.com/Opsy-Daizy/dbt_snowflake_aws_project_end_to_end



