Project Description: As part of my work integrating external data into internal reporting systems, I worked on a pipeline to ingest data from a third-party API into our AWS environment. The business relied on this data for analysis and reporting, but access was inconsistent and sometimes handled manually, which created gaps in data availability and made historical tracking difficult. To improve this, I built a pipeline that automates the extraction of data from the API using Python and stores it in Amazon S3 as part of a raw data layer. I used Apache Airflow to orchestrate the workflow, managing scheduling, retries, and task dependencies to ensure the pipeline runs reliably. I also used Terraform to provision the S3 bucket and related infrastructure, making the setup easy to reproduce and maintain across environments. The focus throughout was on keeping the pipeline simple, but dependable enough for downstream reporting and analytics use. Technologies Used: Python, Apache Airflow, AWS S3, Terraform, SQL Key Achievements: Replaced manual and inconsistent data retrieval with an automated ingestion pipeline Improved reliability of data delivery through scheduling, retry logic, and monitoring in Airflow Established a centralized raw data layer in S3, making data easier to track and reuse Ensured consistent infrastructure setup using Terraform, reducing manual configuration effort Increased confidence in externally sourced data used by downstream reporting teams

Project Description: As part of improving how externally sourced data is integrated into internal systems, I worked on a pipeline to ingest and structure football competition data from a third-party API for analytics and reporting purposes. The data was required for downstream analysis, but there was no consistent way of capturing and storing it internally. This created gaps in availability and made it difficult to work with historical data. To address this, I implemented an end-to-end pipeline that extracts data from the API using Python and orchestrates the workflow with Apache Airflow. The data is first loaded into a PostgreSQL database on AWS RDS, which serves as a staging layer to ensure structure and control before further processing. From there, I used Airbyte to replicate the data into Amazon Redshift, making it accessible for querying and analysis. The setup was containerized using Docker, with Airbyte running on a local Kubernetes environment (Minikube), allowing for a consistent and reproducible deployment. The focus of the implementation was on reliability, clarity of data flow, and maintaining a clean separation between ingestion, staging, and analytics layers. Technologies Used: Python, Apache Airflow, PostgreSQL (AWS RDS), Airbyte, Docker, Minikube (Kubernetes), Amazon Redshift Key Achievements: Delivered a structured pipeline for integrating external API data into the analytics environment Improved data availability by replacing inconsistent access patterns with an automated workflow Established a staging layer (RDS) to improve data organization and control before warehousing Enabled seamless data replication into Redshift for downstream reporting and analysis Implemented a reproducible and maintainable setup using containerization and infrastructure tooling

Project Description: As part of improving data availability for reporting and analytics within a core banking environment, I worked on a pipeline responsible for moving transaction data from the core banking application into the enterprise data warehouse (Oracle Exadata). The business relied heavily on transaction data for regulatory reporting, reconciliation, and operational insights, but the data flow into the warehouse needed to be more structured and reliable. To address this, I implemented a data pipeline using Apache NiFi to handle the ingestion and movement of transaction data from the source system into the data warehouse. NiFi was used to manage data flow, transformation, and routing, ensuring that data was transferred efficiently and in a controlled manner. Apache Airflow was used to orchestrate the overall process, managing scheduling, dependencies, and execution flow. This ensured that data ingestion jobs ran consistently and could be monitored effectively. The solution was deployed within an enterprise server environment, with a focus on reliability, traceability, and maintaining data integrity throughout the pipeline. Once ingested into Oracle Exadata, the data became available for downstream users, enabling reporting teams and analysts to build dashboards and derive insights from accurate and up-to-date transaction data. Technologies Used: Python, Apache NiFi, Apache Airflow, Oracle Exadata, SQL Key Achievements: Delivered a reliable pipeline for moving transaction data from core banking systems into the enterprise data warehouse Improved data availability for reporting, reconciliation, and analytics use cases Automated data ingestion and orchestration, reducing manual intervention and operational overhead Ensured consistent and traceable data flow using NiFi and Airflow Enabled downstream teams to access structured and timely transaction data for decision-making