Built a real-time data pipeline with Databricks and Python using Bronze, Silver, and Gold layers, which reduced data latency by 35%. Designed and implemented a structured data flow that separates raw, cleaned, and business-ready data, making fraud analysis easier and more reliable. Handled late and duplicate events in streaming data to ensure clean and consistent fraud detection results in real time.

Built a serverless ETL pipeline with AWS Kinesis Firehose and Lambda, helping reduce infrastructure and maintenance costs by 50%. Improved analyst query speed by 4x by transforming raw event logs into partitioned Parquet files using PySpark on Amazon EMR. Used Terraform to automate the full AWS infrastructure setup, reducing deployment time from days to a few minutes and keeping environments consistent.

Engineered a streaming pipeline using Kafka, PySpark, and PostgreSQL to process live stock trades, reducing data delay and enabling near real-time analytics for market insights. Processed and transformed streaming stock market data into structured datasets for reporting and analysis, enabling faster and more consistent access to market insights for executives, clients, and stakeholders to support better decision-making. Built monitoring and observability using Prometheus and Grafana to track pipeline health and failures, improving system monitoring and enabling faster issue detection.