Lead Data Engineer
1 viewORMAEPuneHybrid
6+ years
Posted 3 days ago
- Azure
- Docker
- Git
- Kubernetes
- Platform Engineering
- Python
- Spark
Role Overview
We are looking for a Lead Data Engineer with strong experience building scalable batch, near-real-time, and streaming data platforms on Microsoft Azure. The role requires hands-on expertise in Python, PySpark, advanced SQL, Azure data services, and Medallion Architecture, along with a strong understanding of Apache Spark internals.
Key Responsibilities
- Design and build batch, near-real-time, and streaming data pipelines on Azure.
- Develop Bronze, Silver, and Gold data layers using Medallion Architecture.
- Develop and optimize PySpark applications for large-scale transformation, cleansing, enrichment, reconciliation, and validation.
- Optimize Spark jobs: partitioning, shuffles, joins, file sizes, and query performance.
- Integrate data from REST APIs, SFTP, databases, files, and Azure Event Hubs.
- Build reusable Python, PySpark, and SQL components, and CI/CD pipelines for data workloads.
- Partner with platform engineering to deploy and operate containerized PySpark workloads on Kubernetes and AKS.
Required Skills
- 6+ years of hands-on data engineering experience.
- Strong experience with Microsoft Azure data platforms, Spark, Azure Data Lake, and Azure DevOps or Git.
- Advanced Python, PySpark, and SQL skills.
- Experience with batch, streaming, ETL, ELT, and event-driven processing patterns.
- Nice to have: Docker, Kubernetes, and AKS containerized workloads.