Jobs by Fastorial
Back to all jobs

Lead Data Engineer

1 view
ORMAEPuneHybrid

6+ years

Posted 3 days ago

  • Azure
  • Docker
  • Git
  • Kubernetes
  • Platform Engineering
  • Python
  • Spark

Role Overview

We are looking for a Lead Data Engineer with strong experience building scalable batch, near-real-time, and streaming data platforms on Microsoft Azure. The role requires hands-on expertise in Python, PySpark, advanced SQL, Azure data services, and Medallion Architecture, along with a strong understanding of Apache Spark internals.

Key Responsibilities

  • Design and build batch, near-real-time, and streaming data pipelines on Azure.
  • Develop Bronze, Silver, and Gold data layers using Medallion Architecture.
  • Develop and optimize PySpark applications for large-scale transformation, cleansing, enrichment, reconciliation, and validation.
  • Optimize Spark jobs: partitioning, shuffles, joins, file sizes, and query performance.
  • Integrate data from REST APIs, SFTP, databases, files, and Azure Event Hubs.
  • Build reusable Python, PySpark, and SQL components, and CI/CD pipelines for data workloads.
  • Partner with platform engineering to deploy and operate containerized PySpark workloads on Kubernetes and AKS.

Required Skills

  • 6+ years of hands-on data engineering experience.
  • Strong experience with Microsoft Azure data platforms, Spark, Azure Data Lake, and Azure DevOps or Git.
  • Advanced Python, PySpark, and SQL skills.
  • Experience with batch, streaming, ETL, ELT, and event-driven processing patterns.
  • Nice to have: Docker, Kubernetes, and AKS containerized workloads.