Senior Data Engineer - Hadoop, PySpark & Airflow Architect
0 viewsGo Ahead ConsultingDubaiHybrid
10+ years
Posted 3 days ago
- Airflow
- Git
- Go
- Spark
About the Role
Go Ahead Consulting in Dubai, UAE, seeks a Senior Data Engineer to own the design, building, scaling, and maintenance of high-performance big data pipelines, Hadoop clusters, PySpark transformations, and Apache Airflow workflows integrated with GitHub.
Responsibilities
- Architect, build, and optimize big data pipelines and distributed data processing systems
- Manage Hadoop clusters, PySpark distributed jobs, and Apache Airflow DAGs
- Collaborate with data scientists, software engineers, and technical leadership across GCC client engagements
- Implement data governance, pipeline monitoring, and robust security standards
- Track and manage code versioning, reviews, and CI/CD via GitHub
Requirements
- Extensive experience in big data engineering and distributed data platforms
- Solid proficiency with Hadoop ecosystem tools and PySpark transformations
- Practical experience orchestrating complex workflows with Apache Airflow
- Strong version control practices with GitHub and CI/CD pipelines
- Proven ability to design low-latency, scalable data analytics platforms