Jobs by Fastorial
Back to all jobs

Senior Data Engineer (Hadoop, PySpark & Apache Airflow) | Go Ahead Consulting | Dubai, UAE

0 views
Go Ahead ConsultingDubaiHybrid

5+ years

Posted 3 days ago

  • Airflow
  • AWS
  • Devops
  • Docker
  • GCP
  • Git
  • Go
  • Kubernetes

Position Summary

Driving advanced big data engineering, distributed data pipelines, Hadoop architectures, PySpark analytics, and Apache Airflow orchestration in Dubai, United Arab Emirates, Go Ahead Consulting is actively seeking an experienced Senior Data Engineer for a permanent full-time engagement. In this vital data platform and engineering role, you will take full operational, tactical, and technical ownership of designing, building, scaling, and maintaining high-performance big data processing pipelines, Hadoop clusters, PySpark transformations, and automated Airflow DAG workflows integrated with GitHub version control.

Key Responsibilities

  • Architect, design, build, and scale robust big data processing pipelines and distributed data architectures utilizing Hadoop and PySpark
  • Develop, schedule, monitor, and optimize complex workflow orchestration DAGs using Apache Airflow
  • Manage version control, code reviews, branching strategies, and CI/CD integration workflows via GitHub
  • Process, transform, and clean large-scale structured and unstructured datasets to support downstream analytics and machine learning
  • Optimize PySpark jobs, Spark configurations, and Hadoop cluster resource utilization for maximum query performance
  • Collaborate proactively with cross-functional data science, software engineering, and product stakeholder squads
  • Ensure strict adherence to enterprise data security baselines, privacy regulations, and governance frameworks

Required Qualifications & Skills

  • Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or related discipline
  • Minimum of 5+ years of progressive professional hands-on experience in big data engineering and distributed systems
  • Strong operational proficiency in Hadoop ecosystem technologies (HDFS, Hive, YARN) and PySpark
  • Comprehensive working knowledge and practical hands-on experience with Apache Airflow for workflow orchestration
  • Proven expertise in collaborative software development and version control using GitHub
  • Solid practical grasp of data warehousing concepts, SQL tuning, ETL/ELT pipeline design, and performance optimization