Jobs by Fastorial
Back to all jobs

Senior Data Engineer

3 views
FinTop ConsultingDubaiHybrid

10+ years

Posted 2 days ago

  • Azure
  • Devops
  • Kafka
  • Python
  • Spark

Job Title: Senior Data Engineer

Location: Dubai | UAE | Malaysia | Armenia | India

Role Overview:

Fintop Consulting has partnered with one of the pioneers in the financial industry and is looking for a hands-on Senior Data Engineer with strong expertise in Azure Databricks, PySpark, SQL, and Kafka-based streaming technologies.

The role involves building and managing scalable data pipelines, handling complex data integrations, and ensuring data quality and reliability across a high-volume financial data environment. The ideal candidate should have strong experience in trading or brokerage data, production support, performance optimization, and end-to-end data pipeline ownership.

Key Responsibilities:

  • Design, develop, and maintain data ingestion and transformation pipelines using Azure Databricks, PySpark, SQL, and Delta Live Tables (DLT).
  • Build and manage Kafka-based streaming integrations using Confluent Cloud and its connectors.
  • Develop and enhance API-based ingestion frameworks and third-party data integrations.
  • Own data pipelines from source ingestion through Bronze and Silver layers, ensuring data quality, reliability, and SLA adherence.
  • Troubleshoot production issues, perform root cause analysis and implement permanent fixes.
  • Optimize Databricks workloads, Spark performance and cloud infrastructure costs.
  • Implement data governance and deployment practices using Unity Catalog, Databricks Workflows and CI/CD.
  • Collaborate with business and technical stakeholders to define data requirements and integration standards.
  • Mentor junior engineers and contribute to code reviews, technical design and engineering best practices.

Required Skills & Experience:

  • 10+ years of hands-on experience in data engineering.
  • Strong production experience with Azure Databricks, Apache Spark, PySpark, SQL and Delta Lake.
  • Practical experience with Delta Live Tables, Unity Catalog and Databricks Workflows.
  • Strong understanding of data modelling, incremental loads, CDC, partitioning and performance tuning.
  • Hands-on experience with Apache Kafka, preferably Confluent Cloud and Kafka Connect.
  • Experience building API-based data ingestion frameworks.
  • Proven experience optimizing Databricks performance and operational costs.
  • Strong understanding of trading and brokerage data, including trades, positions, balances, deals and accounts.
  • Working knowledge of CRM, client, partner, payments, reconciliation and financial data.
  • Experience supporting production data platforms and resolving critical data issues.

Good to Have:

  • Experience with MetaTrader (MT4/MT5) back-office data.
  • Exposure to trading bridges and liquidity platforms.
  • Experience with enterprise CRM systems.
  • Background in brokerage, trading, or regulated financial services.