Jobs by Fastorial
Back to all jobs

Senior Data Engineer - Databricks / Fabric & Azure Data Factory

0 views
Chosen TalentzCoimbatoreOnsite

5-10 years · ₹12-18 LPA

Posted 6 days ago

  • Azure
  • Git
  • Python

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks or Microsoft Fabric and Azure Data Factory (ADF) to design, develop, and maintain scalable data engineering solutions.

The ideal candidate should have experience in building data pipelines, implementing ETL/ELT processes, transforming and integrating data from multiple sources, and ensuring data quality and reliability. Experience with SQL, Python, data modelling, and cloud-based data platforms is essential.

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF) and either Databricks or Microsoft Fabric.
  • Build and maintain data ingestion pipelines using Azure Data Factory (ADF), integrating data from approved files, APIs, and other source systems. Manage incremental loads, schema changes, failed loads, error handling, and recovery.
  • Develop and maintain Bronze, Silver, and Gold data layers using Azure Databricks or Microsoft Fabric, implementing scalable data transformation logic.
  • Implement source-to-target mappings, reconciliation, incremental processing, historical data handling, and dimensional models.
  • Ensure pipeline data quality through validation rules, completeness and freshness checks, exception handling, reconciliation, and remediation tracking. Implement automated data quality tests using dbt or platform-native capabilities.
  • Follow engineering standards and best practices in pipeline development, participate in code and design reviews, and coordinate technical decisions.
  • Maintain technical metadata, transformation documentation, and lineage in coordination with MDM and Data Governance engineers.

Required Skills

  • 5+ years in data engineering and ETL development.
  • Strong hands-on SQL, Databricks or Microsoft Fabric.
  • Mandatory hands-on experience with Azure Data Factory (ADF) for data ingestion, orchestration, and pipeline development.
  • Experience building and maintaining ETL/ELT pipelines and scalable data integration solutions.
  • Strong understanding of Data Lakehouse architecture, Medallion architecture (Bronze, Silver, Gold), and data warehousing concepts.
  • Experience with source-to-target mapping, data reconciliation, incremental loading, historical data handling, and dimensional modeling.
  • Experience delivering at least two production data platform implementations.
  • Bachelor's degree in Computer Science, IT, Engineering, or a related field.

Good to Have

  • Experience with Fivetran or other data ingestion / ELT tools.
  • Experience with dbt models, tests, documentation, incremental strategies, Git workflows, and performance tuning.
  • Financial services or investment data experience.
  • Exposure to Profisee, Dynamics 365 integrations, Python, and Azure DevOps.