Jobs by Fastorial
Back to all jobs

Senior Data Engineer - Databricks

0 views
ShyftLabsCoimbatoreHybrid

7+ years

Posted 11 days ago

  • AWS
  • Git
  • Python
  • Spark

Position Overview: We are looking for a Senior Data Engineer with strong hands-on experience in Databricks and a solid understanding of Data Governance concepts. The ideal candidate should have 7+ years of overall experience, including a minimum of 5 years of Databricks experience, covering data engineering and/or Databricks infrastructure, along with knowledge of data governance principles and practices. ShyftLabs is a growing data product company that works primarily with Fortune 500 companies.

Job Responsibilities:

  • Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
  • Work on Databricks infrastructure, including workspace configuration, clusters, compute resources, access controls, and performance optimization.
  • Apply Data Governance principles, including data quality, data lineage, metadata management, data cataloging, and access controls.
  • Work with Unity Catalog to manage data permissions, governance policies, and data assets.
  • Develop and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, and Gold).
  • Integrate data from multiple sources, including databases, Amazon S3 (AWS), files, and REST APIs.
  • Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
  • Implement data validation, error handling, logging, monitoring, and data quality checks.
  • Optimize Spark workloads, SQL queries, and data processing pipelines for performance and scalability.
  • Support data discovery, metadata documentation, and governance standards across data platforms.
  • Troubleshoot data pipeline and infrastructure issues and recommend improvements to platform reliability.

Basic Qualifications:

  • 7+ years of overall experience in Data Engineering or a related data technology role.
  • Minimum 5 years of hands-on Databricks experience, including data engineering and/or Databricks infrastructure.
  • Mandatory understanding of Data Governance concepts: data quality, metadata management, data lineage, data cataloging, data ownership, access controls.
  • Strong expertise in Python, PySpark, and Advanced SQL.
  • Hands-on experience with Databricks Lakehouse Platform, Delta Lake, Unity Catalog, Workflows, Jobs, Clusters, Notebooks.
  • Experience developing ETL/ELT pipelines, data transformations, batch processing, and incremental data loading.
  • Good understanding of data warehousing and data modeling concepts.
  • Knowledge of data security, access management, and governance practices.
  • Experience with Spark performance tuning, partitioning, file optimization, and query optimization.
  • Familiarity with Git, CI/CD practices, and production deployment processes.

Benefits:

  • Competitive salary alongside a strong insurance package.
  • Extensive learning and development resources.