Senior Data Engineer - Databricks
0 viewsShyftLabsCoimbatoreHybrid
7+ years
Posted 11 days ago
- AWS
- Git
- Python
- Spark
Position Overview: We are looking for a Senior Data Engineer with strong hands-on experience in Databricks and a solid understanding of Data Governance concepts. The ideal candidate should have 7+ years of overall experience, including a minimum of 5 years of Databricks experience, covering data engineering and/or Databricks infrastructure, along with knowledge of data governance principles and practices. ShyftLabs is a growing data product company that works primarily with Fortune 500 companies.
Job Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
- Work on Databricks infrastructure, including workspace configuration, clusters, compute resources, access controls, and performance optimization.
- Apply Data Governance principles, including data quality, data lineage, metadata management, data cataloging, and access controls.
- Work with Unity Catalog to manage data permissions, governance policies, and data assets.
- Develop and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, and Gold).
- Integrate data from multiple sources, including databases, Amazon S3 (AWS), files, and REST APIs.
- Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
- Implement data validation, error handling, logging, monitoring, and data quality checks.
- Optimize Spark workloads, SQL queries, and data processing pipelines for performance and scalability.
- Support data discovery, metadata documentation, and governance standards across data platforms.
- Troubleshoot data pipeline and infrastructure issues and recommend improvements to platform reliability.
Basic Qualifications:
- 7+ years of overall experience in Data Engineering or a related data technology role.
- Minimum 5 years of hands-on Databricks experience, including data engineering and/or Databricks infrastructure.
- Mandatory understanding of Data Governance concepts: data quality, metadata management, data lineage, data cataloging, data ownership, access controls.
- Strong expertise in Python, PySpark, and Advanced SQL.
- Hands-on experience with Databricks Lakehouse Platform, Delta Lake, Unity Catalog, Workflows, Jobs, Clusters, Notebooks.
- Experience developing ETL/ELT pipelines, data transformations, batch processing, and incremental data loading.
- Good understanding of data warehousing and data modeling concepts.
- Knowledge of data security, access management, and governance practices.
- Experience with Spark performance tuning, partitioning, file optimization, and query optimization.
- Familiarity with Git, CI/CD practices, and production deployment processes.
Benefits:
- Competitive salary alongside a strong insurance package.
- Extensive learning and development resources.