Jobs by Fastorial
Back to all jobs

Data Engineer ( Mid Level )

5 views
VenPep GroupCoimbatoreHybrid

3-5 years

Posted 3 days ago

  • AWS
  • Azure
  • MySQL
  • Node.js
  • PostgreSQL
  • Python
  • Spark
  • SQL Server

Role Overview

We are looking for a skilled Mid-Level Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data processing solutions. The ideal candidate should have strong programming and SQL skills, hands-on experience with Python, Databricks, PySpark, and Delta Lake, and a solid understanding of ETL/ELT processes and data integration across AWS and Microsoft Azure.

Key Responsibilities

  • Design, develop, and maintain scalable batch and incremental data pipelines.
  • Build ETL/ELT workflows to extract, transform, and load data from APIs, databases, and third-party platforms.
  • Implement Medallion Architecture with Bronze, Silver, and Gold layers using Delta Lake tables.
  • Develop and optimize SQL queries, stored procedures, and schemas on PostgreSQL, SQL Server, and MySQL.
  • Build data integration utilities and backend services using TypeScript and Node.js.
  • Implement Change Data Capture (CDC) and incremental loading mechanisms.
  • Develop serverless functions using Azure Functions and orchestrate pipelines using Durable Functions.
  • Implement data quality, deduplication, reconciliation, and monitoring.

Required Skills

  • 3 - 5 years of experience in data engineering, data integration, or related field.
  • Strong proficiency in Python and SQL.
  • Hands-on experience with Databricks, PySpark, and Delta Lake.
  • Knowledge of ETL/ELT development, CDC, and incremental loading.
  • Experience integrating REST APIs (JSON, CSV).
  • Working knowledge of TypeScript and Node.js.
  • Cloud experience with AWS (S3, Glue, Lambda) or Microsoft Azure (ADLS Gen2, Data Factory, Azure Functions).
  • Familiarity with Git, CI/CD, and pipeline troubleshooting.