Data Engineer ( Mid Level )
5 viewsVenPep GroupCoimbatoreHybrid
3-5 years
Posted 3 days ago
- AWS
- Azure
- MySQL
- Node.js
- PostgreSQL
- Python
- Spark
- SQL Server
Role Overview
We are looking for a skilled Mid-Level Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data processing solutions. The ideal candidate should have strong programming and SQL skills, hands-on experience with Python, Databricks, PySpark, and Delta Lake, and a solid understanding of ETL/ELT processes and data integration across AWS and Microsoft Azure.
Key Responsibilities
- Design, develop, and maintain scalable batch and incremental data pipelines.
- Build ETL/ELT workflows to extract, transform, and load data from APIs, databases, and third-party platforms.
- Implement Medallion Architecture with Bronze, Silver, and Gold layers using Delta Lake tables.
- Develop and optimize SQL queries, stored procedures, and schemas on PostgreSQL, SQL Server, and MySQL.
- Build data integration utilities and backend services using TypeScript and Node.js.
- Implement Change Data Capture (CDC) and incremental loading mechanisms.
- Develop serverless functions using Azure Functions and orchestrate pipelines using Durable Functions.
- Implement data quality, deduplication, reconciliation, and monitoring.
Required Skills
- 3 - 5 years of experience in data engineering, data integration, or related field.
- Strong proficiency in Python and SQL.
- Hands-on experience with Databricks, PySpark, and Delta Lake.
- Knowledge of ETL/ELT development, CDC, and incremental loading.
- Experience integrating REST APIs (JSON, CSV).
- Working knowledge of TypeScript and Node.js.
- Cloud experience with AWS (S3, Glue, Lambda) or Microsoft Azure (ADLS Gen2, Data Factory, Azure Functions).
- Familiarity with Git, CI/CD, and pipeline troubleshooting.