Data Engineer (Azure)
0 viewsRapidBrainsBengaluruHybrid
5-8 years
Posted 1 day ago
- Azure
- Devops
- FastAPI
- Git
- Jenkins
- Oracle
- Python
- Spark
Job Description
We are looking for an experienced Data Engineer (Azure) with strong expertise in Python, PySpark, Azure Databricks, and cloud-based data engineering solutions. The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data pipelines, data processing workflows, and data infrastructure using Azure cloud services.
Experience: 5+ Years
Location : Bangalore/Pune/Gurgaon
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Databricks, Python, PySpark, and Spark SQL.
- Implement data engineering solutions using Azure Data Factory, Azure Data Lake Storage Gen2, Azure Functions, Microsoft Fabric, and Azure services.
- Develop, optimize, and maintain ETL/ELT workflows and data processing pipelines.
- Build and maintain APIs using Python frameworks such as FastAPI and REST APIs.
- Develop, configure, and maintain data models, data quality rules, and data warehouse solutions.
- Work with SQL, Microsoft SQL Server, Oracle, RDBMS, and NoSQL databases.
- Analyze and optimize existing PySpark, Python, and SQL code to improve performance and reduce execution time.
- Implement CI/CD pipelines and follow DevOps practices using Git, Jenkins, and related tools.
- Troubleshoot data-related issues and support stakeholders with data infrastructure requirements.
- Maintain technical documentation covering data pipelines, architecture, development processes, and data governance.
- Perform unit testing, integration testing, quality checks, and ensure compliance with SLA/TAT requirements.
- Collaborate with business and technical stakeholders to understand requirements and deliver effective data solutions.
Required Skills & Qualifications
- 5 - 8 years of experience in Data Engineering or software development, with a strong focus on data projects.
- Strong hands-on experience with Azure Databricks, Python, PySpark, and Spark SQL.
- Experience with Azure Data Factory, Azure Data Lake Storage Gen2, Azure Functions, Azure Databases, and Microsoft Fabric.
- Strong SQL skills and experience working with SQL Server, Oracle, RDBMS, and NoSQL databases.
- Experience developing APIs using FastAPI or similar Python frameworks.
- Good understanding of ETL/ELT processes, data warehousing, data modeling, and data quality.
- Familiarity with Git, Jenkins, CI/CD pipelines, and DevOps practices.
- Experience in data pipeline development, performance optimization, and testing.
- Strong communication, problem-solving, and stakeholder management skills.