Jobs by Fastorial
Back to all jobs

Data Platform Engineer

1 view
Umanist NABengaluruHybrid

2+ years

Posted 1 day ago

  • Airflow
  • Git
  • Kafka
  • Microservices
  • MongoDB
  • Python
  • Spark

Data Platform Engineer

  • Experience:** 4-10 Years (Relevant Experience)
  • Location:** Bangalore
  • Work Mode:** Hybrid
  • Employment Type:** Full-Time

Role Overview

We are looking for a skilled Senior Data Engineer - Platform & Operations to manage and improve enterprise data platforms, with responsibilities spanning data engineering, pipeline development, production support and data operations.

The role involves building and maintaining data pipelines, managing large-scale data processing workflows, monitoring platform performance and resolving data-related issues. You will also work closely with business stakeholders and technical teams to understand requirements and deliver reliable data solutions.

The ideal candidate should have strong hands-on experience with Python, PySpark, SQL and Big Data technologies, along with an understanding of production environments, workflow orchestration, data quality and application operations.

Key Responsibilities

1. Data Engineering & Pipeline Development

  • Build, maintain and enhance scalable data pipelines using Python, PySpark, SQL and Apache Spark.
  • Develop data processing solutions for both batch and streaming workloads.
  • Work with large datasets using Hadoop, Hive, HDFS and Spark-based technologies.
  • Handle data extraction, transformation, integration and processing across multiple sources.
  • Implement data models and transformation logic to support business and reporting requirements.
  • Identify performance bottlenecks and optimize existing data pipelines.
  • Investigate data inconsistencies, processing failures and technical defects.
  • Contribute to platform enhancements, automation and data architecture improvements.

2. Data Platform Monitoring & Operations

  • Monitor scheduled data workflows, applications, microservices and reporting activities.
  • Manage data pipeline execution, scheduling and workflow dependencies using Apache Airflow.
  • Investigate failed jobs, delayed data processing and application-related issues.
  • Perform controlled data reloads, job restarts and reprocessing activities whenever required.
  • Monitor data availability, freshness and overall platform performance.
  • Support production deployments, releases and ongoing platform maintenance.
  • Ensure smooth functioning of data processing systems and minimize operational disruptions.

3. Production Support & Incident Management

  • Manage production incidents, service requests and change activities following ITIL practices.
  • Investigate recurring technical issues and perform root-cause analysis.
  • Coordinate with application, infrastructure, development and database teams for issue resolution.
  • Work with ITSM platforms such as ServiceNow to track and manage operational tickets.
  • Maintain technical documents, troubleshooting guides and operational knowledge resources.
  • Provide timely updates to stakeholders regarding production issues and their resolution.

4. Data Quality & Testing

  • Perform data validation and quality checks across different processing stages.
  • Identify missing records, incorrect values, inconsistencies and data processing errors.
  • Conduct unit, integration, regression and functional testing for data solutions.
  • Validate data transformations, reporting outputs and downstream data consumption.
  • Develop mechanisms to improve data accuracy, reliability and monitoring.
  • Ensure that data pipelines meet expected quality and performance standards.

5. Reporting & Business Intelligence

  • Develop and maintain reports and dashboards using Power BI.
  • Monitor scheduled report refreshes and troubleshoot data connectivity or refresh failures.
  • Work with business users to understand reporting requirements and data needs.
  • Ensure accurate and consistent data availability for business reporting.

6. Stakeholder & Requirement Management

  • Collaborate with business users, data engineers, architects and application teams to understand technical requirements.
  • Convert business expectations into technical specifications and data engineering solutions.
  • Participate in requirement-gathering sessions and technical discussions.
  • Evaluate the impact of new requirements on existing data pipelines and platform architecture.
  • Identify opportunities to leverage available data for additional business use cases.
  • Coordinate with cross-functional teams to ensure smooth implementation of requirements.
  • Communicate technical challenges, dependencies and proposed solutions effectively.

Required Technical Skills

*Required Technical Skills

  • Strong programming skills in Python, PySpark and SQL.

  • Hands-on experience with Hadoop, Hive, HDFS, Apache Spark and Spark SQL.

  • Good understanding of ETL processes, data transformation, data modelling and pipeline optimization.

  • Experience with batch processing and Spark Structured Streaming.

  • Working knowledge of MongoDB and relational databases.

  • Experience working with Parquet and Avro file formats.

  • Hands-on exposure to Apache Kafka and Kafka-based data processing.

  • Experience with Apache Airflow for workflow scheduling, monitoring and troubleshooting.

  • Knowledge of Power BI for dashboard development and report refresh management.

  • Understanding of CI/CD pipelines and deployment tools such as GitHub, Bitbucket or equivalent.

  • Working knowledge of Linux / RHEL operating systems.

  • Understanding of ITIL processes, incident management and ServiceNow/ITSM tools.

  • Familiarity with development tools such as VS Code, Jupyter Notebook or equivalent.

Skills: hive,apache spark,apache kafka,spark,pyspark,batch processing,mongodb,apache airflow,etl processes,python,ci/cd pipelines,hdfs,hadoop,sql