Jobs by Fastorial
Back to all jobs

Senior Data Engineer

1 view
LifesightBengaluruOnsite

6-10 years

Posted 8 days ago

  • Airflow
  • GCP
  • Java
  • Kafka
  • Kubernetes
  • Python
  • Spark
  • Terraform

About Factori / Lifesight

Factori is a rapidly growing location and consumer intelligence company built for an AI-first world. Each day we ingest billions of privacy-compliant signals and maintain the world's most comprehensive consumer, mobility and point-of-interest database.

About the Role

We are hiring a Senior Data Engineer to own and scale a petabyte-scale data platform on Google Cloud Platform (GCP), processing TB scale of data every day with Apache Spark. You will lead a small team of data engineers, drive cloud cost optimisation across our data workloads, and build AI-driven workflows.

Key Responsibilities

  • Design, build and operate highly available, fault-tolerant batch and streaming pipelines on GCP that ingest hundreds of terabytes a day into a petabyte-scale data lake and warehouse.
  • Write and tune large Spark workloads (PySpark, Spark SQL, Scala/Java) on Dataproc or Spark on Kubernetes (GKE): partitioning, shuffle and join optimization, skew handling, memory and executor sizing.
  • Build real-time and near-real-time pipelines with Pub/Sub or Kafka and Spark Structured Streaming or Dataflow (Apache Beam).
  • Model data in BigQuery and the lakehouse (Parquet, Avro, Delta Lake or Apache Iceberg) into clean, simplified datasets.
  • Build data cleaning, validation and reconciliation pipelines with automated quality checks.
  • Orchestrate pipelines with Airflow / Cloud Composer and ship through CI/CD and infrastructure as code (Terraform).

Requirements

  • 6+ years in data engineering or backend/software engineering, with at least 4 years building big data pipelines in production.
  • Production experience with BigQuery, Dataproc, Cloud Storage (GCS), Pub/Sub, and Cloud Composer or Airflow.
  • Deep hands-on PySpark, Spark SQL, and performance tuning on multi-terabyte jobs.
  • Strong Python and Java with object-oriented design and clean code practices.
  • Infrastructure automation with Terraform, Docker and Kubernetes.