Senior Data Engineer
1 viewLifesightBengaluruOnsite
6-10 years
Posted 8 days ago
- Airflow
- GCP
- Java
- Kafka
- Kubernetes
- Python
- Spark
- Terraform
About Factori / Lifesight
Factori is a rapidly growing location and consumer intelligence company built for an AI-first world. Each day we ingest billions of privacy-compliant signals and maintain the world's most comprehensive consumer, mobility and point-of-interest database.
About the Role
We are hiring a Senior Data Engineer to own and scale a petabyte-scale data platform on Google Cloud Platform (GCP), processing TB scale of data every day with Apache Spark. You will lead a small team of data engineers, drive cloud cost optimisation across our data workloads, and build AI-driven workflows.
Key Responsibilities
- Design, build and operate highly available, fault-tolerant batch and streaming pipelines on GCP that ingest hundreds of terabytes a day into a petabyte-scale data lake and warehouse.
- Write and tune large Spark workloads (PySpark, Spark SQL, Scala/Java) on Dataproc or Spark on Kubernetes (GKE): partitioning, shuffle and join optimization, skew handling, memory and executor sizing.
- Build real-time and near-real-time pipelines with Pub/Sub or Kafka and Spark Structured Streaming or Dataflow (Apache Beam).
- Model data in BigQuery and the lakehouse (Parquet, Avro, Delta Lake or Apache Iceberg) into clean, simplified datasets.
- Build data cleaning, validation and reconciliation pipelines with automated quality checks.
- Orchestrate pipelines with Airflow / Cloud Composer and ship through CI/CD and infrastructure as code (Terraform).
Requirements
- 6+ years in data engineering or backend/software engineering, with at least 4 years building big data pipelines in production.
- Production experience with BigQuery, Dataproc, Cloud Storage (GCS), Pub/Sub, and Cloud Composer or Airflow.
- Deep hands-on PySpark, Spark SQL, and performance tuning on multi-terabyte jobs.
- Strong Python and Java with object-oriented design and clean code practices.
- Infrastructure automation with Terraform, Docker and Kubernetes.