Senior Data Engineer
1 view6+ years
Posted 1 day ago
- Airflow
- Git
- MySQL
- PostgreSQL
- Python
- Spark
🚀 We're Hiring: Senior Data Engineer
📍 Location: Bangalore
💼 Experience: 6+ Years
About the Role
We're looking for a Senior Data Engineer with strong data architecture and system design skills to design, build, and own end-to-end production pipelines powering analytics and decision-making across the organization. The role centers on deep Python and SQL expertise, robust transaction-level fact tables, and rigorous data quality and auditability, ideally within a finance, risk, or compliance data domain.
What You'll Do
🔹 Own data architecture and system design decisions for pipelines and data models, including grain, partitioning, schema evolution, and scalability trade-offs.
🔹 Take end-to-end ownership of production pipelines: design, build, deploy, monitor, and operate.
🔹 Design and build transaction tables (append-only, immutable event/transaction-grain fact tables) alongside standard fact/dimension models using medallion (bronze/silver/gold) architecture.
🔹 Build and support pipelines for finance, risk, or compliance use cases where accuracy, auditability, and data lineage are critical.
🔹 Implement data quality and auditability controls, including validation checks, reconciliation logic, anomaly detection, and audit trails for every pipeline.
🔹 Write complex, cross-dialect SQL (MySQL + PostgreSQL), including window functions, multi-layered CTEs, CASE-driven logic, COALESCE/NULLIF, type casting, and date/timestamp handling.
🔹 Build idempotent reload patterns (DELETE + INSERT), UNION ALL/set operations, and templated (Jinja-style) SQL, including handling late-arriving/corrected transaction records without double-counting.
🔹 Orchestrate Airflow-style DAGs with cross-pipeline dependencies and secure credential handling.
🔹 Build Python ETL tooling using pandas (CSV ↔ DB), SQLAlchemy + raw drivers (MySQLdb, psycopg2), type hints/dataclasses, and class-based pipeline design.
🔹 Integrate with Google Drive/Sheets APIs, build YAML-driven pipeline configurations, and handle pipeline staging requirements.
🔹 Own CI/CD for pipelines through standard Git workflows.
🔹 Document data models and pipeline logic for maintainability, keeping compliance and audit requirements in mind.
What We're Looking For
Must-Have Skills:
✅ 6+ years of experience in data engineering.
✅ Strong data architecture and system design skills to make sound trade-offs on modeling, partitioning, and scalability - not just write queries.
✅ Demonstrated end-to-end production ownership of data pipelines from design through deployment, monitoring, and operational support.
✅ Expert-level Python and SQL as the primary toolset. Hive/Hadoop is a nice-to-have only.
✅ Robust data quality and auditability practices, including validation, reconciliation, anomaly detection, and audit trails.
✅ Experience working with finance, risk, or compliance data, with an understanding of accuracy, auditability, and data-lineage requirements.
✅ Proven experience designing and building transaction/fact tables.
✅ Expert SQL skills, including window functions, CTEs, CASE logic, and cross-dialect MySQL/PostgreSQL.
✅ Strong Python skills with pandas, SQLAlchemy, type hints/dataclasses, and class-based pipelines.
✅ Airflow-like orchestration experience.
✅ Solid data modeling fundamentals, including fact/dimension models, normalization, and medallion architecture.
✅ YAML-based configuration experience, along with standard CI/CD and Git workflows.
Nice-to-Have Skills
🔹 Hive/Hadoop or Spark experience.
📩 Interested candidates: Share your updated resume
#Hiring #SeniorDataEngineer #DataEngineering #Python #SQL #DataArchitecture #ETL #DataPipelines #BangaloreJobs #TechHiring