Senior Data Engineer - (NPW)
1 viewMilestone Technologies, Inc.KochiHybrid
5+ years
Posted 4 days ago
- Llm
- Platform Engineering
- Python
- Spark
Role Summary
Owns the Enterprise Knowledge Ingestion Framework: pipeline hardening, chunking, metadata extraction, indexing, versioning, and synchronization extension to new sources; owns integration with the client's operational data platform API and the Planisware field expansion for an upcoming release, and the dual-source retrieval architecture with source attribution and fallback protocol; and owns the enterprise ingestion RAG pipeline and Databricks Enterprise Vector Store integration, supporting the Knowledge Discovery and Tribal Intelligence Suite.
Key Responsibilities
- Harden the Enterprise Knowledge Ingestion Framework pipeline
- Implement chunking, metadata extraction, indexing, and versioning
- Extend synchronization to Teams, Smartsheet, and the client's action-tracking system
- Build and maintain integration with the client's operational data platform API
- Deliver Planisware field expansion and verify API completeness
- Build the dual-source retrieval architecture with source attribution and defined fallback protocols
- Build and maintain the enterprise ingestion RAG pipeline
- Own Databricks Enterprise Vector Store integration
- Build knowledge-base pipelines supporting Knowledge Discovery and Tribal Intelligence Suite
Required Skills and Experience
- 5+ years data engineering experience, including ingestion pipeline design
- Experience with chunking and metadata-extraction strategies for RAG pipelines
- Vector store indexing, versioning, and synchronization experience
- Strong Python, REST API integration, and ETL pipeline design skills
- Experience integrating enterprise planning or operational systems
- Experience designing fallback / source-attribution protocols for dual-source retrieval
- Hands-on Databricks experience
- Embeddings and vector-store pipeline management experience
- RAG integration experience
- Education: B.Tech/M.Tech in Computer Science, Data Engineering, or a related field, or equivalent practical experience
Preferred Qualifications
- Prior Smartsheet or Teams API integration experience
- Direct Planisware integration experience
- Databricks Enterprise Vector Store production experience specifically