Databricks AI Architect
2 viewsPersistent SystemsPuneHybrid
11-16 years
Posted today
- Llm
- Python
- Spark
About Position:
We are seeking an experienced Databricks AI Architect to lead the design and implementation of enterprise-scale Artificial Intelligence, Machine Learning, and Generative AI solutions. The ideal candidate will possess deep expertise in Databricks, Python, PySpark, Machine Learning, and modern AI architectures. This role involves architecting scalable AI platforms, driving innovation using Generative AI and Agentic AI technologies, and delivering business-focused AI solutions that leverage large-scale data ecosystems.
- Role: Databricks AI Architect
- Location: All Persistent Locations
- Experience: Between 11 to 16 Years
- Job Type: Full Time Employment
What You'll Do:
- Design, develop, and deploy end-to-end Machine Learning and Generative AI solutions using Databricks, Python, and PySpark.
- Perform Exploratory Data Analysis (EDA) to identify data patterns, trends, quality issues, and actionable insights.
- Build, train, evaluate, optimize, and deploy Machine Learning models for predictive and prescriptive analytics.
- Design scalable feature engineering, data preparation, and model training pipelines using Databricks and PySpark.
- Implement and manage ML lifecycle processes using MLflow, including experiment tracking, model registry, versioning, and deployment.
- Architect Retrieval-Augmented Generation (RAG) solutions using enterprise data sources, vector databases, embeddings, and Large Language Models (LLMs).
- Develop Generative AI applications such as intelligent assistants, chatbots, knowledge search platforms, and document intelligence solutions.
- Design enterprise AI architectures that ensure scalability, security, reliability, and operational excellence.
- Lead architecture discussions and define best practices for AI, ML, and data platform implementations.
- Evaluate and benchmark AI models for accuracy, performance, explainability, and business impact.
- Collaborate with business stakeholders to translate functional requirements into technical AI solutions.
- Drive AI governance, model management, responsible AI practices, and deployment standards.
- Support data engineering initiatives including ingestion, transformation, optimization, and pipeline development within the Databricks ecosystem.
- Contribute to Agentic AI initiatives involving autonomous workflows, intelligent agents, and AI-driven automation.
- Mentor engineering and data science teams on AI architecture, model development, and best practices.
- Stay current with advancements in Machine Learning, Generative AI, Agentic AI, LLMs, MLOps, and cloud-native AI technologies.
Expertise You'll Bring:
- 11+ years of experience in Data Science, Machine Learning, AI Engineering, or Data Platform Architecture.
- Strong hands-on expertise with Databricks and the Databricks Lakehouse Platform.
- Deep expertise in Python and PySpark development.
- Strong understanding of Machine Learning algorithms, model evaluation techniques, and statistical analysis.
- Proven experience building and deploying production-grade Machine Learning and Generative AI applications.
- Hands-on experience with LLMs, prompt engineering, fine-tuning, embeddings, and vector databases.
- Experience implementing RAG architectures and enterprise AI search solutions.
- Proficiency in MLOps practices, MLflow, model deployment, CI/CD, and model monitoring.
- Experience with cloud platforms (AWS, Azure, or GCP) and cloud-native data services.
- Knowledge of big data technologies, distributed computing, and data lakehouse principles.
- Familiarity with Agentic AI, multi-agent frameworks, and autonomous workflow design.
- Strong problem-solving, architectural design, and analytical capabilities.
- Excellent communication and stakeholder management skills.
Benefits:
- Competitive compensation and benefits
- Continuous learning and certifications in advanced AI technologies
- Collaborative and innovative work culture