Lead AI Engineer - Conversational AI & Semantic Search (Databricks / LangChain)

Responsibilities

• Design and build a conversational AI / agentic application using LangChain and/or LangGraph, deployed on Databricks
• Implement semantic search / retrieval-augmented generation (RAG) pipelines using vector embeddings and a vector database (e.g., Databricks Vector Search, Chroma, Pinecone, FAISS)
• Define and tune similarity-matching logic (embedding models, distance metrics, thresholding, ranking of "% match" style results)
• Integrate with data sources (Databricks Unity Catalog tables, Delta Lake, APIs) to source and prepare the underlying knowledge base/corpus
• Own the ML/LLMOps lifecycle: experimentation, evaluation, versioning, deployment, and monitoring of the chatbot in production
• Collaborate with business stakeholders to translate a new use case into technical requirements
• Write clean, maintainable, well-documented Python code and establish testing and CI/CD practices for the AI application
• Ensure appropriate handling of data privacy, access control, and cost management (token usage, compute)

Languages