Senior AI Machine Learning Engineer (Agentic Systems & LLM Evaluation)

Responsibilities

• Build agentic systems tailored to various Software Development Lifecycle (SDLC) personas, translating requirements into actionable implementations
• Partner closely with cross-functional teams to understand workflows and ensure alignment with stakeholder needs
• Design and implement robust evaluation test cases for LLMs using LLMs as judges and deterministic methodologies
• Evaluate model performance and ensure reliability using tools such as LangSmith, LangFuse, or MLFlow
• Develop and fine-tune prompt frameworks to align LLMs with diverse agent requirements
• Utilize programming expertise in Python and AI/ML frameworks to enhance agentic systems' capabilities
• Monitor, debug, and optimize AI systems’ performance using advanced testing and automation practices
• Leverage modern SDKs like LangGraph or similar frameworks to efficiently build intelligent workflows
• Foster continuous improvement by incorporating feedback, testing outcomes, and insights into future iterations
• Act as a thought leader on AI testing standards, governance, and best practices in agentic system development

Languages