All vacancies
Senior Data Scientist AI Evaluation
Alpaca
Remote · AmericasSalary not disclosedfull-timeAlpaca Careers
Your Role: We're looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our models and agents are actually right. You'll be a senior individual contributor who turns ambiguous quality questions into ground truth, scoring methods, and eval loops that the company can trust—and uses those results to make the systems better.
Responsibilities
- Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
- Build repeatable eval loops: Track quality over time and catch regressions before release.
- Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.
- Drive iteration
- Translate eval results into actionable recommendations for system improvements.
- Establish quality standards
- Set evaluation guidelines, documentation, and review practices.
- Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.
Languages
- Work format
- Remote
- Seniority
- Senior
- Posted
- 7 Oct 2026
- Last verified
- In the last 5 days
Similar roles
- Data ScientistNewel Health · Remote · worldwide, UTC+4 accepted
- Data Scientistmicro1 · Remote · worldwide, UTC+4 accepted
- Data Scientist (AI Data & LLM Specialist)Eclipse Laboratories · Remote · worldwide, UTC+4 accepted
- Freelance Data Scientist with Ontology and Graph expertiseadesso Belgium · Remote · worldwide, UTC+4 accepted
- Lead AI Engineer with agentic coding harnessesEPAM Systems · Remote · Armenia
- Senior AI Developer with Model Context ProtocolDataArt Armenia · Armenia · On-site