Runs the team day to day
In-house, owns the opening
Joined the team recently
From your own Gmail and LinkedIn.
About the role
This is a remote role open to candidates in Canada and the United States for an engineer who defines and measures quality for Cresta's conversational AI agents. Cresta's AI Receptionist answers calls for small businesses, and the hire will be the first person to own how well those conversations go. The work mixes reading and labeling real conversations, designing automated judges and golden sets, and checking those judges against human reviewers. Fixing problems is welcome but secondary to producing an accurate, prioritized view of agent quality.
What you’ll do
- Define the evaluation rubric covering adherence, outcome, and conversation experience such as latency, turn-taking, and tone.
- Write annotation guidelines and a failure taxonomy, and run calibration sessions that measure inter-rater agreement.
- Review production conversations each week, group failures into clusters, and route each one to the right owner with supporting evidence.
- Build the pipeline that scores production conversations using automated checks, LLM-based judges, and human review.
- Check each automated judge against human labels before relying on its scores.
- Design sampling strategies across intent, language, caller type, and audio conditions.
- Maintain the golden set and regression suite that every change is tested against before release.
- Report quality by segment with uncertainty stated, and work with engineering and product on priorities.
- Document methods so results can be reproduced by others.
What they’re looking for
- 3+ years in software engineering, applied ML, or data-heavy research, including 1-2 years working with LLM systems in production.
- A bachelor's or master's in computer science, computational linguistics, cognitive science, statistics, or a related field, or equivalent practical experience.
- Experience building and running an end-to-end evaluation system for a production LLM product.
- A solid understanding of evaluation methods, including when golden sets, judges, and calibration fail.
- Understanding of how LLM agents work, including prompts, tool calls, context handling, and multi-turn state.
- Strong Python skills and experience with LLM APIs.
- Ability to write a rubric that independent reviewers score consistently.
- Professional fluency in at least one language other than English.
- Strong written communication.
What’s on offer
- Location-based base pay plus equity.
- Medical, dental, and vision coverage.
- Flexible PTO and paid parental leave.
- Retirement savings plan.
- Remote work setup budget.
- Monthly wellness and communication stipend.
Skills
Questions about this role
Is this role remote?
Yes, it is remote and open in Canada and the United States.
What is the first hiring step?
After an initial resume review, qualified applicants complete a self-paced CoderPad technical assessment before moving to conversational interview rounds.
How many years of experience are needed?
At least 3 years of professional experience, including 1-2 years working hands-on with LLM systems in production.
Is a second language required?
Yes, professional fluency in at least one language other than English is listed as a requirement.