Leads the team this role joins
In-house, owns the opening
Director or head of the group
From your own Gmail and LinkedIn.
About the role
This is a full-time remote engineering role for someone who will own the evaluation harness and quality gate for AI agents from the start. It suits engineers experienced in evaluating ML, LLM or non-deterministic systems who can design benchmarks and reason about noisy metrics. The role is a standing owner for measurable agent quality, replacing a model where evaluation came in late as a specialist function. Success in the first 90 days means a first reference agent with a published scorecard and a gated evaluation path.
What you’ll do
- Build and maintain the MVP eval harness, including golden tasks, exception tasks, scorecard metrics and regression packs.
- Wire evaluations into CI so quality regressions fail builds and releases.
- Define and maintain release gate thresholds with product and the tech lead.
- Prepare the path for later adversarial and drift testing without overbuilding the MVP.
- Use AI to draft candidate eval cases and failure hypotheses, and validate them before trusting them.
What they’re looking for
- Experience evaluating ML, LLM or non-deterministic systems.
- Strong test and benchmark design capability.
- Comfort working with noisy metrics, thresholds and probabilistic behaviour.
- Good scripting and automation skills.
- An approach that treats AI quality as an operating system rather than a QA afterthought.
What’s on offer
- Fully remote position.
Skills
Questions about this role
Is this role remote?
Yes. The posting describes it as a remote, full-time position.
What experience is required?
Evaluating ML, LLM or non-deterministic systems, designing tests and benchmarks, and scripting automation.
What should be delivered in the first 90 days?
A published scorecard and gated evaluation path for the first reference agent, with golden and exception tests running automatically.