Get Started
Hunting for you, 24x7Get the iPhone app

Your hunt runs in your pocket, and tells you the moment a strong match opens.

Download on theApp Store
Software EstrategicoPosted 15 days ago

Agent Quality / Evals Engineer

Who we'd reach at Software EstrategicoFound and reached by Power Apply
Best leadHiring manager

Leads the team this role joins

Recruiter

In-house, owns the opening

Head of the team

Director or head of the group

Power Apply

From your own Gmail and LinkedIn.

About the role

This is a full-time remote engineering role for someone who will own the evaluation harness and quality gate for AI agents from the start. It suits engineers experienced in evaluating ML, LLM or non-deterministic systems who can design benchmarks and reason about noisy metrics. The role is a standing owner for measurable agent quality, replacing a model where evaluation came in late as a specialist function. Success in the first 90 days means a first reference agent with a published scorecard and a gated evaluation path.

What you’ll do

  • Build and maintain the MVP eval harness, including golden tasks, exception tasks, scorecard metrics and regression packs.
  • Wire evaluations into CI so quality regressions fail builds and releases.
  • Define and maintain release gate thresholds with product and the tech lead.
  • Prepare the path for later adversarial and drift testing without overbuilding the MVP.
  • Use AI to draft candidate eval cases and failure hypotheses, and validate them before trusting them.

What they’re looking for

  • Experience evaluating ML, LLM or non-deterministic systems.
  • Strong test and benchmark design capability.
  • Comfort working with noisy metrics, thresholds and probabilistic behaviour.
  • Good scripting and automation skills.
  • An approach that treats AI quality as an operating system rather than a QA afterthought.

What’s on offer

  • Fully remote position.

Skills

Questions about this role

Is this role remote?

Yes. The posting describes it as a remote, full-time position.

What experience is required?

Evaluating ML, LLM or non-deterministic systems, designing tests and benchmarks, and scripting automation.

What should be delivered in the first 90 days?

A published scorecard and gated evaluation path for the first reference agent, with golden and exception tests running automatically.