Get Started
Hunting for you, 24x7Get the iPhone app

Your hunt runs in your pocket, and tells you the moment a strong match opens.

Download on theApp Store
Software EstrategicoPosted 15 days ago

Senior AI Quality Engineer (LLM Evaluation & Automation)

Who we'd reach at Software EstrategicoFound and reached by Power Apply
Best leadEngineering manager

Leads the team you would join

Technical recruiter

Owns the opening

Team member

On the team, can refer you

Power Apply

From your own Gmail and LinkedIn.

About the role

This is a senior, full-time, remote AI quality engineering role open worldwide, focused on evaluating LLM and agent behavior. It suits engineers who can build evaluation harnesses, define release gates, and measure non-deterministic systems. The role owns the evaluation harness and quality gate from the start, replacing a late-stage evals model with a standing owner for measurable agent quality. Early work centers on a first reference agent with a published scorecard and an automated evaluation path.

What you’ll do

  • Build and maintain the MVP evaluation harness, including golden tasks, exception tasks, scorecard metrics, and regression packs.
  • Wire evaluations into CI so that quality regressions fail builds and releases.
  • Define and maintain release-gate thresholds together with Product and the Tech Lead.
  • Lay the groundwork for later adversarial and drift testing without overbuilding the MVP.
  • Use AI to draft candidate eval cases and failure hypotheses, and validate them before relying on them.

What they’re looking for

  • Experience evaluating ML, LLM, or other non-deterministic systems.
  • Strong test and benchmark design capability.
  • Comfort working with noisy metrics, thresholds, and probabilistic behavior.
  • Good scripting and automation skills.
  • An AI-first approach that treats AI quality as an operating discipline rather than an afterthought.

What’s on offer

  • Remote position open worldwide.

Skills

Questions about this role

Is this role remote?

Yes, it is a remote position open worldwide.

What should I deliver in the first 90 days?

A published scorecard and gated evaluation path for the first reference agent, with golden and exception tests running automatically.

What does the team expect me to define?

What good enough to ship means in measurable terms, together with Product and the Tech Lead.