Get Started
Hunting for you, 24x7Get the iPhone app

Your hunt runs in your pocket, and tells you the moment a strong match opens.

Download on theApp Store
ServiceNowPosted 9 days ago

Senior Machine Learning Engineer, Agent Eval Platform

Who we'd reach at ServiceNowFound and reached by Power Apply
Best leadEngineering manager

Leads the team you would join

Tech lead

Leads the work day to day

Technical recruiter

Owns the opening

Power Apply

From your own Gmail and LinkedIn.

About the role

This is a senior full-time machine learning role focused on evaluating AI agents, based in Santa Clara, California, and suited to engineers with at least two years of applied ML or ML-adjacent experience. The work centres on building the measurement loop that tells the team whether a change makes an enterprise agent better, why a conversation failed, and which part of the harness needs fixing. The team improves the system around the model (instructions, tool descriptions, routing, knowledge and memory) rather than training the model itself. The role blends ML with systems engineering, with the aim of producing reproducible evaluation signals across hundreds of customer organisations. LLM-as-judge methodology is treated as an open research question, so the role involves real judgement about when a signal is trustworthy enough to act on.

What you’ll do

  • Design a shared base judge with per-item rubrics stored as configuration beside each dataset.
  • Split checks between deterministic validators for verifiable world state and an LLM judge for fuzzy criteria such as policy compliance and efficiency.
  • Build scoring that reports its own confidence so uncertain judgements route to a human reviewer.
  • Run a standing calibration loop against human-labelled trajectories with the annotation team.
  • Guard against correlated blind spots between the user simulator and the judge.
  • Investigate where offline simulation and production results diverge, and keep the test suite refreshed with new production failures.
  • Build components that surface agent weaknesses, generate realistic synthetic scenarios, and recommend which harness component should change.
  • Verify proposed changes on held-out cases and turn each fixed failure into a permanent regression test.
  • Build the reproducible infrastructure that runs this loop at scale across customer organisations.

What they’re looking for

  • At least 2 years in applied ML, data science, or ML-adjacent engineering, with shipped and used work.
  • Experience turning subjective human judgement into a measurement other people can act on.
  • Strong applied ML fundamentals and comfort evaluating, prompting and fine-tuning LLMs as components.
  • Strong Python and the discipline to ship production-grade code.
  • Clear communication about complex problems and the ability to show that a metric means what it claims.
  • High ownership, a bias toward shipping, and comfort with ambiguity.
  • Experience in at least three of: LLM-as-judge or automated evaluation design, human annotation programmes, search ranking or recommender evaluation, small-model fine-tuning, reward modelling or RLHF, agent trajectory analysis, versioned prompt engineering.

What’s on offer

  • Base pay of $161,300 to $274,200, plus equity where applicable, variable incentive compensation and benefits.
  • Health plans, flexible spending accounts, a 401(k) with company match, ESPP, matching donations, flexible time away and family leave.
  • Work persona assigned by the employer, which may be flexible, remote, or required in office.

Skills

Questions about this role

How much does this role pay?

The base pay range is $161,300 to $274,200, plus equity where applicable, incentive compensation and benefits.

How many years of experience do I need?

At least two years in applied ML, data science, or ML-adjacent engineering.

Is this role based in the office?

The role is listed in Santa Clara, California, and the employer assigns a work persona that may be flexible, remote, or required in office.

Does this role train AI models?

No. The posting says the team improves the system around the model and does not train the model itself.