Get Started
Hunting for you, 24x7Get the iPhone app

Your hunt runs in your pocket, and tells you the moment a strong match opens.

Download on theApp Store
ProductPosted 5 days ago

Senior Data Scientist AI Evaluation

United StatesSeniorRemoteFull-time$41,950 – $59,893 a year
Who we'd reach at ProductFound and reached by Power Apply
Best leadEngineering manager

Leads the team you would join

Tech lead

Leads the work day to day

Technical recruiter

Owns the opening

Power Apply

From your own Gmail and LinkedIn.

About the role

This is a full-time remote senior individual contributor role for a data scientist who will design how the company measures whether its models and agents are correct. The role suits candidates with 6 to 10 years of quantitative data science or ML experience, and the posted salary is between 41,950 and 59,893 USD per year. It sits within a brokerage infrastructure company and focuses on building a greenfield evaluation practice.

What you’ll do

  • Define ground truth, metrics and scoring methods for models and agents.
  • Build repeatable evaluation loops that track quality and catch regressions.
  • Work with engineering and analytics engineering to run evaluation harnesses.
  • Turn evaluation results into recommendations for system improvements.
  • Set evaluation guidelines, documentation and review practices.
  • Mentor others and build a culture of measurable AI quality.

What they’re looking for

  • A track record of quantitative measurement, such as LLM evaluation, metric validation or experimentation.
  • Statistical and ML foundations, including sample sizing, confidence intervals and validating automated graders.
  • Proficiency in Python and SQL.
  • Experience evaluating models in production.
  • Strong judgement in defining quality metrics for ambiguous outputs.
  • Clear communication with technical teams and leadership.
  • 6 to 10 years in quantitative data science or ML, with focus on measurement or evaluation.

Nice to have

  • Hands-on LLM or agent evaluation in production, including LLM-as-judge calibration and CI regression gates.
  • Evaluating text-to-SQL or analytics agents.
  • Background in fintech, brokerage or similar domains.
  • Fluency with AI tools in research and engineering.

What’s on offer

  • Competitive salary and stock options.
  • Health benefits.
  • One-time USD 500 home-office setup.
  • Monthly USD 150 stipend.

Skills

Questions about this role

Is this role remote?

The metadata lists the role as remote, and the posting does not name a specific location.

How many years of experience do I need?

6 to 10 years in quantitative data science or ML is expected.

How much does it pay?

The posted range is 41,950 to 59,893 USD per year, plus stock options and benefits.