Leads the team you would join
Leads the work day to day
Owns the opening
From your own Gmail and LinkedIn.
About the role
This is a full-time remote senior individual contributor role for a data scientist who will design how the company measures whether its models and agents are correct. The role suits candidates with 6 to 10 years of quantitative data science or ML experience, and the posted salary is between 41,950 and 59,893 USD per year. It sits within a brokerage infrastructure company and focuses on building a greenfield evaluation practice.
What you’ll do
- Define ground truth, metrics and scoring methods for models and agents.
- Build repeatable evaluation loops that track quality and catch regressions.
- Work with engineering and analytics engineering to run evaluation harnesses.
- Turn evaluation results into recommendations for system improvements.
- Set evaluation guidelines, documentation and review practices.
- Mentor others and build a culture of measurable AI quality.
What they’re looking for
- A track record of quantitative measurement, such as LLM evaluation, metric validation or experimentation.
- Statistical and ML foundations, including sample sizing, confidence intervals and validating automated graders.
- Proficiency in Python and SQL.
- Experience evaluating models in production.
- Strong judgement in defining quality metrics for ambiguous outputs.
- Clear communication with technical teams and leadership.
- 6 to 10 years in quantitative data science or ML, with focus on measurement or evaluation.
Nice to have
- Hands-on LLM or agent evaluation in production, including LLM-as-judge calibration and CI regression gates.
- Evaluating text-to-SQL or analytics agents.
- Background in fintech, brokerage or similar domains.
- Fluency with AI tools in research and engineering.
What’s on offer
- Competitive salary and stock options.
- Health benefits.
- One-time USD 500 home-office setup.
- Monthly USD 150 stipend.
Skills
Questions about this role
Is this role remote?
The metadata lists the role as remote, and the posting does not name a specific location.
How many years of experience do I need?
6 to 10 years in quantitative data science or ML is expected.
How much does it pay?
The posted range is 41,950 to 59,893 USD per year, plus stock options and benefits.