Leads the team this role joins
Decides on early hires
Runs hiring at the company
From your own Gmail and LinkedIn.
About the role
This is a fully remote, project-based role for an associate who joins a talent pool supporting AI model evaluation. It suits people with zero to five years of experience in QA, assessment design, research support or subject work involving question writing. The work centres on writing tasks and rubrics, grading model outputs and documenting failures, and projects typically last from one week to several months.
What you’ll do
- Write questions and tasks that test one model capability at a time.
- Build rubrics that separate strong, partial and weak answers.
- Produce reference answers and edge cases that challenge models.
- Grade model outputs against a rubric and explain each score.
- Group recurring error patterns in failed answers and record them.
- Run identical prompts across several models and compare the results.
- Check benchmarks and flag questions that are compromised or ineffective.
- Take part in small user studies on ratings and model interactions.
What they’re looking for
- Zero to five years of professional experience.
- Background in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support or subject work involving question writing.
- Experience writing test cases, exam questions, rubrics or evaluation tasks, and checking work against written criteria.
- Comfort using spreadsheets or Python to organise and analyse results.
- Clear written communication and the ability to compare model outputs or join rating studies.
Nice to have
- Familiarity with evaluation tools such as Inspect, lm-eval-harness, HELM or promptfoo.
What’s on offer
- Pay of $30 to $60 per hour, depending on project, field and experience.
- Paid for hours worked.
- Fully remote working.
Skills
Questions about this role
Is this role remote?
Yes, it is fully remote.
How much does it pay?
Pay is $30 to $60 per hour, depending on project, field and experience, and is paid for hours worked.
How many years of experience do I need?
The posting asks for zero to five years of professional experience.
How long do projects last?
Projects typically run from one week to several months.