Who we'd reach at OpenTrain AIFound and reached by Power Apply
Best leadTeam lead
Runs the team day to day
Recruiter
In-house, owns the opening
Recent hire
Joined the team recently
From your own Gmail and LinkedIn.
About the role
This is a remote, part-time contract role for engineers with at least three years of experience who design and run evaluations for coding agents, open to applicants in 52 countries. Pay is USD 96.15-125 per hour, and applicants apply through OpenTrain. The work covers evaluation frameworks, benchmark creation and the study of systematic weaknesses in coding systems.
What you’ll do
- Design evaluation frameworks for coding agents, covering benchmark specifications, scoring methods, rubrics and quality standards.
- Lead research into coding-model performance and its systematic weaknesses.
- Develop datasets, golden examples and evaluation protocols.
- Build experimentation tools, data generation tools, review workflows and evaluation pipelines.
- Analyse model behaviour and failures, and turn them into improvements for training and evaluation.
- Establish reproducible practices for coding-agent assessment and report results through technical reports and benchmark studies.
What they’re looking for
- Fluent English.
- At least three years in software engineering, machine learning, AI research, evaluation or a related technical field.
- Strong software engineering with Python, C++ or a comparable language.
- Experience designing, reviewing or validating assessments, coding tasks, benchmarks or evaluation methods.
- Familiarity with LLMs, coding agents, reinforcement learning or model evaluation.
- Ability to build tooling, automate workflows, analyse complex systems and explain findings clearly.
Nice to have
- Publications, open-source contributions or demonstrated technical leadership in AI, machine learning or software engineering.
- Experience with frontier AI systems, coding-agent research or agentic workflows.
- Benchmark or dataset design at scale, tool use, reinforcement learning or post-training methods.
What’s on offer
- USD 96.15-125 per hour.
- Part-time contract, fully remote.
Skills
LLMsCoding evaluationAI evaluationML systemsPythonC++Benchmark designReinforcement learningCoding agentsTechnical writing
Questions about this role
How many years of experience are needed?
At least three years in software engineering, machine learning, AI research, evaluation or a related discipline.
How much does it pay?
USD 96.15-125 per hour.
Which programming languages are expected?
Python, C++ or a comparable language, along with fluent English.