Leads the team this role joins
In-house, owns the opening
Put the opening up
From your own Gmail and LinkedIn.
About the role
This is a full-time, fully remote role for a mid-level engineer who evaluates and improves frontier coding agents. It suits engineers who combine software engineering, machine learning and model evaluation skills, and who are comfortable in fast-moving research settings. The company is remote-first and states that the role is open to candidates in a list of countries that includes the United States, United Kingdom, Canada, Germany, India and the Philippines.
What you’ll do
- Design and own evaluation frameworks for coding agents, including benchmark specs, scoring methods, rubrics and quality standards.
- Lead research on measuring and improving coding model performance across software engineering tasks.
- Build datasets, golden examples and evaluation protocols for frontier coding systems.
- Analyze model behavior and failure modes, and turn findings into training and evaluation improvements.
- Build tooling for large-scale experiments, data generation, review workflows and evaluation pipelines.
- Set best practices for coding-agent assessment with a focus on rigor and reproducibility.
- Work with researchers, engineers and applied AI teams to design experiments.
- Contribute to technical reports, benchmark studies and research communication.
What they’re looking for
- Strong software engineering background in Python, C++ or a comparable language.
- 3 years or more in software engineering, machine learning, AI research, evaluation or a related field.
- Experience designing, reviewing or validating assessments, benchmarks, coding tasks or evaluation methods.
- Familiarity with LLMs, coding agents, reinforcement learning or model evaluation.
- Ability to build tooling, automate workflows and run systematic experiments.
- Strong analytical skills for investigating model behavior.
- Clear written and verbal communication for technical and non-technical audiences.
- Comfort working in fast-moving environments with ambiguity and shifting priorities.
Nice to have
- Experience working on frontier AI systems, coding agents or model evaluation research.
- Experience designing benchmarks or datasets for machine learning systems at scale.
- Familiarity with agentic workflows, tool use, reinforcement learning or post-training methods.
- Publications, open-source contributions or technical leadership in AI, machine learning or software engineering.
- A track record of driving ambiguous projects from concept to delivery.
What’s on offer
- Base salary of $200,000 to $260,000 USD per year.
- Equity compensation for all employees.
- Possible performance-based bonuses, subject to role and company policy.
- Up to 100% reimbursement of health insurance premiums.
- Paid time off and a 401(k) plan with company match.
- Fully remote working arrangement.
Skills
Questions about this role
How much does it pay?
The listed base salary is $200,000 to $260,000 USD per year, with equity and possible performance bonuses.
Is this role remote?
Yes. It is fully remote and open to candidates in a list of countries that includes the United States, United Kingdom, Canada, Germany, India and the Philippines.
How many years of experience do I need?
The listing asks for 3 or more years in software engineering, machine learning, AI research, evaluation or a related field.
Is this a full-time job?
Yes, it is a full-time position.