Leads the team this role joins
Decides on early hires
Runs hiring at the company
From your own Gmail and LinkedIn.
About the role
This is a full-time, remote applied AI research and engineering role focused on evaluating and improving LLM agents used in real enterprise workflows. It suits engineers with applied research or LLM system experience who can design evaluations, analyze agent failures and run fast experiments, with base pay of 200,000 to 250,000 USD. The team works with researchers, domain experts and enterprise customers, and the work includes building evaluation tooling alongside agent evaluations.
What you’ll do
- Design and implement evaluation frameworks, benchmarks and datasets for LLM agents and multi-step AI workflows.
- Find, analyze and prioritize model, agent and workflow failures in production settings.
- Run quick experiments to test hypotheses and improve agent reliability.
- Work with enterprise customers and researchers to deploy and tune AI systems in production workflows.
- Create training and evaluation data, task specifications and feedback mechanisms.
- Build Python evaluation pipelines and experimentation tools.
- Write technical reports and benchmark findings that guide future improvements.
What they’re looking for
- A strong background in applied AI research, research engineering, machine learning or LLM-based systems.
- Hands-on experience designing evaluations for AI agents, including tool use, multi-step workflows and task completion.
- Strong Python skills and experience building evaluation pipelines or ML infrastructure.
- Experience with benchmarks, datasets, metrics and experimental methods for AI systems.
- Ability to diagnose agent failures and turn findings into measurable gains.
- Understanding of LLM behavior, agent architectures, and reliability, robustness and generalization challenges.
- Clear communication across research, engineering and customer teams.
Nice to have
- Experience evaluating or optimizing agents in production or enterprise settings.
- Familiarity with reinforcement learning, reward modeling or post-training.
- Experience building research infrastructure or automated experimentation frameworks.
- Experience at a frontier AI lab or in an applied AI research environment.
- Publications, open-source benchmarks or evaluation framework contributions.
- Experience working directly with enterprise customers or technical stakeholders.
What’s on offer
- Base salary of 200,000 to 250,000 USD per year.
- Equity eligibility for all employees.
- Possible performance-based bonuses, subject to role and company policy.
- Up to 100 percent reimbursement of health-insurance premiums, paid time off, and a 401(k) plan with company match.
Skills
Questions about this role
How much does it pay?
Base salary is 200,000 to 250,000 USD per year, plus equity eligibility and possible performance bonuses.
Is this role remote?
Yes, it is a fully remote position open to candidates based in 55 countries.
What benefits are offered?
Benefits include up to 100 percent reimbursement of health-insurance premiums, paid time off and a 401(k) plan with company match.