Freelance Agent Evaluation Engineer
Skills
About the role
This is a freelance, project-based role building and grading evaluation tasks for AI coding agents. It suits an experienced software developer who wants part-time, remote contract work designing realistic developer environments and tests that judge whether an AI agent's solution is correct. Work is paid per accepted task rather than as a salary.
What you’ll do
- Build realistic simulated developer environments with codebase, infrastructure and context
- Design tasks from intermediate states of those environments, including prompts and success criteria
- Write tests that verify agent solutions, accepting valid approaches and rejecting incorrect ones
- Review agent solutions and QA feedback to refine tasks and tests
What they’re looking for
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing functional and integration tests
- English proficiency at B2 level or above
- Minimum 3 years of professional experience in a related domain such as QA-automation, testing or cybersecurity
What’s on offer
- Paid per accepted task, up to the equivalent of $40/hr depending on qualification tier and pace
- Part-time, remote, freelance work that fits around other commitments
Questions about this role
Is this role remote?
Yes, it's fully remote and freelance.
How much does it pay?
Paid per accepted task, up to the equivalent of $40/hr depending on tier and speed.
How many years of experience do I need?
5+ years in software development, with at least 3 years in a related domain like QA-automation or cybersecurity.
Is this a permanent job?
No, it's project-based freelance work, not permanent employment.
What tech skills are needed?
Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka and Redis.
Related roles
Machine Learning Engineer - Agentic AI Evaluation Frameworks
A senior ML evaluation engineering role at Apple in Cupertino, building evaluation frameworks for generative and agentic AI products. Suits engineers with 7+ years of ML, evaluation or software engineering experience.
Software Engineer (AI-Powered Advertising Agents - AgenticOS)
A software engineering role at PubMatic in Pune building AI agents for programmatic advertising, open to candidates with 2-10 years of experience.
Agentic Python Engineer
Evaboot is hiring a remote Agentic Python Engineer to build SDR agents and self-improving company systems. The role is suited to strong Python developers who use agentic coding tools.
Cloud Engineer 3 - AI Agents
A Bangalore-based cloud engineering role at Adobe building AI agents and tooling for cloud cost optimization.
Engineer - Engine Testing
Automotive engine-testing engineer role in Chennai for professionals who can plan and execute validation, analyse failures and support issue closure. The position suits candidates with hands-on experience in calibration, NVH, durability, instrumentation and test planning.
Agentic AI Engineer
Remote full-time Agentic AI Engineer role at General Dynamics Information Technology. The posting is aimed at engineers building enterprise AI systems with retrieval, model operations and automated decision support.