Leads the team you would join
Leads the work day to day
Owns the opening
From your own Gmail and LinkedIn.
About the role
This is a full-time, fully remote role for a mid-level engineer who builds evaluation and continuous learning features for Pydantic's Logfire observability platform. The engineer owns the evals product, the open source pydantic-evals library, and work that turns production traces into improvements for AI systems. Candidates need at least five years of software engineering experience and must work in a timezone between Pacific and Central European time.
What you’ll do
- Own and grow Logfire's evals product, including datasets, experiments, evaluators and the analysis interface.
- Build the loop that turns production telemetry into eval cases, detects regressions and shows what changed.
- Advance continuous learning work by analysing failure patterns and proposing prompt and variable changes with safe A/B rollouts.
- Develop the pydantic-evals library, including evaluators, reports and integrations.
- Work with the Pydantic AI team to make evals and improvement first-class for agent developers.
- Define and instrument metrics for adoption and improvement.
What they’re looking for
- Experience building evaluation, experimentation, prompt management or LLM observability systems
- Working knowledge of LLM-as-judge, rubric design, dataset curation and statistical measures
- A product mindset that turns vague goals into datasets, metrics and workflows
- Backend skills with FastAPI and Postgres, and willingness to work in React and TypeScript on owned surfaces
- Genuine interest in AI engineering
- At least five years of software engineering experience
- Living and working in a timezone between PT and CET
- Willingness to travel up to four times a year to offsites in the EU, UK and US
- Willingness to join an on-call rotation for owned systems
Nice to have
- Experience with an evals or observability platform such as Braintrust, Langfuse, LangSmith, Arize, Humanloop or Logfire
- Open source contributions to eval or AI tooling
- Familiarity with Pydantic, Pydantic AI and OpenTelemetry GenAI conventions
- Background in ML or experimentation methods such as A/B testing and statistics
What’s on offer
- Competitive salary and stock options
- Fully remote work with flexible hours and no office requirement
- Comprehensive healthcare for you and your dependents
- 33 days of paid time off, including public holidays, which can be taken or not
- Equipment budget for a home office
- Regular international offsites and learning opportunities
Skills
Questions about this role
Is this job fully remote?
Yes, the role is fully remote, but the engineer must work in a timezone between PT and CET.
What experience is required?
At least five years of software engineering experience.
What should I include in my application?
Share something you have built in the evals or AI space, such as a project, library, talk or blog post.
Does the job involve travel?
Yes, the employer expects travel of up to four times a year to offsites in the EU, UK and US.