Get Started
Hunting for you, 24x7Get the iPhone app

Your hunt runs in your pocket, and tells you the moment a strong match opens.

Download on theApp Store
PydanticDate not stated

Evals & Continuous Learning Engineer

Who we'd reach at PydanticFound and reached by Power Apply
Best leadEngineering manager

Leads the team you would join

Tech lead

Leads the work day to day

Technical recruiter

Owns the opening

Power Apply

From your own Gmail and LinkedIn.

About the role

This is a full-time, fully remote role for a mid-level engineer who builds evaluation and continuous learning features for Pydantic's Logfire observability platform. The engineer owns the evals product, the open source pydantic-evals library, and work that turns production traces into improvements for AI systems. Candidates need at least five years of software engineering experience and must work in a timezone between Pacific and Central European time.

What you’ll do

  • Own and grow Logfire's evals product, including datasets, experiments, evaluators and the analysis interface.
  • Build the loop that turns production telemetry into eval cases, detects regressions and shows what changed.
  • Advance continuous learning work by analysing failure patterns and proposing prompt and variable changes with safe A/B rollouts.
  • Develop the pydantic-evals library, including evaluators, reports and integrations.
  • Work with the Pydantic AI team to make evals and improvement first-class for agent developers.
  • Define and instrument metrics for adoption and improvement.

What they’re looking for

  • Experience building evaluation, experimentation, prompt management or LLM observability systems
  • Working knowledge of LLM-as-judge, rubric design, dataset curation and statistical measures
  • A product mindset that turns vague goals into datasets, metrics and workflows
  • Backend skills with FastAPI and Postgres, and willingness to work in React and TypeScript on owned surfaces
  • Genuine interest in AI engineering
  • At least five years of software engineering experience
  • Living and working in a timezone between PT and CET
  • Willingness to travel up to four times a year to offsites in the EU, UK and US
  • Willingness to join an on-call rotation for owned systems

Nice to have

  • Experience with an evals or observability platform such as Braintrust, Langfuse, LangSmith, Arize, Humanloop or Logfire
  • Open source contributions to eval or AI tooling
  • Familiarity with Pydantic, Pydantic AI and OpenTelemetry GenAI conventions
  • Background in ML or experimentation methods such as A/B testing and statistics

What’s on offer

  • Competitive salary and stock options
  • Fully remote work with flexible hours and no office requirement
  • Comprehensive healthcare for you and your dependents
  • 33 days of paid time off, including public holidays, which can be taken or not
  • Equipment budget for a home office
  • Regular international offsites and learning opportunities

Skills

Questions about this role

Is this job fully remote?

Yes, the role is fully remote, but the engineer must work in a timezone between PT and CET.

What experience is required?

At least five years of software engineering experience.

What should I include in my application?

Share something you have built in the evals or AI space, such as a project, library, talk or blog post.

Does the job involve travel?

Yes, the employer expects travel of up to four times a year to offsites in the EU, UK and US.