Leads the team this role joins
In-house, owns the opening
Director or head of the group
From your own Gmail and LinkedIn.
How this role compares
One of 14 open Applied AI Engineer roles we are tracking today.
Posted 20 days ago, with 12 of the others posted since and 1 still open from before.
Counted across the roles Frontrunner has open right now.
About the role
FICO is hiring a mid-level Applied AI Engineer for a full-time remote role open to candidates eligible to work in the United States. The role sits in the consumer software team behind myFICO Scores and focuses on making AI-powered consumer features measurable, safe and better over time. It combines backend engineering in Python and Java with hands-on work on large language model systems, including prompts, evaluation, guardrails and model changes.
What you’ll do
- Refine system prompts, tool descriptions and evaluation prompts for Personalized Credit Insights and the AI chat assistant, using production data to guide each change.
- Grow the evaluation framework by turning production failures into test cases and running regression suites before any prompt or model change.
- Measure how reliable LLM-based evaluators are against human-labeled data, tracking precision, recall and agreement over time.
- Design, tune and monitor guardrails such as input and output classifiers, PII redaction, prompt-injection defenses and financial-advice rules set with Legal and Compliance.
- Lead model upgrades by comparing quality, cost and latency, then rolling changes out through A/B tests.
- Reduce cost and latency with prompt caching, model routing and context management.
- Build internal tools for conversation analysts and QA engineers, including transcript viewers, labeling workflows and dashboards.
- Work with analytics, legal and compliance teams on fairness and bias evaluation of AI outputs.
- Use AI coding assistants and agentic workflows in delivery while reviewing their output critically.
What they’re looking for
- A bachelor's or master's degree in computer science, engineering or a related field, plus professional software engineering experience.
- Strong Python and the ability to read and contribute to Java Spring Boot microservices.
- Experience running LLM-powered applications in production, including prompt design, tool calling, structured outputs and streaming.
- Experience building evaluations for LLM systems, such as test sets, regression suites and LLM-as-judge methods, including checking evaluator quality against human labels.
- Working knowledge of Amazon Bedrock or a comparable managed LLM platform, along with AWS services such as S3, Lambda and DynamoDB.
- Solid SQL and comfort analyzing production data in a warehouse such as Amazon Redshift.
- Production ownership across code review, CI/CD, release management and on-call support within Agile or Scrum teams.
- Clear written and verbal communication for explaining evaluation results and tradeoffs to product, legal and executive audiences.
Nice to have
- Guardrail tooling and LLM security testing.
- RAG and retrieval evaluation.
- Fairness and bias evaluation, and model migrations.
- Datadog observability and Tableau BI work.
- Experience in regulated lending areas such as FCRA, ECOA, UDAAP or CFPB guidance.
What’s on offer
- Target base pay of $105,000 to $165,000 per year, set by candidate knowledge, skills and experience.
- Competitive compensation, benefits and rewards programs.
- A people-first environment with work-life balance, employee resource groups and social events.
- Opportunities for professional development and career growth.
Skills
Questions about this role
Is this role remote?
Yes. The role is remote and open to candidates eligible to work in the United States.
How much does it pay?
The target base pay range is $105,000 to $165,000 per year.
Do I need a degree?
A bachelor's or master's degree in computer science, engineering or a related field is listed, alongside demonstrated professional software engineering experience.
What experience is needed?
The posting asks for professional software engineering experience, including LLM applications in production and experience building evaluations for LLM systems.