Get Started
Hunting for you, 24x7Get the iPhone app

Your hunt runs in your pocket, and tells you the moment a strong match opens.

Download on theApp Store
BrillioPosted 9 days ago

Lead Quality Engineer

Who we'd reach at BrillioFound and reached by Power Apply
Best leadHiring manager

Leads the team this role joins

Founder

Decides on early hires

Head of talent

Runs hiring at the company

Power Apply

From your own Gmail and LinkedIn.

About the role

This is a full-time lead quality engineering role based in Bangalore for engineers with four to six years of testing experience who can evaluate AI-powered products and microservices. The work centers on testing AI agents for safety, bias and robustness with Python-based tools, alongside conventional test automation. The role sits in the AI and data engineering department at Brillio.

What you’ll do

  • Build and run evaluation frameworks for AI agents, focused on Responsible AI and safety metrics such as bias, toxicity and alignment failures.
  • Design synthetic test environments and simulations that stress agents under varied conditions.
  • Run adversarial testing, red teaming and prompt injection tests to find weaknesses.
  • Add evaluation pipelines to CI/CD for continuous quality checks.
  • Work with policy and legal teams to check agent evaluations against regulatory and ethical standards.
  • Analyze evaluation results and advise development teams on risk and safety priorities.
  • Write and run automated test suites in Python with PyTest and AI testing libraries.
  • Track defects in Jira, especially performance regressions and alignment failures.

What they’re looking for

  • Four to six years of quality engineering experience, including hands-on testing of AI-powered applications and microservices.
  • Python programming and PyTest.
  • Experience with DeepEval, Ragas, Giskard and LangSmith.
  • Familiarity with Google Agent Development Kit and Vertex AI.
  • Safety evaluation skills, including red teaming and adversarial testing.
  • Prompt injection testing experience.
  • API testing for AI and ML systems.
  • Data analysis with Pandas and SQL.
  • A bachelor's degree in computer science, artificial intelligence, information technology or software engineering.

Nice to have

  • Experience with Gemini models, Hugging Face and data poisoning techniques.
  • Experience with garbage-in and garbage-out testing tools.
  • Background in AI ethics or AI policy.
  • Visualization tools for evaluation results.
  • ISTQB Foundation Level or Certified Software Tester certification.
  • Certification in AI/ML testing or Responsible AI, such as Google Cloud's Professional Machine Learning Engineer.

Skills

PythonPyTestDeepEvalRagasGiskardLangSmithGoogle Agent Development KitVertex AIRed teamingAdversarial testingPrompt injection testingAPI testingPandasSQLJiraJenkinsSeleniumPostman

Questions about this role

How many years of experience does this role need?

Four to six years of quality engineering experience, including hands-on work with AI-powered products and microservices.

Which tools should I know?

Python, PyTest, DeepEval, Ragas, Giskard, LangSmith, Google Agent Development Kit, Vertex AI and SQL are the core tools named.

Is a degree required?

A bachelor's degree is listed in the education requirements.