AI Test Engineer
Skills
About the role
This is an AI Test Engineer role in Hyderabad focused on testing production agentic and multi-agent AI systems. You'll own QA strategy across development, staging and production, validating reliability, accuracy, latency, orchestration, security and scale.
What you’ll do
- Define QA strategy for agentic and multi-agent systems across environments.
- Mentor QA engineers and establish standards for test harnesses and reviews.
- Partner with Operations, Data Science, MLOps and Platform teams throughout the SDLC.
- Test planner-executor loops, tool calls, task decomposition and inter-agent handoffs.
- Validate state, context windows, memory stores and prompt or graph behavior.
- Build fuzzing scenarios for adversarial inputs, tool delays and degraded APIs.
- Create resilience suites covering chaos, failover, retries, circuit breakers and degraded modes.
- Set latency SLOs and measure orchestration, model, tool and queue performance.
- Run soak tests, canary checks and automated rollback validation.
- Build ground-truth and macro-validation pipelines for multi-step outcomes.
- Test guardrails for toxicity, PII, hallucination and policy compliance.
- Design load and stress tests for multi-agent graphs and queues.
What they’re looking for
- At least 7 years in software QA or testing, including at least 2 years with AI, ML or LLM systems.
- Hands-on experience testing agentic or multi-agent architectures.
- Strong Python skills for test harnesses, simulators and fixtures.
- Experience with LLM evaluation methods, guardrails and prompt testing.
- Expertise in distributed-systems testing, latency profiling, resiliency and message queues.
- Familiarity with LangChain, LangGraph, LlamaIndex, DSPy, OpenAI orchestration, Azure OpenAI orchestration or similar.
- Proficiency with CI/CD, observability, feature flags and canary releases.
- Understanding of privacy, security, compliance and model safety.
- Strong communication and cross-functional leadership skills.
Nice to have
- Multi-agent simulators, graph testing and latency emulation.
- MLOps, model versioning, evaluation pipelines and LLM experimentation.
- AWS, serverless, containers and event-driven architecture.
- Prior ownership of AI workload cost, latency or service-level objectives.
What’s on offer
- Full-time permanent work in Hyderabad.
- The role includes ownership of QA for agentic AI from development through production.
Questions about this role
Where is the AI Test Engineer role based?
It is based in Hyderabad.
How much experience is required?
The qualifications ask for 7+ years in software QA or testing, including 2+ years with AI, ML or LLM systems.
What will the engineer test?
The work covers agent orchestration, tool calls, memory, prompts, reliability, latency, accuracy, guardrails and scale.
Which programming language is required?
Strong Python skills are required for test harnesses, simulators and fixtures.
Is the role permanent?
Yes. It is listed as full-time and permanent.
Related roles
AI Engineer (LLM / Chatbot / RAG)
Hyderabad AI Engineer role for a senior technical lead building production LLM, chatbot, RAG, and agentic AI systems.
Solutions Engineer AI Engineer
Remote solutions engineering role in India for building conversational voice and video AI applications in life sciences. The role suits engineers with at least three years of Python, API, LLM, or AI solutions experience.
Engineer - Engine Testing
Automotive engine-testing engineer role in Chennai for professionals who can plan and execute validation, analyse failures and support issue closure. The position suits candidates with hands-on experience in calibration, NVH, durability, instrumentation and test planning.
Senior AI-Native Engineer
Senior hybrid AI engineering role in Gurgaon for an engineer with five or more years of experience and production AI/ML background. You'll design agentic applications, evaluation frameworks, and monitored AI capabilities end to end.
AI Engineer – Generative AI
Onsite AI Engineer role in Hyderabad, Ahmedabad, or Indore building generative-AI and agentic systems. The position suits an engineer with at least six years of experience across LLMs, RAG, Python, and cloud deployment.
Principal Engineer - Gen AI, LLM, RAG, Agentic AI
A principal engineering role at Wells Fargo in Bengaluru focused on GenAI, RAG and agentic AI architecture, evaluation and governance. Suited to a senior technical leader with deep GenAI stack and AI security expertise.