AI Agentic Tester

AI Agentic Tester

Denver, MO (Remote)

Must-Have Skills

* AI Agentic Testing

* Functional Testing

* Generative AI Testing

* Large Language Models (LLMs)

* AI Agents & Agentic AI

* Retrieval-Augmented Generation (RAG)

* Prompt Engineering Validation

* AI Model Validation & Evaluation

• API Testing (Postman, REST APIs, Swagger)

* Python

• Test Automation (Selenium / Playwright / PyTest)

• AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)

* End-to-End Workflow Testing

* Multi-Agent Testing

* Azure OpenAI / AWS Bedrock / Gemini

Core Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.

• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.

• Test RAG pipelines, prompt engineering, and AI agent orchestration.

• Execute API testing using Postman, Swagger, and REST APIs.

• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.

• Validate multi-agent interactions, tool calling, and workflow execution.

• Apply AI testing methodologies and evaluation metrics to ensure response quality.

• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.

Originally posted on Himalayas