AI Evaluator & QA Specialist Needed
Employer not named by the sourceRemote
Frontier is not the employer and does not collect applications.
About this role
Python, Testing / QA, Test Automation, Machine Learning (ML), Mathematics, Data Science, Prompt Engineering, AI Model Development, AI Quality Assurance, AI Training Data · We are seeking experienced AI Evaluators, QA Specialists, and Technical Pod Leads to evaluate and benchmark frontier LLMs and agentic AI systems for client pilot projects at JudgeMyAI.
Key Responsibilities: • Evaluate AI model outputs across multi-turn reasoning, agentic tool-use, and code generation (SWE-bench). • Stress-test models via adversarial prompts, edge-case analysis, and red-teaming. • Apply and calibrate evaluation rubrics (RLHF, SFT, accuracy, safety, and hallucination checks). • Review and annotate dataset batches with high inter-annotator agreement.
Requirements: • Experience with LLM evaluation, prompt engineering, or AI benchmarking (prior work on platforms like Turing, Outlier, micro1 or enterprise AI labs is a strong plus). • Background in Python, Software Engineering, Data Science, or Advanced STEM. • Strong analytical reasoning and strict attention to detail.
Engagement Details: • Remote and flexible hours (asynchronous execution). • Fixed milestone or hourly payouts in USD.