AI Quality Analyst

Our client, a global consulting firm, is hiring an AI Quality Analyst to join the team on a contract basis in Dublin, Ireland. The successful candidate will apply their legal, regulatory and compliance expertise to complex policy-enforcement decisions, evaluating both human reviewer performance and AI-generated outputs.
 

Responsibilities

• Evaluate and classify content decisions and AI-generated outputs across text, image and video, ensuring accuracy, consistency and quality.

• Apply strong critical thinking and subject-matter expertise to complex, sensitive and ambiguous cases.

• Participate in calibration and team collaboration sessions to ensure consistent interpretation and application of policies.

• Assess the quality of human reviewer decisions and provide clear, structured and actionable feedback to improve performance.

• Evaluate AI-generated outputs against defined quality, safety and compliance standards, providing structured feedback to support model improvement.

• Develop and maintain high-quality gold-standard datasets for training, benchmarking and measuring human and AI performance.

• Identify errors, recurring trends, inconsistencies and areas of ambiguity, recommending improvements to policies, guidelines and evaluation frameworks.

• Clearly communicate findings and decision-making, confidently highlighting errors, risks and areas requiring further review.

• Contribute to the ongoing development of Trust & Safety policies, quality standards and evaluation processes as technology, regulations and organisational requirements evolve.

Skillset

• At least 4 years of professional experience in compliance, legal, regulatory affairs, Trust & Safety, risk management, policy enforcement or a related discipline.

• Strong knowledge of AI governance, Responsible AI, Online Safety and content policy enforcement.

• Practical understanding of key regulatory frameworks, including the EU AI Act, GDPR and Online Safety legislation.

• Proven ability to interpret and apply complex policies, regulations, standards and legal requirements to real-world scenarios.

• Excellent critical thinking and analytical skills, with the ability to assess complex, nuanced and ambiguous cases.

• Experience conducting quality assessments, identifying errors and providing clear, evidence-based feedback.

• Strong written and verbal communication skills, with the ability to clearly articulate decisions, findings and recommendations.

• Exceptional attention to detail, with a consistent, objective and evidence-based approach to decision-making.

• Experience working with AI-generated content, AI systems, content policies or technology risk is highly desirable.

• Comfortable working collaboratively in calibration sessions and contributing to the development and refinement of guidelines, policies and quality frameworks.