Senior AI Data Operations Analyst

Overview

NOTE: This is a remote role but must work EST hours. This is for a MAT leave. 

We are seeking a Senior AI Data Operations Analyst to join our media clients Services Recommendations team. This team owns the Home page experience—what millions of listeners see and interact with every day across music, podcasts, audiobooks, and more.

In this role, you will sit within the core product organization to ensure our recommendation engines and AI/LLM-powered features deliver high-quality, relevant experiences. You will act as a key driver of data quality and data annotation strategy, bridging the gap between product managers, data scientists, and engineers to build the datasets needed to train and evaluate our next-generation agentic features.

This is a hands-on, highly collaborative role focusing on qualitative data analysis, evaluation frameworks, and human-in-the-loop AI quality.

What You’ll Do

• Execute & Champion Data Annotation: Perform hands-on data annotations and lead larger, cross-functional annotation sessions to generate high-quality training datasets for recommendation models.
• Define Quality Standards: Establish criteria, metrics, and qualitative success measures for core Home page features and the LLM judges evaluating them.
• Run Structured Qualitative Evaluations: Design and execute qualitative testing using internal tools, keeping human judgment at the center while utilizing AI/LLM tools to scale evaluation efforts.
• Support Flagship AI Initiatives: Partner on major, publicly announced product initiatives (e.g., taste profile and agentic home experiences).
• Build Reusable Frameworks: Improve and maintain evaluation processes, guidelines, and documentation adopted across product groups.
• Communicate Insights: Present qualitative findings, data trends, and quality risks clearly to product, design, research, and engineering leads.

Who You Are

• Data Quality / Annotation Background: Proven experience in data annotation, data quality, or product/content quality analysis with direct ownership over evaluation workflows.
• Strong Qualitative & Analytical Skills: Highly comfortable running structured qualitative evaluations, handling large data sets, and turning subjective feedback into clear quality metrics.
• AI/ML Familiarity: Strong functional understanding of machine learning product development and how ML/LLM/agentic features are evaluated and trained. (Note: You do not need to write code or build models, but you must understand how data quality impacts AI outputs).
• Process & Framework Builder: Demonstrated ability to create or refine evaluation frameworks and guidelines that help team members maintain quality standards.
• Strong Communicator: Excellent written and verbal communication skills, comfortable presenting findings and guiding cross-functional teams through evaluation initiatives.
• Tool Proficiency: Comfortable working with standard data tools (Excel, Google Sheets) and learning internal proprietary annotation/eval platform tools.

Nice-to-Haves:

• Experience in media, music, or streaming entertainment platforms (though data annotation experience in other tech industries is fully welcome).

The target hiring compensation range for this role is $47.00/hr to $51.00. Compensation is based on several factors including, but not limited to education, relevant work experience, relevant certifications, and location.

**About Skill:**

Skill connects the best professional, IT, engineering, financial and administrative talent with the world’s biggest brands. Our eligible talent get access to benefits such as health benefit contributions, retirement plans with match and flexible spending accounts.

Skill is an equal-opportunity employer. We evaluate qualified applicants without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, and other legally protected characteristics. We’re about creating an inclusive environment-one where different backgrounds, experiences, and perspectives are valued, and everyone can contribute, grow their careers, and thrive.

#LI-CF1