WORK LOCATION: Remote, CALIFORNIA USA

WORK SCHEDULE: Mon - Fri, business core hours 

START DATE: ASAP

COMPENSATION:  $127,000 - $157,000 per hour

We are seeking a Data Scientist specializing in analytics and statistical analysis to join our AI Rater Program. This role is central to uncovering insights from rater-generated data and helping our client improve AI-driven ad relevance and related projects. You will analyze operational and labeling data to identify trends, surface bias, and design frameworks that strengthen both program performance and client outcomes.

This is a strategic role where you will design models, build analytics frameworks, and collaborate closely with linguists, program managers, and raters. Your insights will guide operational decisions, shape client recommendations, and drive innovation at the intersection of AI, linguistics, and human judgment.

Main Responsibilities:

• Develop, implement, and maintain program and rater performance metrics (e.g., quality, bias, consistency, efficiency).

• Design and analyze experiments and statistical tests to uncover drivers of performance and label patterns.

• Build dashboards, reports, and automated monitoring systems in Looker Studio, Sisense, or similar BI tools.

• Identify and highlight potential biases or preferences in labeling data (e.g., topic-specific or cultural/linguistic bias).

• Collaborate with linguists, raters, and program managers to translate findings into operational and client-facing insights.

• Conduct applied analytics using Python, SQL, and statistical modeling, with opportunities to apply predictive modeling, NLP, and bias detection algorithms.

• Manage and explore datasets ranging from structured spreadsheets/CSVs to larger-scale rater outputs (scaling to millions of rows as needed).

• Partner with client teams by supporting linguists and PMs in meetings where technical depth and context are required.

• Drive process improvements and automation, surfacing insights that reduce variability, improve efficiency, and strengthen program resilience.

Qualifications: 

• Strong proficiency in Python, SQL and R

• Applied expertise in statistical modeling and experimental design (e.g., sampling, confidence intervals, agreement metrics).

• Experience working with moderate-to-large datasets, with ability to scale analyses as data volume grows.

• Familiarity with BI/visualization tools such as Looker Studio, Sisense, Tableau, or PowerBI.

• Experience in Google Cloud Platform (GCP) or other cloud data environments preferred.

• Excellent ability to communicate insights clearly through reports, dashboards, and presentations to both technical and non-technical stakeholders.

• Strong collaboration skills, with experience working in cross-functional teams (linguists, operations, engineering, client teams).

• Comfortable supporting client meetings as a subject-matter expert, while primarily focused on internal analysis and insights.

Preferred Skills

• Experience with bias detection, NLP, or applied AI/ML models.

• Knowledge of rater evaluation methods and quality metrics such as Krippendorff’s alpha, chance-corrected agreement, or structured sampling.

• Background in quality, linguistic or cultural data analysis is a plus.

• Exposure to human computation programs or crowdsourced labeling workflows.

Studies & Experience:

• English B2 - Required.

• Oral and written comprehension.

• Appropriate use of language.

• 2 years in a quantitative role (Data Scientist, Product Analyst, or similar) with proven ability to navigate ambiguous environments.

• Master’s degree in Statistics, Mathematics, Computer Science, Economics, or a related field OR equivalent practical experience.

• An advanced degree is a plus but not required.

Benefits and Perks:

• Competitive salary, and benefits package.

• Opportunity to work at the intersection of AI, linguistics, and human judgment, driving impact in a rapidly evolving industry.

• Remote position

• Exposure to cutting-edge AI projects with a global client.

• Growth opportunities as the program scales.

• Annual merit increase based on performance.