Senior Manager, Machine Learning (Data Operations)

About the role

Responsibilities

• Labeling quality & methodology: Define annotation guidelines, taxonomies, and edge-case protocols for each labeling program. Establish gold standard datasets, inter-annotator agreement (IAA) targets, and audit sampling processes. Identify and remediate mislabeled data in existing datasets.
• Platform & tooling: Serve as the primary user and requirements driver for Label Studio — defining project configuration needs, workflow designs, pre-labeling pipeline requirements, and integration points with ML infrastructure. Partner with the data engineering team that builds and maintains the platform.
• Cross-functional partnership: Work with ML engineers, data scientists, and product managers to translate model requirements into well-structured labeling tasks. Challenge teams on task design when labeling instructions are ambiguous or likely to produce unreliable labels.
• Vendor management: Source, onboard, and manage external labeling vendors and BPOs in coordination with Coalition's operations team. Set quality SLAs, run calibration sessions, and manage feedback loops to labelers. Hold vendors accountable to accuracy, not just throughput.
• Measurement & improvement: Define and track operational metrics — label accuracy, IAA scores, cost per label, turnaround time — and use them to drive continuous improvement. Identify opportunities for active learning, model-assisted labeling, and pre-annotation to reduce cost without sacrificing quality.

Skills and Qualifications

• 5+ years in ML data operations, data labeling, or a related field (ML engineering, data science, or data engineering with heavy labeling exposure)
• Deep understanding of annotation quality frameworks: IAA, consensus labeling, gold standard evaluation, error taxonomy, and calibration workflows
• Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, Prodigy, or similar acceptable)
• Track record managing outsourced labeling vendors or BPOs for ML data production
• Familiarity with common ML labeling tasks: text classification, NER, document extraction, intent detection
• Comfortable working in Python and SQL; bonus if you've built tooling around labeling workflows or quality measurement
• Strong opinions on what makes labeled data good or bad, and the willingness to push back when it's bad
• Experience in insurance, cybersecurity, or fintech is a plus but not required

Compensation

As a remote-first organization, our compensation reflects the cost of labor across several Canadian geographic markets.

In Alberta, British Columbia & Ontario the base salary for this position ranges from $136,800/year up to $186,800/year.

For all other locations, the base salary for this position ranges from $123,100/year up to $153,900/year.

Consistent with applicable laws, an employee's pay within this range is based on a number of factors, which include but are not limited to relevant education, skills, job-related knowledge, qualifications, work experience, credentials, and/or geographic location. Your Recruiter can share more on the specific target salary range for your location during the interview process. Coalition, Inc. reserves the right to modify this range as needed.

Vacancy Status: This posting is for an existing vacancy.

Use of AI: We utilize AI-assisted tools to help us organize and review the high volume of applications we receive. However, our human recruiting team makes all final hiring decisions and interview selections, valuing personal connection over algorithms.

Application Updates: Consistent with our commitment to transparency, if you are interviewed for this role, we will provide a status update on your application within 45 days of your final interview.

Perks

• 100% medical, dental, and vision coverage
• Flexible PTO
• Annual home office stipend and WeWork access
• Mental & physical health wellness programs like Headspace, Lumino, and more!
• Competitive compensation and opportunity for advancement