Analyst, Applied AI
heymilo · Colombo ·
- Work mode
- Hybrid
- Category
- AI engineering
- Company size
- 11-50
Designs and evaluates AI interview tasks by creating realistic workflows, grading rubrics, and ground-truth datasets to assess large language models (LLMs) in real-world hiring scenarios, ensuring accurate and meaningful performance measurements.
HeyMilo is building AI interviewers that automate and improve hiring through conversational AI. We work closely with companies to bring AI into real hiring workflows. We also run an applied AI team that studies where today's models succeed and fail across verticals.
The Role
We're hiring an Analyst to join our Applied AI team in Colombo. You'll design the tasks, rubrics, and datasets we use to evaluate AI systems on real-world workflows, and define the ground truth that decides whether a model's attempt counts as correct.
The quality of everything we build downstream depends on the quality of the tasks and ground truth you produce.
What you'll do
Design realistic, multi-step tasks that test AI systems on real-world workflows, and author the grading rubrics that score them
Build and curate ground-truth datasets from real operational data and domain research
Review model attempts against your tasks: analyze failure modes, catch scoring loopholes, and identify where models pass for the wrong reasons
Calibrate task difficulty so results separate models in a meaningful way
Analyze score distributions across models and turn results into clear, defensible findings
Contribute to research write-ups and published evaluations alongside our engineers and academic reviewers
What we're looking for
Master's or PhD in AI, Machine Learning, Computer Science, Statistics, or a closely related quantitative field
Strong understanding of how large language models are trained and evaluated
Hands-on experience assessing LLM outputs: evals, annotation, red-teaming, or benchmark work
Comfortable with Python and data tooling for building datasets and analyzing results
Exceptional attention to detail and intellectual honesty; you flag a broken task even when the scores look good
Clear written communication; you can explain a subtle failure mode to both engineers and non-technical readers
Bonus
Familiarity with reinforcement learning concepts (reward design, verifiable rewards)
Experience contributing to published research or open evaluation suites
Domain knowledge in recruiting, staffing, or other operations-heavy industries
Why join
Ground-floor role on a new applied AI team with direct ownership of what we measure and publish
Work reviewed by and published alongside experienced researchers
High visibility, fast-paced, execution-driven environment
Competitive pay and benefits