We're building the evaluation platform that will serve all of Apple's generative AI and agent systems. Evaluating non-deterministic AI systems is one of the hardest unsolved problems in production ML — and one Apple has to get right at scale. We're building the platform that makes it tractable for every team here.
This is a hands-on engineering role with a lot of autonomy. You'll write a lot of Python and own meaningful pieces of the platform end-to-end. You'll be partnering closely with research engineers, model and serving teams, product and feature teams, and the infra and data platform groups this work integrates with.
Build and ship: Take ownership of features and services within the evaluation platform: APIs, SDKs, orchestration components, evaluation runners. You'll have the room to make calls on your own work and the support to deliver it well.
Productionize ML research: Partner with research engineers to take their prototype code and turn it into reliable services. You'll learn their world quickly and translate research patterns into clean Python that holds up under real load.
Move fast, responsibly: You'll get scoped problems with room to figure out the how. We trust you to balance speed with care, to know when something needs a quick prototype and when it needs a design doc, tests, and a careful rollout.
Improve as you go: Notice the rough edges and pick them up. The flaky test, the slow build, the confusing API, the runbook that's out of date. We want someone who leaves the codebase a little better every week.
Developer experience: Help build the SDKs and abstractions that other Apple teams use to evaluate their models and agents. You'll feel the friction of bad ergonomics directly, which puts you in a great position to fix it.
Operational ownership: Your code runs in production. You write the tests, set up the CI, add the metrics, and stay close when something breaks. You don't need to be an SRE, but you take care of what you ship.
Minimum Qualifications
- 4-8 years of software engineering experience building and shipping production services.
- Strong Python. You're fluent with FastAPI, Pydantic, and the modern Python ecosystem. You write code that's clean, tested, and easy for the next person to pick up.
- Builder's mindset. You enjoy shipping. You're comfortable iterating quickly on scoped problems and knowing when to slow down for the parts that need it.
- Fluency with AI coding tools. You actively use tools like Claude Code (or equivalents) in your day-to-day workflow, including features like skills, slash commands, and agent-style workflows. You have a good intuition for when to lean on them, when to steer them, and how to get high-quality output.
- Familiarity with the agentic LLM landscape. You stay current on how modern LLM systems work in production — tool use, MCP servers, agent frameworks, context management, multi-step reasoning. You can hold a real conversation about the tradeoffs.
- Hands-on evaluation experience. You've built evaluations for your own agents or LLM systems, or you've worked with evaluation orchestration frameworks like Inspect, Braintrust, LangSmith, Promptfoo, or equivalents (including internal tooling). You understand what makes an evaluation trustworthy vs. theatrical.
- Real working knowledge of LLMs in production. You're comfortable with prompt iteration, dataset curation, judge models, and statistical reasoning about non-deterministic outputs. You understand the lifecycle around models even if you haven't trained them yourself.
- Solid engineering fundamentals. You understand testing, CI/CD, containerization (Docker), and basic observability. You've shipped services that others depend on and stayed close when they broke.
- Clear communicator. You write clear PRs, ask sharp questions, and flag blockers early. You're comfortable disagreeing thoughtfully and changing your mind when the argument is good.
- Ownership. When something is broken or unclear, you tend to pick it up rather than wait. You either move it forward or surface it clearly.
Preferred Qualifications
- Experience working on developer platforms, internal tools, or SDKs
- Production experience with LLM/agent systems — building, evaluating, or operating them
- Familiarity with job orchestration frameworks (Temporal.io, Airflow, or similar)
- Distributed compute experience (Ray, Dask, or Kubernetes-based job systems)
- Experience with experiment tracking or ML lifecycle tooling (Weights & Biases, MLflow, etc.)
- Startup or early-stage experience where you wore multiple hats and shipped under constraint