Interested in building the distributed systems that let customers customize foundation models at scale? The SageMaker Training and Model Customization team builds the services customers use to fine-tune and post-train models against their own accuracy, reliability, and cost targets. We are hiring a Software Development Manager to lead the team that owns those capabilities.
Model customization is the fastest-moving layer of the machine learning stack. Post-training has moved from supervised fine-tuning to reinforcement learning against verifiable rewards, and from single-turn tasks to agentic training where a model learns by acting in an environment over long trajectories. Each shift changes the shape of the workload. Reinforcement learning puts an inference engine inside the training loop. Agentic training adds environments and tool calls, and moves the bottleneck from one run to the next. Your team turns each new technique into a capability customers can use, without rebuilding the platform every time the research moves.
Underneath the techniques, this stays a hard distributed systems problem. Training jobs run across large accelerator fleets for days at a time, and a single node failure can stall the job. The team works in PyTorch and across frameworks including Verl, FSDP, Megatron and vLLM, and contributes upstream to the open source projects the platform depends on.
As the manager you own the team and its charter. You will hire and grow engineers, set technical direction with your senior engineers and product managers, and decide what the team does and does not build. You own the roadmap and defend its tradeoffs with leadership. You own the service in production, including on-call health, operational metrics, and the requests that come with sitting in the critical path of customer training workloads.
Key job responsibilities
- Own the team's charter and roadmap, and defend its tradeoffs with senior leadership.
- Hire, develop and retain engineers in a specialized field with a scarce talent pool.
- Set technical direction with your senior engineers, applied scientists and product managers.
- Decide what the team builds, and what it does not.
- Turn new post-training techniques into supported customer capabilities on a predictable schedule.
- Own the service in production, including on-call health, availability and operational metrics.
- Improve training throughput and cost per run across large accelerator fleets.
A day in the life
Your morning starts in a design review. The team is working out how to keep an inference engine and a training engine in the same loop for reinforcement learning without leaving accelerators idle, and the answer will shape the future of the platform. You deep dive into possible scenarios, because at this scale a node dies routinely and recovery has to be normal rather than exceptional.
Then a charter call. A new post-training technique is three months out of the research literature and two customers are already asking for it. With your senior engineers and product manager you decide whether it becomes a first-class capability, a recipe on top of what you already have, or a no. That decision is yours to make, and the industry does not wait for it.
You spend an hour in 1:1s, including a career conversation with an engineer who wants to own the parallelism work. After lunch you sit a hiring debrief and make the call. Later you meet the applied scientists to understand what they have validated, and you argue about sequencing.
You close the day writing a page for leadership: what you are not building this half, and why that is the right trade.
About the team
We build the managed services customers use to customize foundation models. A customer brings a task, a dataset, and a way to score a good answer. Our services run the post-training loop on their behalf. We own that entire path, from the customer-facing API down to the training session infrastructure and the accelerator fleet it runs on.
That makes this a foundational charter. Model customization is how a customer turns a general model into one that is theirs, and every customization experience on SageMaker is built on what we deliver. When we add a technique, it becomes available to every customer on the platform. When we make the training loop faster, every customer's job gets cheaper.