About the Company
Our client is a seed-stage, stealth-mode startup building a high-performance AI inference platform from the ground up. The founding team brings deep AI-infrastructure experience and is tackling the hardest problems in LLM serving — scheduling, KV cache management, request routing, and the runtime systems that power model inference at scale — with Rust at the core of the stack. This is a ground-floor opportunity to shape the architecture of a fast, reliable inference system without legacy constraints.
Recently founded · Small founding team · Industry: AI infrastructure / LLM inference
The Role
You'll join a small, fast-moving team building a new inference system from scratch. This is a role for a systems engineer who lives and breathes inference internals — attention, KV cache, batching, scheduling — and wants to own the whole stack rather than a narrow slice.
What you'll be doing
- Build a new inference runtime from scratch in Rust, owning batching, scheduling, request routing, and the full serving stack.
- Design and implement KV cache management, prefix caching, and optimizations that cut latency and cost per token.
- Scale serving across GPUs and nodes, tackling multi-GPU and multi-node challenges directly.
- Profile, benchmark, and ship performance improvements across the entire inference pipeline.
- Work closely with the founding team on the core architectural decisions that define the platform.
Tech stack: Rust, Python, PyTorch, C++, Go, vLLM, SGLang, TensorRT-LLM, CUDA, Triton, NCCL
Requirements
- 2–10 years of experience as a backend or distributed systems engineer.
- Hands-on experience building, operating, or optimizing LLM inference or serving systems at the engine, router, or runtime layer — beyond just calling hosted APIs.
- A deep working knowledge of transformer inference internals: attention, KV cache, batching, scheduling, and where the real bottlenecks are.
- Performance-critical backend or distributed systems experience where latency, throughput, and cost were first-order concerns.
- Hands-on time with a production inference engine (vLLM, SGLang, or TensorRT-LLM) plus strong systems-language skills (Rust, C++, Go, or systems-level Python/PyTorch). If you haven't used Rust yet, you should be able to become productive in it within a few weeks of joining.
Nice to Haves
- Time on an inference team at a model provider, accelerator vendor, or research lab, or open-source contributions to vLLM, SGLang, or Dynamo.
- A CS or systems degree from a strong program.
- Production Rust experience, CUDA/Triton kernel work, multi-GPU or multi-node serving (NCCL, NVLink, RDMA), prefix caching, speculative decoding, or prefill/decode disaggregation.
Why Join
- Own the architecture of a brand-new inference system with no legacy constraints.
- Solve the hardest problems in LLM serving, from KV cache management to request routing.
- A performance engineer's dream: latency and cost per token are the whole game.
- High visibility on a small, elite team where your code powers the core engine.
Details
- Location: Palo Alto, CA
- Work policy: Full-time, on-site five days a week
- Compensation: $230K–$350K + equity
- Visa sponsorship: Open to visa transfers (OPT, H-1B) and new sponsorships (new H-1B, TN)
- Employment type: Full-time