Staff-level SDET building automated test frameworks, regression gates, and release-qualification pipelines for GPU inference and distributed LLM serving systems. Day to day involves benchmarking, fault injection, numerical-correctness validation, and CI/CD using Python, Kubernetes/Slurm/Ray, and InfiniBand/NCCL.
You will build automated test frameworks, regression gates, and release qualification pipelines for GPU inference systems. You will validate distributed serving workloads, performance models, numerical correctness, resilience, and recovery; integrate observability and CI/CD; and investigate software and hardware failures.
Responsibilities
Design automated test frameworks, regression gates, and release qualification pipelines
Benchmark and stress-test distributed LLM serving frameworks
Build workload replay and benchmarking tools
Validate model accuracy, precision stability, determinism, and output correctness
Engineer fault-injection and recovery tests for GPU clusters
Integrate test pipelines with telemetry and continuous monitoring
Requirements
Eight or more years of software engineering experience as an SDET, infrastructure quality lead, or systems test engineer
Experience bringing up, provisioning, and validating multi-node GPU clusters
Knowledge of LLM serving engines, prefill and decode disaggregation, KV-cache management, and dynamic batching
Expert Python programming and test automation framework experience
Kubernetes, Slurm, or Ray
InfiniBand, RoCE, or NCCL
Root-cause analysis, stress testing, and node failure simulation