MLOps / Serving Engineer

• Deploy fine-tuned LLMs using vLLM, TensorRT-LLM, or Triton with continuous batching on AWS GPU instances
  
• Build shadow-mode deployment: run fine-tuned model alongside production, log comparison data without impacting live traffic
  
• Execute staged rollout: canary (5%) → gradual ramp (25% → 50% → 100%) with automated rollback on quality degradation
  
• Optimize inference for input-heavy workloads (~17K token inputs, ~130 token outputs): prefill throughput, KV-cache, INT8 quantization
  
• Build monitoring dashboards: latency, throughput, accuracy metrics, cost per request
  
• Design auto-scaling; implement high-availability (2× instances); automated rollback triggers on end-to-end quality metrics
  

Requirements

Benefits