Valce Talent Solutions · Mexico, MX ·
Monitor and maintain AI/ML models and agent-based systems in production, handling incident management, governance, and CI/CD pipelines using Python, Docker, Azure, and monitoring/logging tooling.
KRISLITE PTE LTD · Singapore
Copart · Dallas, TX - Headquarters
Steampunk · McLean, VA, US
AI Engineer – AI Ops (Monitoring AI Systems in Production)
Role purpose
An AI Engineer in AI Ops & Governance is responsible for operating, monitoring, and maintaining AI models and agent-based systems once they are live in production. The role ensures AI systems remain reliable, performant, compliant, and aligned with business expectations over time, closing the “last‑mile” gap between model development and sustainable production use
Key Responsibilities
Production Monitoring & Health
Monitor AI models and agents in production for performance, latency, errors, and availability.
Track statistical health indicators such as model drift, data distribution changes, and output stability.
Observe business KPIs linked to AI behaviour (e.g. accuracy impact, false‑positive cost, efficiency).
Incident Management & Recovery
Detect and triage production incidents related to AI behaviour or degradation.
Execute rollbacks, throttling, or model disabling where thresholds are breached.
Support root‑cause analysis and post‑incident reviews to prevent recurrence.
Model & Agent Lifecycle Operations
Support deployment, versioning, and release of AI models and agents using CI/CD‑style pipelines.
Maintain registries and metadata covering model ownership, lineage, risk classification, and approvals.
Support models and agents move safely through environments (dev → test → production).
Governance, Risk & Compliance
Ensure AI systems adhere to Responsible AI principles, internal controls, and audit requirements.
Maintain audit trails, logs, and approval artefacts required by risk, compliance, and regulators.
Support fairness, bias, explainability, and transparency monitoring in production.
Tooling & Platform Integration
Integrate AI systems with monitoring, logging, and alerting platforms (e.g. dashboards, metrics stores).
Work with cloud infrastructure (containers, event streaming, APIs) supporting scalable AI operations.
Collaborate with product, engineering, and data teams to standardise AI Ops patterns and blueprints.
Skills & Experience (Baseline)
Strong Python skills and experience supporting ML or LLM‑based systems.
Understanding of Model Ops / MLOps, especially the operational phase after deployment.
Experience with:
Monitoring and logging systems
CI/CD pipelines
Containerised deployments (e.g. Docker‑based runtimes)
Familiarity with cloud platforms (Azure preferred) and production troubleshooting.
Ability to work cross‑functionally with product, data science, engineering, and risk teams