As the Principal AI/ML Engineer — Post-Deployment Governance within AI Validation & Monitoring (AVM), you will serve as the enterprise technical and methodological authority for post-deployment monitoring and reporting, measurement, lifecycle evidence, and Post Deployment Monitoring (PDM) and Post Deployment Reporting Summary (PDRS) governance. You will define risk-proportionate AIA Governance requirements and standards for monitoring readiness; performance and functionality; patient safety; adoption and fidelity; outcomes; change and retesting; metrics, formulas, baselines, targets, and thresholds; subgroup interpretation; uncertainty; and evidence confidence. You will apply data science, AI/ML engineering, statistical, and systems expertise to determine whether evidence is traceable, appropriately interpreted, proportionate to risk, and decision-ready.
Within AIA Governance, you will review drafted monitoring, reporting, measurement, and PDRS content; direct corrections and alternate approaches; consult on complex cases; establish precedent; and escalate unresolved technical or policy issues.
- Provide strategic and technical leadership for enterprise post-deployment governance, measurement, monitoring and reporting, and PDM and PDRS standards.
- Define risk-proportionate requirements across pilot, full implementation, post-deployment change, recurring PDRS, and legacy-product pathways.
- Establish standards for signals, metrics, formulas, baselines, targets, thresholds, uncertainty, evidence confidence, outcomes, and subgroup interpretation.
- Define monitoring-readiness expectations for sources, owners, collection methods, cadence, versions, limitations, lineage, Data Cards, Model Cards, handoffs, and sustainable ownership.
- Provide authoritative SME review of Governance Operations Product Lead assessment content and evidence for policy alignment, sufficiency, traceability, methodological adequacy, and decision readiness.
- Apply data science, statistical, AI/ML engineering, and systems methods to assess metric validity, source fitness, threshold logic, analyses, limitations, and conclusions.
- Review observability, logging, telemetry, workflow signals, version context, change detection, and monitoring and reporting continuity through significant changes.
- Own complex or precedent-setting questions involving monitoring, thresholds, evidence insufficiency, vendor limitations, significant change, revalidation continuity, lifecycle action, PDRS, or CAIO escalation.
- Recommend corrections, alternate methods, interim controls, additional evidence, action plans, re-review, retesting, or revalidation.
- Set precedent, issue final AVM direction, and escalate policy, clinical, cross-domain, or enterprise impasses.
- Lead PDRS templates and rubrics, evidence-confidence and escalation methods, metric libraries, executive presentation standards, and governance acceptance criteria.
- Convert recurring gaps into policy, playbooks, standard findings, rubrics, examples, training, calibration, and Product Lead enablement.
- Define enterprise requirements for TRex workflows, evidence objects, traceability, dashboards, portfolio visibility, and reusable governance capabilities.
- Coordinate with product teams, vendors, platforms, legal, committees, and enterprise groups on methods, tooling, specifications, and ownership.
- Provide clear complex-case findings that communicate limitations, confidence, required actions, and escalation triggers to technical and non-technical audiences.
- Mentor and calibrate engineers, analysts, and Product Leads; foster consistent methods and cross-lane coordination with Validation & Evaluation.
- Support audit sampling, quality assurance, enterprise learning, and continuous improvement while preserving AVM’s review-and-consultation boundary.
- Provide mentorship, guidance, and technical leadership to junior engineers. May have supervisory responsibilities.
- A master’s degree in engineering, computer science, mathematics, health science, or a related field with 7 years of relevant experience, or a bachelor’s degree with 9 years of relevant experience.
- Extensive (7+ years) experience applying AI and machine learning in production healthcare environments or similar highly regulated or technology focused industries, showcasing an acute understanding of healthcare technology.
- Demonstrated leadership in managing complex projects, with a proven ability to navigate intricate project requirements and deliver successful outcomes
- Proven success in fostering collaboration across diverse teams and effectively communicating complex technical concepts to non-technical stakeholders.
- Demonstrated expertise in cloud infrastructure environment and software development tools.
- Experience working with large, complex, and heterogeneous data sets, preferably in healthcare.
- Strong skills in AI/ML techniques and frameworks.
- Expertise with best practices in data engineering, data science, AI Engineering, and the MLOps communities.
- In-depth knowledge of healthcare domain, including clinical workflows, electronic health records, medical terminologies, regulatory requirements, and industry standards.
- Demonstrated leadership in administration, education, software development, and technical reporting.
-
Experience mentoring and training less-experienced team members, coupled with strong interpersonal, communication, and time management skills.
Preferred Qualifications:
- A Ph.D. or other doctorate is preferred.
- Experience with healthcare industry informatics standards, best practices, and common data models. Participation in national or international standards organizations or other domain-specific professional organizations, or extensive implementation experience with common data, development, and deployment standards.
- Excellent communication, collaboration, and stakeholder management skills, with the ability to effectively engage with diverse stakeholders and translate complex technical concepts and results to non-technical audiences.
- Demonstrated experience leading technical/quantitative teams in a regulated environment.
- Familiarity with systems or quality engineering best practices, regulatory standards, and compliance frameworks, with the ability to adapt these effectively to different project scenarios.
- Demonstrated experience creating risk management files and verification/validation strategies for digital health technology products within the healthcare industry.
- Demonstrated expertise in user-centered design, human factors engineering, usability testing methodologies, and evaluation across AI product development. Ability to lead expert reviews using established usability practices and methods. Presents findings in easy-to-understand terms for the business or clinical practice.
- Strong problem-solving abilities, critical thinking skills, and a passion for driving innovation and positive change in healthcare through AI technology.
- Demonstrated hands-on leadership using the TRex assessment application to govern AI tools deployed in EPIC, ANIMATE, and comparable clinical environments, including post-deployment standards, metric thresholds, evidence confidence, significant-change review, revalidation, and executive escalation.