Data Scientist
Department: Technology
Location: Remote
FLSA Status: Exempt
People Leader: No
Travel: Less than 10%
Summary
Company Summary
Accuity partners with hospitals and health systems through a technology-enabled, physician-led model that improves clinical documentation integrity, coding accuracy, reimbursement optimization, and quality outcomes.
Job Summary
The Data Scientist develops, evaluates, and monitors the predictive and generative AI systems that drive automation across Accuity’s clinical documentation and revenue cycle workflows. Reporting to the Vice President, Data Science and AI, this role combines classical machine learning with applied work on large language model and agent-based systems, and is accountable for establishing whether those systems are measurably good enough to be trusted inside a clinical workflow.
Evaluation is the spine of the role. The Data Scientist builds and maintains the ground truth datasets, evaluation frameworks, and metrics that determine production readiness, designs samples so results are comparable and defensible, and analyzes performance by clinical and payer segment rather than in aggregate. This includes independently reproducing and pressure-testing results reported by external AI development partners rather than accepting them as delivered.
This role owns methodology, evaluation, and model development. Production deployment, serving infrastructure, and the engineering of AI systems into live workflows are owned by AI Engineering, and the Data Scientist partners closely with that function to move validated work into production.
Responsibilities
Model Development and Applied AI
- Develop, train, and improve predictive and machine learning models supporting chart triage, prioritization, documentation integrity, and revenue cycle outcomes.
- Maintain and improve models already running in production, including retraining cadence, feature review, threshold tuning, and recalibration as data and workflow change.
- Identify and correct sampling and selection bias in training data, including bias introduced when a model’s own decisions determine which records are subsequently observed.
- Contribute to applied work on large language model and agent-based systems, including prompt and workflow design, tool integration, and systematic failure mode analysis.
- Build feedback loops so findings from clinical audit and production review flow back into model, threshold, and system decisions rather than stopping at a report.
- Prototype and test new approaches, and state plainly and early when an approach does not work.
Evaluation, Ground Truth, and Measurement
- Build and maintain ground truth datasets, including sourcing, annotation coordination with clinical subject matter experts, quality assessment, and remediation of known data defects.
- Design evaluation samples deliberately, understanding when a stratified or weighted sample is appropriate, when a near-production distribution is required, and when results across runs are not comparable.
- Define, compute, and document the metrics that determine production readiness, and ensure another person can reproduce them from source data.
- Analyze performance by clinically meaningful segment, including diagnosis group, service line, payer, and case complexity, rather than reporting aggregate accuracy alone.
- Analyze error in both directions, distinguishing false positives from false negatives and quantifying the clinical and financial consequence of each.
- Maintain evaluation tooling and harnesses so experiments are repeatable and every result is traceable to a specific dataset, configuration, and version.
Production Monitoring and Model Governance
- Monitor deployed models and AI systems for accuracy, calibration, drift, and subgroup performance, and surface issues before they appear as business impact.
- Maintain documentation of model design, assumptions, limitations, training data, and known failure modes to a standard that supports audit and clinical review.
- Support AI governance requirements, including human-in-the-loop design, decision traceability, and evidence of validation.
- Contribute to the definition and computation of metrics that carry commercial or contractual weight, ensuring they are reproducible and auditable.
Clinical and Business Partnership
- Work directly with physicians, coders, and clinical subject matter experts to define correct outcomes, resolve disagreements about ground truth, and validate model behavior against real clinical judgment.
- Translate clinical and operational problems into well-posed analytical problems, and translate results back into terms a clinical or business audience can act on.
- Present findings, limitations, and recommendations honestly, including where results are inconclusive or where the available data will not support the question being asked.
- Help the business understand what a model can and cannot be relied on to do.
External Partner Collaboration
- Work alongside external AI development partners, independently reproducing and verifying reported results rather than accepting them as delivered.
- Review partner methodology, sampling design, and metric definitions, and raise discrepancies clearly and early.
- Absorb knowledge from partner-delivered systems so Accuity can evaluate, maintain, and improve them without ongoing external dependency.
Engineering Partnership and Ways of Working
- Partner with AI Engineering to move validated models and systems into production, providing the specifications, evaluation criteria, and acceptance thresholds that define success.
- Partner with Data Architecture on the data assets, structure, quality, and access required for model development and evaluation.
- Write clean, reviewable, version-controlled code and work in shared repositories rather than personal notebooks.
- Document work so another data scientist can reproduce it without the original author present.
Security and Compliance
- Handle protected health information in accordance with HIPAA, HITRUST, and Accuity policy at all times.
- Follow approved practice for data access, data movement, model routing, and any use of third-party services involving PHI.
- Support audit readiness by producing accurate, timely evidence of validation and evaluation work.
Other Duties as Assigned
- Perform additional responsibilities as needed to support data science, technology, and organizational goals.
Qualifications
Education and Credentials
- Bachelor’s degree in computer science, statistics, mathematics, engineering, data science, or a related quantitative field required.
- Master’s degree in a related field preferred.
Experience
- 4+ years of applied data science, machine learning, or applied AI experience, including work that reached production.
- Demonstrated experience building and evaluating models that were actually used to make decisions, rather than research or coursework alone.
- Experience with evaluation methodology, including ground truth construction, sampling design, and metric definition.
- Exposure to large language model or agent-based systems, including prompt design and systematic evaluation of non-deterministic output, preferred.
- Strong proficiency in Python and SQL.
- Healthcare data experience required; experience with clinical coding, DRG assignment, CDI, or claims and revenue cycle data strongly preferred.
- Experience working with protected health information in a HIPAA-regulated environment preferred.
- Experience with cloud data and machine learning platforms, preferably Microsoft Azure and Databricks, preferred.
Core Competencies
- Evaluation Rigor: Measures with methods that are reproducible, auditable, and honest about what they do and do not prove.
- Data Skepticism: Assumes a dataset has defects until checked, and finds contaminated labels and bad ground truth before they shape a conclusion.
- Scientific Honesty: Reports negative and inconclusive results as readily as positive ones, and does not let a preferred conclusion select the analysis.
- Statistical Judgment: Chooses appropriate samples, methods, and metrics for the question, and knows when a result is not comparable to another.
- Analytical Depth: Looks past aggregate numbers to segment, subgroup, and error-pattern behavior where the real story usually sits.
- Clinical Curiosity: Engages seriously with the clinical and coding domain rather than treating it as a source of features.
- Applied Orientation: Works toward a decision that gets made or a system that gets used, rather than analysis for its own sake.
- Communication: Explains method, result, uncertainty, and limitation clearly to technical, clinical, and business audiences.
- Reproducibility: Produces work another person can rerun and reach the same answer, with code, data, and configuration under version control.
- Collaboration: Works effectively with clinicians, engineers, and external partners, including when the disagreement is technical.
- Adaptability: Moves between classical modeling, applied AI, and data quality work as the problem requires.
- Remote-Work Effectiveness: Communicates proactively, documents decisions, and maintains visibility into work in a fully remote organization.
Additional Requirements
1) Physical Requirements: The requirements described here are representative of those that must be met by an employee to successfully perform the essential functions of this job with or without reasonable accommodations. Unless otherwise indicated, Accuity positions require interaction with people and technology while either sitting or standing. Employees must be able to communicate via phone, email, etc. and sit for extended periods of time, with or without reasonable accommodations. Physical effort and exposure to physical risk are limited to that of an office role / environment.
2) Position and Employment Statement: While this job description is intended to be an accurate reflection of the job requirements, management reserves the right to modify, add or remove duties from a job and to assign other duties as necessary and at any time. All positions at Accuity Delivery Systems, LLC, are at-will employment, and a position description is not a guarantee of a job or of job responsibilities.