Senior Psychometrician and Data Manager at University of California, San Francisco – San Francisco | Jobyssey

Multitudes is seeking a senior Psychometrician and Data Manager to lead the design, validation, analysis, and strategic use of data across its educational implementation initiatives. This individual will oversee the organization's measurement framework and data systems, ensuring that assessments and analytics meet the highest standards of scientific rigor, validity, reliability, equity, accessibility, and usability.

Reporting to the Chief Operating Officer (COO), the Psychometrician and Data Manager serves as the scientific and technical lead for the Multitudes Project across its research, education, and product initiatives. The role combines expertise in psychometrics, educational measurement, quantitative methods, and applied data science with strategic leadership to translate complex statistical findings into actionable insights that advance the development and implementation of the Multitudes screening platform.

The ideal candidate holds deep expertise in psychometrics, educational measurement, quantitative psychology, statistics, or a related discipline, with experience in literacy, education, neuroscience, developmental science, or behavioral research. This individual will collaborate across multidisciplinary teams, contribute to assessment development, oversee data infrastructure, mentor analysts, and ensure data quality, governance, and regulatory compliance across multiple institutional partners.

In addition, we expect the ideal candidate to lead the organizational data architecture and analytics strategy. Build together with their team, scalable systems for data collection, management, quality control, visualization, and reporting. Oversee complex statistical analyses and interpretation of longitudinal and intervention data. Develop dashboards, technical reports, white papers, and presentations for internal and external stakeholders. Translate data into actionable recommendations for program improvement, policy, and research dissemination.

Because our implementation work is grounded in research, we expect the ideal candidate to collaborate with interdisciplinary teams, including neuroscientists, literacy researchers, educators, clinicians, and software developers. Contribute to grant writing, research publications, conference presentations, and external partnerships. Support the development of innovative measures that integrate educational, cognitive, and behavioral data and help establish Multitudes as a national leader in evidence-based measurement and evaluation.

In addition, the ideal candidate is expected to build and mentor a high-performing analytics and research team. Establish best practices for data governance, reproducibility, documentation, and ethical data use, and supervise external research and evaluation partners as needed.

The role will provide psychometric and measurement leadership for the development, validation, refinement, and scaling of Multitudes assessments and related measurement systems. Responsibilities will include supporting the research team in designing and overseeing studies of reliability, validity, measurement invariance, item functioning, growth, and sensitivity to intervention effects, while applying appropriate modern psychometric and statistical approaches, including Item Response Theory (IRT), Classical Test Theory, multilevel and structural equation modeling, latent growth modeling, and Bayesian methods. The role will also help ensure that assessments and measurement approaches are culturally responsive, equitable, and appropriate for multilingual and diverse student populations.

The role will lead data strategy, analytics, and research collaboration, including developing scalable approaches to data collection, management, quality assurance, analysis, visualization, and reporting. This includes overseeing analyses of large-scale longitudinal and intervention datasets and translating findings into actionable recommendations for assessment refinement, program improvement, policy, and research dissemination. The role will collaborate closely with interdisciplinary partners—including literacy researchers, neuroscientists, educators, clinicians, data scientists, and software developers—and contribute to grant proposals, peer-reviewed publications, technical reports, white papers, conference presentations, and external research partnerships. The work will also support the development of innovative measures integrating educational, cognitive, behavioral, and other relevant data.

The role will strengthen research and analytics infrastructure and organizational capacity by establishing best practices for data governance, reproducibility, documentation, ethical data use, fairness, and bias evaluation. Responsibilities may include building and mentoring a high-performing research and analytics team and overseeing external research or evaluation partners. The ideal candidate will bring advanced training in psychometrics, quantitative psychology, educational measurement, statistics, biostatistics, or a related field; substantial experience with assessment development, validation, advanced statistical modeling, and large and complex datasets; and strong quantitative programming and communication skills. Experience in literacy, language development, bilingual education, developmental science, neuroscience, educational technology, responsible AI, or school-based research is particularly valuable.

Department Overview

Multitudes is a State of California-funded initiative that bridges neuroscience and education to identify reading difficulties, including dyslexia, in early childhood and support timely intervention. The project has developed English and Spanish screening instruments and is expanding into mathematics, additional languages, and other domains related to brain health.

Multitudes is part of the UCSF Dyslexia Center within the Edward and Pearl Fein Memory and Aging Center. The Dyslexia Center brings together neuroscientists, clinicians, geneticists, and educators to improve diagnosis, intervention, and lifelong outcomes for individuals with learning differences through multidisciplinary research spanning language, cognition, aging, and brain health.

The Department of Neurology is one of the nation's leading academic neurology departments, with over $120 million in annual operations and the highest NIH funding among U.S. neurology departments. Its mission is to deliver exceptional clinical care, conduct transformative research, and educate future leaders in neurology and neuroscience.

of time

Essential Function (Yes/No)

Key Responsibilities

(To be completed by Supervisor)

Yes Lead and oversee psychometric analyses using Classical Test Theory, Item Response Theory (IRT), equating, measurement invariance, latent growth modeling, multilevel modeling, structural equation modeling, Bayesian methods, and related approaches.

yesDirect and provide technical oversight to the research team in planning research studies, including sampling strategies, instrument development, validation studies, longitudinal analyses, and intervention evaluations.

yesLead and oversee the development, validation, refinement, and scaling of educational assessments that are reliable, valid, equitable, culturally responsive, and appropriate for multilingual populations.

yesDefine technical requirements for research studies, product development, and platform analytics.

yesDevelop and oversee scalable data architecture, pipelines, quality assurance processes, dashboards, and reporting systems. Manage complex datasets supporting research collaborations across UCSF, schools, universities, and external partners.

yesTranslate complex quantitative findings into actionable recommendations for researchers, educators, clinicians, and product teams.

yes

Supervise and mentor junior analysts and research staff.

Assign and prioritize work; establish deliverables and performance expectations; review work products and technical approaches; provide coaching and technical direction; manage workload and resource allocation; address performance and workflow issues; and ensure staff accountability for data quality, documentation, timelines, and program objectives.

yesCollaborate on scientific publications, grant proposals, technical reports, conference presentations, and external partnerships.

yesEstablish best practices for data governance, reproducibility, documentation, ethical data use, and regulatory compliance.

yesSupport long-term data strategies, including infrastructure for artificial intelligence, predictive analytics, and precision education initiatives.

100%

(To update total %, enter the amount of time in whole numbers (without the % symbol - e.g., 15, 20) then highlight the total sum (e.g., 1%) at the bottom of the column and press F9. The total sum should add up to 100%.)

Required Qualifications

• Bachelor's degree in related area and five or more years of relevant experience which includes three or more years of supervisory experience and / or equivalent experience / training
• 5+ years of advanced experience in psychometrics and quantitative research.
• Experience managing large and complex datasets
• Strong communication skills with the ability to explain technical findings to non-technical audiences.
• Familiarity with highly complex logical data design and data mapping or data conversion.
• Experience building and leading multidisciplinary research and analytics teams.
• Demonstrated ability to work with others from diverse backgrounds. Demonstrated effective communication and interpersonal skills. Demonstrated service orientation skills.
• Demonstrated ability to communicate technical information to technical and non-technical personnel at various levels in the organization.
• Self-motivated and works independently and as part of a team. Strong organizational skills.
• Demonstrated advanced problem-solving skills. Able to learn effectively and meet deadlines.
• Advanced analytical and design skills, including the ability to abstract information requirements from real-world processes to understand information flows in computer systems.
• Ability to represent relevant information in abstract models. Critical thinking skills and attention to detail.
• Demonstrated skill in managing technical staff.
• Requires critical thinking, persuasion / negotiation, mentoring, leadership / management, problem solving, human resources policies and procedures relating to management responsibilities, verbal and written communication, quantitative abilities, analytical, strong technical database systems skills.
• Demonstrated expertise in assessment development, IRT and advanced statistical modeling, longitudinal data analysis, validation studies, and educational or behavioral measurement.
• Strong programming and analytic skills in R, Python, SAS, or SPSS.

Preferred Qualifications

• PhD in Psychometrics, Quantitative Psychology, Educational Measurement, Statistics, Biostatistics, or related field.
• Experience in data integration and the specification of data services.
• Experience in literacy, language development, bilingual education, neuroscience,
• Experience working with schools, universities, nonprofit organizations, or public-sector initiatives.
• Familiarity with AI-enabled educational technologies, digital assessment platforms, and predictive analytics.

Explore more jobs