Member of Technical Staff
Thomas Talent Network · San Francisco, California ·
- Category
- Software engineering
- Visa sponsorship
- No
Member of Technical Staff (Research Scientist) at an AI healthcare software startup, focused on building benchmarks and evaluation methodologies for large language models — evaluating new model releases, constructing datasets with labelers, writing white papers, and scaling auto-evaluation. Core tech: Python, PyTorch/TensorFlow, Git, LLM infrastructure.
Our client, an AI healthcare software startup, is seeking a Member of Technical Staff (Research Scientist) to join their team. In this role, you'll be at the forefront of developing benchmarks and evaluation methodologies for large language models, helping shape how the most advanced AI systems are tested and validated before deployment in enterprise environments.
Key Responsibilities:
- Evaluate new AI models as they're released (e.g., DeepSeek, Gemini)
- Create new benchmarks from scratch, including hiring labelers, constructing datasets, and writing white papers
- Improve underlying methods for auto-evaluation of generated text
- Work closely with engineering to implement and scale evaluation methodologies
- Collaborate with top AI labs and enterprise customers to understand evaluation needs
Qualifications:
- 0-3 years of research experience in AI/ML, particularly in benchmarking, evaluation methodologies, or language models
- Experience working in team environments, including development sprints, Git best practices, and pull request review
- Previous experience at AI startups, research labs (e.g., Anthropic, DeepMind), or as a founder/early employee at your own company is highly valued
- Published research (especially in NLP or benchmarking) is valued but not required; applied work is prioritized over purely academic publications
- Experience building new benchmarks and evaluation methodologies preferred
- Academic background in Computer Science, Machine Learning, or a related field (Master's or PhD preferred); candidates with a Bachelor's degree and 0-3 years of relevant experience will also be considered
- Background from top tech/AI programs preferred
- NLP research experience with publications in reputable journals preferred
- Strong Python programming skills, with the ability to build clean, maintainable code
- Experience with deep learning frameworks (PyTorch, TensorFlow) and techniques such as diffusion models and language modeling
- Interest in and familiarity with LLM infrastructure
- Experience with Django, Flask, or other Python-based HTTP servers is a plus
- Strong communication skills; able to give input to others and receive/integrate feedback effectively
Job Reference: X985X343
Package Details: Salary & equity
Visa Sponsorship: Open to visa transfers (e.g. OPT, H1B transfers), NOT new sponsorship. Will support relocation or transportation as needed.