LLM Red-Teamer

LLM Red-Teamer

Role Type: Contractor

Location: Remote

Scope of Work

Develop complex, adversarial multi-turn conversations and task-based scenarios aligned with detailed project specifications.

Author clear, precise evaluation rubrics to rigorously assess model responses against defined behavioral targets.

Iteratively test conversations and tasks against frontier LLMs, escalating difficulty and nuance until the desired quality threshold is achieved.

Deliver comprehensive task packages, including transcripts, target behaviors, binary rubrics, and supporting rationale or evidence.

Validate LLM outputs, documenting model strengths and failure modes relative to the project specification.

Maintain calibration with team leads and quality control contacts as project requirements evolve.

Contribute independently, producing high-quality deliverables at a steady and consistent pace.

Preferred Qualifications

Exceptional written English skills, with clarity, precision, and strong structural organization.

Prior experience in AI human data environments (RLHF, SFT, evaluations, annotation, or prompt engineering).

Deep familiarity with large language models, including the ability to anticipate and identify common failure patterns.

Demonstrated ability to work autonomously, interpreting and executing complex specifications with minimal oversight.

Proven critical thinking and meticulous attention to detail.

Experience designing evaluation items or rubrics is advantageous.

Background in writing-intensive or analysis-centric fields such as research, editorial, technical writing, or quality assurance is a plus.

Compensation Structure

Compensation is output-based; experts are paid per task that meets the project specifications. The time required to complete work may vary depending on the expert's experience and workflow. Minimum submission requirements apply. Experts must submit a minimum of tasks per week.

Start Timeline & Availability

We typically fill roles within 48 hours and are looking for experts ready to jump in right away. If selected, we expect you to start your first tasks within 24–48 hours of completing onboarding.