Senior Python Engineer
Mindrift · Oman ·
- Seniority
- Senior
- Category
- Software engineering
- Experience
- 8+ years
Mindrift · Oman ·
Creates challenging tasks, simulated developer environments, and tests used to evaluate AI coding agents — designing tasks from intermediate codebase states, writing tests that verify agent solutions, and iterating on them from QA feedback. Core stack: Python/FastAPI, TypeScript/React, Docker, Postgres, Kafka, Redis.
We're building a dataset to evaluate AI coding agents – how well a model handles real‑world developer tasks.
You will create challenging tasks and evaluation criteria within realistic simulated environments:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non‑trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions – writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
Apply? Pass qualification(s)? Join a project? Complete tasks? Get paid.
Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.