SWE-Bench Task Auditor
MercorLegit · not rated yetPosted on 28 Aug 2026
Pay: $70–90/hr
- Who can apply
- Open only to people in United States
- Hours
- 40 hrs/week · Contract
- Where
- Remote, work from home
- Job type
- Expert (SME) review
- Openings
- 3 openings
At a glance: A contract, remote expert (SME) review job with Mercor. It pays $70–90/hr for 40 hours a week. Only people in United States can apply.
What happens after you click Apply
- You leave AI Trainer HQ and go to Mercor’s own site.
- Upload your résumé on work.mercor.com
- Confirm your work authorization
- Sit the AI interview, which takes about 20 minutes
- Complete any written assessment attached to the listing
- On receiving an offer, sign it, configure payouts and verify your identity
How this pay compares
This job pays more than 56% of the 120 other finance & accounting jobs on the board that state pay; 38% pay more.
- This job
- $80/hrMiddle of $70–90/hr
- Median on Mercor
- $80/hrFrom 295 other paid jobs
- Median in finance & accounting
- $65/hrFrom 120 other paid jobs
Requirements
- Qualifications
- Not specified in the listing
- Languages
- English
- Equipment
- Computer with a reliable internet connection
About this job
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
Basic Qualifications
• 3+ years professional software engineering
• Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
• Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
• Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
• Familiarity with SWE-Bench (Verified) or similar repository benchmarks
• Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
• Prior code-review or task-grading experience
Listing history
- First seen on our board
- (9 days ago)
- Posted by Mercor
- Last checked
- Source
- Mercor
We check every live job against its source at least every 24 hours. How we check jobs
