AI Evaluation Rubric Auditor
MercorLegit · not rated yetPosted 6 days ago
Pay: $60–85/hr
- Who can apply
- Open only to people in Argentina, Brazil, Canada, Chile, Colombia, United Kingdom, Peru, United States, Uruguay
- Hours
- 15 hrs/week · Contract
- Where
- Remote, work from home
- Job type
- Response evaluation
- Openings
- 11 openings
At a glance: A contract, remote response evaluation job with Mercor. It pays $60–85/hr for 15 hours a week. Only people in 9 listed countries can apply.
Apply link: aitrainerhq.com/go/job/28rpuldi62 (opens in a new tab)
What happens after you click Apply
- You leave AI Trainer HQ and go to Mercor’s own site.
- Upload your résumé on work.mercor.com
- Confirm your work authorization
- Sit the AI interview, which takes about 20 minutes
- Complete any written assessment attached to the listing
- On receiving an offer, sign it, configure payouts and verify your identity
How this pay compares
This job pays more than 44% of the 131 other finance & accounting jobs on the board that state pay; 55% pay more.
- This job
- $73/hrMiddle of $60–85/hr
- Median on Mercor
- $80/hrFrom 340 other paid jobs
- Median in finance & accounting
- $80/hrFrom 131 other paid jobs
Requirements
- Qualifications
- Not specified in the listing
- Languages
- English
- Equipment
- Computer with a reliable internet connection
About this job
The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
\- Was the information available?
\- Does the rubric accurately grade what the prompt asked for?
\- Would a valid answer have been marked wrong?
\- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
\- Experience providing feedback to writers/reviewers
\- Consistent availability
\- Can read a long agent transcript and find where it went wrong.
\- Precise writing, and you check the data instead of trusting the reference answer.
Listing history
- First seen on our board
- (5 days ago)
- Posted by Mercor
- (6 days ago)
- Last checked
- Source
- Mercor
We check every live job against its source at least every 24 hours. How we check jobs