Impress employers and recruiters.
Choose from hundreds of resume examples.
Impress employers and recruiters.
Choose from hundreds of resume examples.
Target Resume Now
About the Role
This role is critical for ensuring the quality and trustworthiness of Harvey's human evaluations as the company scales globally. You will set the standard for evaluation data and methodology, ensuring that Product, Engineering, and AI Research can rely on the output for critical decisions.
Responsibilities
- Own the quality bar for Harvey’s human evaluations, defining standards and making outputs decision-grade for EPD.
- Author and maintain evaluation guidelines, instructions, benchmarks, and gold references.
- Serve as the arbiter on ambiguous or disputed judgments.
- Standardize and streamline rubric and evaluation design into repeatable templates and documented methodology.
- Own contract-attorney quality, including onboarding, calibration training, inter-rater reliability, and feedback loops.
- Define and maintain the failure-mode/error taxonomy for eval results.
- Run QA on vendor and contract-attorney deliverables.
- Ensure the quality bar holds across jurisdictions and non-English geographies.
- Establish a standard, documented way to analyze eval results and build operational dashboards.
- Support ALR in a review step to certify evaluations before they scale.
- Partner with ALR and Analytics to align human eval with online signal and expand insights.
Requirements
- 6+ years in product operations, research operations, evaluation/QA operations, or quality program management.
- Proven track record of owning quality inputs (guidelines, instructions, benchmarks, QA procedures) for complex, expert-driven or human-in-the-loop work.
- Experience onboarding, training, and calibrating a distributed pool of expert raters, annotators, or reviewers.
- Experience running the feedback loop that improves rater quality over time.
- Grounding in measurement concepts (calibration, inter-rater reliability, sampling, rubric design) to independently set up and own quality.
- Comfort with interpreting evaluation data, natively or with AI tool support.
- Record of scaling and streamlining quality processes under shipping pressure, with a bias toward documentation and reproducibility.
- Ability to work deeply with domain experts and translate nuanced judgment into repeatable, documented standards.
- Strong cross-functional coordination across Product, Engineering, Research, ALR, and data providers/vendors.
- Clear communicator who can build credibility and trust with stakeholders.
- Bias to action and high ownership.
Skills
- Product Operations
- Research Operations
- Evaluation Operations
- Quality Program Management
- Guideline Authoring
- Instruction Authoring
- Benchmark Creation
- QA Procedures
- Rater Onboarding
- Rater Training
- Rater Calibration
- Inter-rater Reliability
- Rubric Design
- Measurement Concepts
- Evaluation Data Interpretation
- Process Scaling
- Process Streamlining
- Documentation
- Reproducibility
- Cross-functional Coordination
- Communication
- Ownership
Location
- Global
Work Type
- Full-time
Experience Level
- Senior
Salary/Compensations
- $155,400 - $233,200 USD
About the Company
- Harvey is transforming how legal and professional services operate by combining frontier agentic AI, an enterprise-grade platform, and deep domain expertise.
- The company has strong product-market fit and world-class investor support, and is scaling fast.
- Harvey's team moves fast, takes ownership, and is deeply committed to the mission, operating with intensity, staying close to customers, and pushing for excellence.
- Company values include Decisiveness, Simplicity, and Job's Not Finished.
Equal Opportunity
- Harvey is an equal opportunity employer and does not discriminate on the basis of race, gender, sexual orientation, gender identity/expression, national origin, disability, age, genetic information, veteran status, marital status, pregnancy or related condition, or any other basis protected by law.
- We are committed to providing reasonable accommodations to applicants with disabilities, and requests can be made by emailing accommodations@harvey.ai.