Impress employers and recruiters.
Choose from hundreds of resume examples.

Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

About the Role

Contra Labs is seeking a dedicated individual to manage research projects focused on how creatives use AI, improve creative tools, and protect originality. This role involves bridging the gap between clients, creative experts, and internal engineering teams to ensure rigorous and useful results for AI labs and product companies.

Responsibilities

• Define study methodologies, including annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs.
• Manage project delivery, including scope, evaluator staffing, quality routing, timelines, and client relationships from kickoff to final readout.
• Ensure data quality and inter-rater reliability, making real-time adjustments when necessary.
• Conduct qualitative and quantitative analysis, translating findings into actionable recommendations, reports, and workshops.
• Develop and refine systems for repeatable work, collaborating with engineering on tooling to improve process efficiency and reliability.
• Create case studies and benchmark publications to enhance Contra Labs' market presence.

Requirements

• 3+ years of experience in designing studies, running evaluations, analyzing data, and presenting findings.
• Experience with data labeling, annotation, human-in-the-loop systems, or AI/ML evaluation workflows.
• Proven track record of managing multiple concurrent projects, including annotation, evaluation, or QA work.
• A strong perspective on creative quality and what constitutes 'good' work.
• Ability to design studies, run them, analyze data, and translate insights into actionable directions.
• Understanding of creative quality and the ability to articulate it.
• Skill in building frameworks to faithfully capture creative signal.
• Focus on human judgment in creative work and designing systems around it.
• Ability to transform messy work into clear, predictable, and measurable processes.
• High ownership, low ego, and a bias for action, with a focus on shipping and iterating.
• Based in San Francisco, California.

Skills

• Study design
• Evaluation design
• Data analysis
• Qualitative analysis
• Quantitative analysis
• Client relationship management
• Project management
• Data labeling
• Annotation
• Human-in-the-loop systems
• AI/ML evaluation
• Creative quality assessment
• Process improvement
• Statistical methods
• Inter-rater reliability
• Sampling
• Regression
• Significance testing
• RLHF
• LLM evaluation
• Benchmark design
• Ethnography
• UX research
• Design study methods

Location

• San Francisco, California

Work Type

• Full-time

Experience Level

• 3+ years experience

Education Level

• Master's or PhD in HCI, cognitive science, psychology, design research, or related field (preferred)
• Equivalent industry experience

Salary/Compensations

• $150,000 to $200,000 USD plus equity

Benefits

• Medical
• Dental
• Vision
• 401k matching
• Company laptop

About the Company

• Contra Labs is a human-centered research lab studying creative AI usage and tool improvement.
• They connect global creative talent with AI labs and product companies.
• Their work powers benchmarking, evaluation, and post-training for leading AI models and applications.
• Built on Contra, a professional network for independent creatives, they use human judgment to train and evaluate AI tools.
• They are scaling across client engagements including post-training dataset creation, benchmarking programs, and evaluation partnerships.