Impress employers and recruiters.
Choose from hundreds of resume examples.

Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

About the Role

Scale works with leading AI labs to provide high-quality data and accelerate progress in GenAI research. This role is on the evaluation pod within the GenAI Research Organization, focusing on building benchmarks and diagnosing model failure modes in text and multimodal modalities.

Responsibilities

• Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents, focusing on RCA.
• Design and build benchmarks and evaluation methods that measure LLM capabilities in text and multimodal modalities.
• Apply post-training expertise (SFT, RLHF, reward modeling) to connect observed failures to data and training interventions.
• Publish research findings in top-tier AI conferences.

Requirements

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning.
• Experience with LLM evaluation or benchmark development.
• Excellent written and verbal communication skills.
• Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals.
• Previous experience in a customer-facing role.

Skills

• LLM post-training (SFT, RLHF, reward modeling)
• Evaluation
• Text modalities
• Multimodal modalities
• Deep learning
• Reinforcement learning
• Large-scale model fine-tuning
• RLHF
• Preference modeling
• Instruction tuning
• LLM evaluation
• Benchmark development

Location

• San Francisco
• New York
• Seattle

Work Type

• Full-time

Experience Level

• Master's degree
• Ph.D.

Education Level

• Master's degree
• Ph.D.

Salary/Compensations

• $180,600—$225,750 USD

Benefits

• Comprehensive health, dental and vision coverage
• Retirement benefits
• Learning and development stipend
• Generous PTO
• Commuter stipend

About the Company

• Scale's mission is to develop reliable AI systems for the world's most important decisions.
• Products provide high-quality data and full-stack technologies powering leading models.
• Helps enterprises and governments build, deploy, and oversee AI applications.
• Works closely with industry leaders like Meta, Ernst & Young, Mayo Clinic, Time Inc., the Government of Qatar, and U.S. government agencies including the Army and Air Force.
• Expanding team to accelerate AI application development.

Equal Opportunity

• Scale is an inclusive and equal opportunity workplace.
• Committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability status, gender identity or Veteran status.
• Committed to working with and providing reasonable accommodations to applicants with physical and mental disabilities.
• Complies with the United States Department of Labor's Pay Transparency provision.