Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

About the Role

We build and operate the offline evaluation platform that gates Microsoft Copilot’s quality. Teams across Copilot depend on us to run their scenarios against the product, score the responses, and produce the scorecards that decide what ships. As Copilot becomes a fleet of autonomous agents, the way we measure quality has to be reinvented, and you will be one of the engineers building it. As a Senior Software Engineer on the Copilot evaluation team, you will be a key engineer on a platform designed to be AI-first and agentic-first, where autonomous agents are not just measured, but help operate and extend the system itself. You will design and deliver major components of the platform and partner closely with senior and principal engineers on the systems that keep evaluation trustworthy across a deep chain of dependencies.

Responsibilities

Partner with engineers and stakeholders across the Copilot organization to turn quality questions into rigorous, reproducible evaluations, building the scenarios, metrics, and scorecards that gate what ships.
Design and build major components of an AI-first, agentic-first evaluation platform where autonomous agents are first-class operators, delivering services, pipelines, and tooling that agents can run, extend, and reason about, with the observability and guardrails that make agent-driven operation trustworthy.
Contribute to the platform’s hardest challenge, delivering trustworthy results across a deep chain of dependencies driven far outside their normal operating envelope, implementing graceful degradation, intelligent retry, and dependency-aware gating so reliability holds under sustained, growing load.
Build the capabilities that make agentic Copilot experiences measurable end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice, including the simulation, scraping, and scoring that capture agent behavior.
Write and review extensible, maintainable code and mentor less-experienced engineers, contributing to modernization of the evaluation runtime and platform so it scales with fast-growing demand while improving cost and latency.
Act as a Designated Responsible Individual (DRI) for a system that gates releases, helping instrument the reliability, throughput, and dependency health of the platform and working on-call so the results teams depend on stay correct and timely.
Stay current with agentic systems, large-scale AI evaluation, and autonomous operations, adopting new tools and patterns and sharing what works to raise the bar for how the team measures quality.

Requirements

Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python OR equivalent experience.
Ability to meet Microsoft, customer and/or government security screening requirements.
Must pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Skills

Large-scale distributed systems
Reliability engineering
Building dependable services across interacting dependencies under heavy or unusual load
Building AI-first or agentic systems
Systems, APIs, or tooling intended to be operated by autonomous agents
Observability and guardrails for agent-driven operation
AI/ML evaluation, measurement, or benchmarking systems
LLM-based grading
Metric design
Large-scale data and quality pipelines

Location

U.S.

Work Type

Full-time

Experience Level

Senior
4+ years technical engineering experience
6+ years technical engineering experience
8+ years technical engineering experience

Education Level

Bachelor's Degree in Computer Science or related technical field
Master's Degree in Computer Science or related technical field

Salary/Compensations

USD $119,800 - $234,700 per year (U.S. typical base pay range)
USD $160,200 - $261,000 per year (San Francisco Bay area and New York City metropolitan area)

Benefits

Eligible for benefits and other compensation

About the Company

Microsoft’s mission is to empower every person and every organization on the planet to achieve more.
As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals.
Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Equal Opportunity

Microsoft is an equal opportunity employer.
All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances.
If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.