Impress employers and recruiters.
Choose from hundreds of resume examples.
Impress employers and recruiters.
Choose from hundreds of resume examples.
About the Role
We build and operate the offline evaluation platform that gates Microsoft Copilot’s quality. Teams across Copilot depend on us to run their scenarios against the product, score the responses, and produce the scorecards that decide what ships. As Copilot becomes a fleet of autonomous agents, the way we measure quality has to be reinvented, and you will lead that reinvention. As a Principal Software Engineer, you will own the technical vision for a platform built AI-first and agentic-first, taking on our hardest problem: delivering trustworthy results across a deep chain of dependencies. You will architect how we evaluate agentic Copilot experiences and set the standard for how the whole product defines and measures quality. This opportunity will allow you to shape the evaluation strategy for one of the world’s most visible AI products, work at the frontier of large-scale agentic systems and reliability engineering, and grow your influence as a technical leader.
Responsibilities
- Sets the technical direction for the Copilot offline evaluation platform and partners with teams across the Copilot organization to turn ambiguous quality questions into rigorous, reproducible evaluations.
- Leads the architecture of an AI-first, agentic-first evaluation platform in which autonomous agents are first-class operators.
- Owns the platform’s hardest challenge: delivering trustworthy results across a deep chain of dependencies driven far outside their normal operating envelope.
- Defines how Copilot’s agentic experiences are evaluated end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice.
- Leads by example and mentors engineers across teams to produce extensible, maintainable systems, driving modernization of the evaluation runtime and platform.
- Stays at the frontier of agentic systems, large-scale AI evaluation, and autonomous operations, bringing in new trends and patterns and sharing that knowledge.
Requirements
- Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python OR equivalent experience.
- Ability to meet Microsoft, customer and/or government security screening requirements.
- Must pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Skills
- Large-scale distributed systems
- Reliability engineering
- AI-first or agentic systems
- Designing systems, APIs, and tooling intended to be operated by autonomous agents
- Observability, guardrails, and safety mechanisms
- AI/ML evaluation, measurement, or benchmarking systems
- LLM-based grading
- Metric design
- Large-scale data and quality pipelines
Location
- U.S.
Work Type
- Full-time
Experience Level
- Principal
- 6+ years technical engineering experience
- 8+ years technical engineering experience
- 12+ years technical engineering experience
Education Level
- Bachelor's Degree in Computer Science or related technical field
- Master's Degree in Computer Science or related technical field
Salary/Compensations
- USD $142,800 - $274,800 per year (U.S.)
- USD $188,000 - $304,200 per year (San Francisco Bay area and New York City metropolitan area)
Benefits
- Certain roles may be eligible for benefits and other compensation.
About the Company
- Microsoft’s mission is to empower every person and every organization on the planet to achieve more.
- As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals.
- Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Equal Opportunity
- Microsoft is an equal opportunity employer.
- All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances.
- If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.