Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

Tailor your resume to this Senior Technical Product Manager – Agent Evaluation & Observability role.

Rezi rewrites your resume against American Express's job description. Free.

Resume score gauge reading 58 out of 100

Don't guess if your resume is good enough.

See how it scores against the Senior Technical Product Manager – Agent Evaluation & Observability posting at American Express — free, in seconds.

About the Role

The Amex Digital Labs team drives innovation at American Express by originating, incubating, and managing products and platforms essential to our customers’ digital lives. We are looking for a Senior Technical Product Manager to lead the roadmap for tools and workflows that help teams test and improve agent-enabled customer experiences. You will help teams measure how well agents complete customer tasks, understand failures, and use the findings to improve their experiences.

Responsibilities

Own the roadmap for shared agent evaluation and observability capabilities, from user discovery and requirements through launch, adoption and improvement.
Partner with domain owners to define representative journeys, prompt sets, expected tool behavior and outputs, scoring rubrics and acceptance criteria.
Define requirements for automated evaluation, human review, and tools that help teams diagnose failures.
Define product requirements for traces, events, diagnostic views and review tools that connect customer intent, tool or API calls and observed outcomes where data access permits.
Work with engineering, analytics and existing enterprise evaluation and observability teams to extend shared capabilities, reuse instrumentation and standardize evidence.
Identify where limited access to partner data prevents a complete assessment.
Partner with Identity and domain owners to test policy-relevant behavior, including denied actions, revoked access and required approvals.
Define meaningful measures such as task completion, correct tool use, response quality, failure and recovery rates, latency and cost.
Define how testing and monitoring work together across controlled test environments, end-to-end partner integrations and production.
Measure adoption and usefulness of evaluation workflows, including review effort, coverage and time to identify problems.
Translate measurement gaps into a prioritized backlog, clear requirements, and actionable acceptance criteria.
Recommend when to extend existing tools, integrate a solution or build a missing capability.
Build privacy and data governance into evaluation and telemetry requirements, including permitted uses of partner-provided data, appropriate access, minimization and retention.
Communicate uncertainty, release risks and tradeoffs clearly.

Requirements

6-8+ years of experience in product management, product development, software engineering, technical platforms, analytics, quality tooling, or AI products.
Experience owning evaluation, testing, observability, analytics or related technical products through launch, adoption and iteration, with measurable user or business outcomes.
Practical understanding of AI or agent-enabled systems, including nondeterministic behavior, tool or API interactions, and the need for both automated tests and human judgment.
Ability to define metrics, interpret evidence and identify sampling, coverage, or measurement limitations.
Familiarity with traces, logs, APIs and technical troubleshooting.
Strong product discovery, roadmap and prioritization skills, including translating ambiguous quality problems into usable tools and repeatable workflows.
Cross-functional leadership and clear communication, with the ability to balance customer outcomes, engineering constraints and privacy or risk requirements.
Experience with LLM or agent evaluation, scenario design, human review, model-assisted scoring or production failure analysis.
Familiarity with MCP, function calling, telemetry standards or evaluation frameworks; ability to use code or queries to investigate evidence.
Experience building shared capabilities adopted by multiple teams or delivering products in financial services or another regulated industry.
Employment eligibility to work with American Express in the United States is required as the company will not pursue visa sponsorship for these positions.

Skills

Product management
Product development
Software engineering
Technical platforms
Analytics
Quality tooling
AI products
AI or agent-enabled systems
Nondeterministic behavior
Tool or API interactions
Automated tests
Human judgment
Metrics definition
Evidence interpretation
Sampling limitations
Coverage limitations
Measurement limitations
Traces
Logs
APIs
Technical troubleshooting
Product discovery
Roadmap development
Prioritization
Cross-functional leadership
Clear communication
Customer outcomes
Engineering constraints
Privacy requirements
Risk requirements
LLM evaluation
Agent evaluation
Scenario design
Human review
Model-assisted scoring
Production failure analysis
MCP
Function calling
Telemetry standards
Evaluation frameworks
Code usage
Query usage

Location

United States

Work Type

Hybrid
Onsite
Virtual

Experience Level

Senior
6-8+ years

Education Level

Bachelor’s degree or equivalent experience

Benefits

Competitive base salaries
Bonus incentives
6% Company Match on retirement savings plan
Free financial coaching and financial well-being support
Comprehensive medical, dental, vision, life insurance, and disability benefits
Flexible working model with hybrid, onsite or virtual arrangements depending on role and business need
20+ weeks paid parental leave for all parents, regardless of gender, offered for pregnancy, adoption or surrogacy
Free access to global on-site wellness centers staffed with nurses and doctors (depending on location)
Free and confidential counseling support through our Healthy Minds program
Career development and training opportunities

About the Company

The core of our company is not just the products we offer, but the personal connections our customers have with our brand.
At American Express, our culture is built on a 175-year history of innovation, shared values, and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues.
From delivering differentiated products to providing world-class customer service, we operate with a strong risk mindset, ensuring we continue to uphold our brand promise of trust, security, and service.
As part of Team Amex, you’ll experience our powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career.
Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express.

Equal Opportunity

American Express is an equal opportunity employer and makes employment decisions without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability status, age, or any other status protected by law.
American Express will consider for employment all qualified applicants, including those with arrest or conviction records, in accordance with the requirements of applicable state and local laws, including the California Fair Chance Act, the Los Angeles County Fair Chance Ordinance for Employers, and the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance.
For positions covered by federal and/or state banking regulations, American Express will comply with such regulations as it relates to the consideration of applicants with criminal convictions.
US Job Seekers - Click to view the “Know Your Rights poster.