Description

As part of the AWS Applied AI Solutions organization, we have a vision to provide business applications, leveraging Amazon's unique experience and expertise, that are used by millions of companies worldwide to manage day-to-day operations. We accomplish this by accelerating our customers' businesses through delivery of intuitive and differentiated technology solutions that solve enduring business challenges. We blend vision with curiosity and Amazon's real-world experience to build opinionated, turnkey solutions.

We are looking for a Senior Product Manager, Technical to define and drive the product vision for AI Agent Evaluations within our Core Services AI Foundations team. You will own the end-to-end evaluation framework that enables application development teams to measure, benchmark, and continuously improve the quality, safety, and reliability of their AI-powered agents. This includes defining the product strategy for evaluation tooling, quality scoring methodologies, regression testing frameworks, and human-in-the-loop review workflows that give builders confidence their agents perform as intended before and after deployment. You will work backward from the needs of development teams in Applied AI Solutions and AWS building agentic AI applications and define how they assess agent behavior across dimensions including correctness, safety, groundedness, and customer satisfaction. You will partner closely with engineering and applied science teams to translate complex evaluation methodologies into intuitive, self-service products that scale across diverse use cases and agent architectures.

The Core Services AI Foundations team within AWS Applied AI Solutions builds the foundational platform that enables application development teams to ship production-grade AI agents and applications with confidence. We provide the shared infrastructure, tooling, and guardrails that handle the hardest cross-cutting concerns in agentic AI: evaluation, identity and access management, observability and analytics, data and knowledge management, foundational agents, and user experience.

Key job responsibilities
Own the product vision, strategy, and roadmap for AI agent evaluation capabilities, including automated evaluation pipelines, human review workflows, and quality benchmarking tools

Write detailed product requirements that articulate evaluation framework APIs, scoring algorithms, and integration patterns

Partner with applied scientists and engineers to translate research into production-grade product features

Drive customer discovery to generalize evaluation requirements from diverse agent architectures, domains, and deployment patterns

Define success metrics and instrumentation strategies that make evaluation product adoption, coverage, and impact visible to stakeholders

Own go-to-market strategy, including positioning evaluation tooling relative to existing AWS services and third-party alternatives

Drive alignment across dependent teams (identity, observability, data platform) to ensure evaluation signals integrate seamlessly with the broader Core Services AI Foundations platform

A day in the life
You split your time between customer discovery, technical design, and cross-team alignment. On any given day you might be meeting with an application development team to understand how they assess agent quality, writing the narrative for a new evaluation capability, reviewing a technical design document with engineers, probing assumptions about scale, and defining acceptance criteria. You operate independently, make trade-offs across competing priorities, and own outcomes from requirements through launch and post-launch performance.

About the team
ABOUT AWS:
Diverse Experiences
Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying.

Why AWS
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses.

Work/Life Balance
We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud.

Inclusive Team Culture
Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity and AmazeCon conferences, inspire us to never stop embracing our uniqueness.

Mentorship and Career Growth
We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.