Sr Applied Scientist, AI Agent Evaluation, AWS Applied AI Solution - Core Services
Amazon
DESCRIPTION
As part of the AWS Solutions organization, we have a vision to provide business applications, leveraging Amazon’s unique experience and expertise, that are used by millions of companies worldwide to manage day-to-day operations. We will accomplish this by accelerating our customers’ businesses through delivery of intuitive and differentiated technology solutions that solve enduring business challenges. We blend vision with curiosity and Amazon’s real-world experience to build opinionated, turnkey solutions. Where customers prefer to buy over build, we become their trusted partner with solutions that are no-brainers to buy and easy to use.
Lead our pioneering AI initiative at AWS and define the future of AI agents. Shape how businesses leverage artificial intelligence by creating foundational building blocks used across AWS business applications.
The Core Services team within AWS Applied AI Solutions is creating the foundations that will power the next generation of AI agents from small business to enterprise scale. As a scientific leader, you'll drive our new agentic AI building blocks initiative, pioneering the development of reusable AI components that accelerate and standardize AI product delivery across AWS business applications such as contact center, supply chain, healthcare and life sciences. You'll own one critical capability area such as agent identity and governance, agent collaboration & orchestration, agent evaluation, or knowledge management, and have the privilege to define what we build and how we build it.
This role combines the excitement of a startup environment with the scale of AWS. You'll research state-of-the-art open source and internal tools, tackle highly ambiguous problems, and lead scientific innovation building a V1 product from the ground up. You won't just be implementing someone else's vision — you'll chart the course, define the roadmap, and create novel scientific solutions that eliminate non-differentiating work while ensuring enterprise-grade quality and consistency. If you thrive on ownership, are passionate about AI research and development, and want to fundamentally influence how AWS builds AI products, this role offers an extraordinary opportunity to make your mark.
Key job responsibilities
- Design and implement evaluation frameworks for AI agents, including benchmarking tools, annotation systems for RLHF, and standardized patterns for memory orchestration and retrieval that ensure consistent performance across diverse use cases
- Develop deep expertise in a strategic research area within AI agent systems, becoming the organization's scientific expert in areas such as agent design and evaluation.
- Identify and devise new research solutions for ill-defined customer or business problems that require novel methodologies and paradigms to be invented at the product level
- Articulate key potential scientific challenges of ongoing or future customer needs or business problems in AI agents, and present interventions to address them
- Lead the design, implementation, and successful delivery of solutions for scientifically-complex problems, writing "critical-path" code
- Write clear, useful narratives and documentation describing inventions, solutions, and design choices that enable others to understand and reproduce your work
- Demonstrate detailed knowledge of your team's solutions and systems while proactively driving utilization and improvement upon the state of the art
- Independently assess alternative AI technologies and choose the right approaches to be integrated into your systems
- Apply and drive your team to adopt best practices in scientific research, code development, and technical documentation
- Influence across multiple teams to build consensus on scientific approaches and architectural decisions
- Actively mentor and develop other scientists and engineers, elevating the technical capabilities of the organization
A day in the life
Your morning starts with a team standup, followed by collaborative sessions with software engineers and product managers to understand requirements and internal customer needs. You'll dedicate focused time to researching the latest scientific advances in AI agents and designing novel approaches for your capability area. Throughout the day, you'll work with engineers and scientists within your team to implement solutions, applying scientific rigor to validate approaches. You might lead a scientific design review gathering diverse perspectives to strengthen your methodologies. Your day could include conducting experiments, analyzing results, designing new evaluation methodologies, meeting with internal customers to understand their AI agent needs, or presenting your scientific vision to leadership stakeholders.
About the team
Applied AI Solutions Core Services is a comprehensive suite of core computing services and solutions provided by Amazon Web Services (AWS) to support the foundational infrastructure and operations needs of customers. It encompasses a wide range of essential services and capabilities across AWS. The Core Services team focuses on continuously improving the security, availability, and cost-effectiveness of these critical enterprise-wide services for AWS customers.
Why AWS
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses.
Diverse Experiences
Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying.
Work/Life Balance
We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud.
Inclusive Team Culture
Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness.
Mentorship and Career Growth
We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.