Own Search Quality & Evals Strategy:

• Define the evaluation framework for Teams Search, covering both classical IR metrics (NDCG, precision@k) and LLM-era metrics (relevance, groundedness, faithfulness, helpfulness).
• Lead the design of human evaluation pipelines, including task design, rater guidelines, inter-rater reliability, and feedback loops into model/ranking improvements.
• Build and maintain an offline eval harness: query sets, golden datasets, annotation workflows, and regression benchmarks used to gate every major search update.
• Partner with Applied Science and engineering to translate eval signals into roadmap priorities and quality SLAs.

Drive AI-Powered Search Features:

• Define product requirements for semantic search, hybrid retrieval (BM25 + dense embeddings), re-ranking, and Copilot-integrated search experiences in Teams.
• Own the PM relationship with the relevance science team, translating user problems into model requirements, evaluation criteria, and launch guardrails.
• Work with the Copilot/AI team to integrate retrieval-augmented generation (RAG) patterns into Teams Search, ensuring quality, safety, and latency bars are met before ship.

Define Metrics and Success:

• Own the success metrics hierarchy for Teams Search: engagement, satisfaction (CSAT/NPS), task completion, zero-result rate, abandonment, and AI answer quality.
• Instrument A/B experiments end-to-end: hypothesis, metric selection, power analysis, readout, and decision framework.
• Build a culture of evidence-based decision making; push back on intuition-only decisions with data.

Stakeholder Leadership:

• Collaborate with partner teams across M365 Search, Copilot, to align on shared infrastructure, signals, and quality bars.
• Represent Teams Search in cross-org quality reviews, red-lines, and AI safety reviews.
• Mentor junior PMs on the team; model strong PM craft: crisp specs, rigorous eval thinking, and structured decision-making.

Required/Minimum Qualifications:

• Bachelor's Degree AND 8+ years experience in product/service/program management or software development.
  • OR equivalent experience.
• 8+ years of product management experience, with at least 3 years in search, information retrieval, or a related ML/AI-powered product area.
• Deep, hands-on experience with evaluation frameworks for search or AI systems. You have personally designed eval sets, defined metrics, and used eval results to drive product decisions.
• Solid understanding of IR fundamentals: ranking, relevance, query understanding, recall/precision trade-offs.
• Experience with LLM-era eval challenges: hallucination detection, groundedness, response quality, and human eval design for generative outputs.
• Demonstrated ability to work cross-functionally with applied scientists, ML engineers, and data teams. You are comfortable reading model cards, experiment results, and offline eval reports.

Other Requirements: Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:

• Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Preferred Qualifications:

• Experience shipping search or retrieval features in a large-scale consumer or enterprise product (100M+ users).
• Familiarity with RAG pipelines, vector databases, hybrid retrieval architectures, or semantic search systems.
• Experience with annotation platforms (Scale AI, Appen, internal tools) and managing human evaluation programs.
• Knowledge of Microsoft 365 data model, Graph API, or Teams platform is a strong plus.
• Prior experience in a Hyderabad/Bangalore engineering or PM role, with ability to influence across time zones.
• MBA or Master's in a technical field.
• High impact: Search is used by every Teams user, every day. Quality improvements are felt immediately at 300M+ scale.
• AI-first: You will be at the center of Microsoft's Copilot transformation, defining how AI search quality is measured, shipped, and trusted.
• IDC ownership: This is not a satellite role. IDC Bangalore has full feature ownership within Teams Search, and you will lead strategy, not just execute.
• Growth: Evals discipline is one of the fastest-growing skill sets in the industry. This role makes you a domain expert at one of the world's leading AI companies.

#Evals

#Teams

#AI

#SEARCH

#PM

#CAPIDC

This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.