Research Scientist / Engineer, Foundation Model Evaluation
Apple
Cupertino, CA, USA
USD 181,100-318,400 / year + Equity
Posted on Mar 26, 2026
We are looking for a Research Scientist or Engineer to join our Foundation Model Evaluation team. In this role, you will design and build evaluation methodology that measures what matters - how well our models perform at the frontier of key capabilities, and how well they serve real users across Apple products on billions of active devices. You will turn evaluation insights into signals that make models better.
This is a hands-on role focused on the models that power Apple products used daily by over a billion people. You will design evaluation systems where the outcome is not just a score, but an actionable signal - one that drives model improvement and predicts real user experience. Working alongside model training and product teams, you will close the loop between evaluation and improvement. Our work spans three areas: • Frontier capability assessment: benchmarking against the state of the art in reasoning, code, knowledge, and agentic workflows • Product-aligned evaluation: measuring model quality in ways that reflect real user experience • Evaluation-to-training integration: feeding actionable insights back into the model development cycle You may focus on one area or work across multiple, depending on your background and interests.
- Benchmark Design & Development: Design and implement evaluation benchmarks,
- metrics, and test suites that rigorously measure model capabilities across reasoning,
- knowledge, code, and agentic workflows.
- Product-Aligned Evaluation: Develop evaluation methods that capture how models
- behave in real product settings, and validate that evaluation metrics predict user-
- perceived quality and product outcomes.
- Evaluation Methodology Research & Tooling: Research and apply state-of-the-art
- evaluation techniques — including scoring frameworks, model-based judging, and
- contamination-resistant benchmark design. Build reusable tools, scorer libraries, and
- analysis frameworks that scale across the team's benchmark portfolio.
- Experimental Analysis: Design and execute rigorous experiments comparing model
- capabilities, engage with third-party vendors on benchmarking, and perform detailed gap
- analysis to guide model development priorities.
- Cross-Team Collaboration: Work closely with model training, training data, and product
- teams to ensure evaluation insights inform training strategies, data decisions, and product
- quality improvements.
- 3+ years of experience in AI model evaluation, NLP, or a related area (e.g., natural
- language generation, information retrieval, or conversational AI)
- Strong fundamentals in machine learning, natural language processing, and statistical
- analysis
- Proficiency in Python and experience with ML frameworks (PyTorch, JAX, or
- equivalent)
- Demonstrated ability to translate research insights into practical implementations
- Strong experimental design skills: ability to design rigorous comparisons and draw valid
- conclusions from results
- Clear technical communication: ability to distill evaluation results into actionable
- recommendations for cross-functional partners
- MS or PhD in Computer Science, Machine Learning, Natural Language Processing or a related
- technical field. Equivalent practical experience will be considered.
- PhD in Computer Science, Machine Learning, NLP, or a related field
- Direct experience evaluating large language models, e.g. benchmark design, model-based
- judging
- Track record of collaborating with model training and data teams to turn evaluation
- findings into training improvements
- Experience building reusable evaluation tooling or analysis frameworks adopted across
- teams
- Familiarity with human evaluation methodology and experience partnering with
- annotation teams or vendors to assess model quality
Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant.