数据科学家，代理人评估与质量
Data Scientist, Agent Evaluations & Quality
我们正在寻找一位在数据科学和AI产品质量领域有丰富经验的候选人，加入一家快速发展的AI生产力初创公司。该公司专注于构建能够处理邮件、日历、浏览器和业务软件任务的自主代理。你将负责从头到尾衡量代理的质量：将模糊的产品行为转化为严谨且可操作的评估系统，直接指导工程和产品决策。
你将负责：
- 架构并维护自动化评估流程，以衡量代理在不同功能和产品界面中的质量。
- 将代理功能转化为明确的成功标准，包括复杂多步骤任务的通过、部分通过和失败定义。
- 构建具有代表性的黄金数据集和回归套件，涵盖常见工作流、边缘情况、模糊请求和对抗场景。
- 定义并跟踪指标，如任务成功率、工具选择准确性、指令遵循度、事实一致性、延迟、成本和可靠性。
- 设计确定性和基于模型的评分系统，校准LLM-as-a-judge系统，并测量评分者一致性、误报和漏报。
- 分析追踪记录、工具调用、模型输出和生产结果，以识别根本原因并建立有用的故障分类。
- 使用严格的离线实验和生产证据来比较模型、提示、工具和能力实现。
- 构建仪表板和发布质量信号，使评估结果对工程、产品和管理层易于理解并可操作。
- 与能力工程师合作，推荐改进措施并验证修复是否在成本、延迟或可靠性方面没有不可接受的退化。
我们希望你具备：
- 5年以上数据科学、机器学习或分析相关职位经验，重点是在生产系统中评估系统、指标框架或质量测量方面。
- 有设计和实施评估框架、评分系统和成功标准的经验，适用于生产环境中的ML或AI系统。
- 精通Python和SQL，能够大规模构建自动化数据管道和生产质量分析代码。
- 扎实的统计和实验设计知识：抽样、方差、不确定性量化、偏差检测、混杂变量和非确定性系统的显著性测试。
- 有真实数据开发经验：标注指南、数据标注、数据清洗和数据验证。
查看英文原文
ABOUT THE ROLE
This role sits at the intersection of applied data science and AI product quality for a small, fast-moving AI productivity startup building autonomous agents that handle email, calendar, browser, and business software tasks. You will own the measurement of agent quality end-to-end: turning ambiguous product behavior into rigorous, actionable evaluation systems that directly guide engineering and product decisions.
WHAT YOU'LL DO
- Architect and maintain automated evaluation pipelines that measure agent quality across capabilities and product surfaces.
- Translate agent capabilities into explicit success criteria, including pass, partial-pass, and failure definitions for complex multi-step tasks.
- Build representative gold datasets and regression suites covering common workflows, edge cases, ambiguous requests, and adversarial scenarios.
- Define and track metrics such as task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability.
- Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and measure grader agreement, false positives, and false negatives.
- Analyze traces, tool calls, model outputs, and production outcomes to identify root causes and build a useful failure taxonomy.
- Compare models, prompts, tools, and capability implementations using rigorous offline experiments and production evidence.
- Build dashboards and release-quality signals that make evaluation results understandable and actionable for engineering, product, and leadership.
- Partner with capability engineers to recommend improvements and verify that fixes raise quality without unacceptable regressions in cost, latency, or reliability.
WHAT WE'RE LOOKING FOR
- 5+ years in data science, machine learning, or analytics roles, with a focus on evaluation systems, metrics frameworks, or quality measurement for production systems.
- Demonstrated experience designing and implementing evaluation frameworks, grading systems, and success criteria for ML or AI systems in production.
- Strong Python and SQL proficiency with the ability to build automated data pipelines and production-quality analysis code at scale.
- Solid statistical and experimental design knowledge: sampling, variance, uncertainty quantification, bias detection, confounding variables, and significance testing for non-deterministic systems.
- Experience with ground-truth data development: labeling guideline design, annotation quality control, ambiguity resolution, and dataset maintenance.
- Working knowledge of LLM behavior, tool use, retrieval systems, multi-step execution, and practical failure modes of language model systems.
- Ability to connect quantitative patterns to individual system traces and identify failure origins across model, prompt, context, tools, data, and application logic.
- Experience communicating evaluation results, methodology, uncertainty, and trade-offs to both technical and non-technical stakeholders.
- Comfort operating with high ownership in ambiguous, fast-moving environments, independently turning open-ended quality questions into evaluation systems.
- Experience with LLM-as-a-judge systems, agentic or multi-step task evaluation, or benchmarking platforms for AI systems is a strong plus.
LOCATION
On-site in Palo Alto, California, United States. Visa sponsorship is not available for this role.