技术员工，代理
Member of Technical Staff, Agents
职位描述
这是一个动手实践的工程岗位，专注于构建和改进能够从头到尾处理实际高管工作的自主AI代理，包括日程安排、电子邮件、电话和关系管理。你将位于一个小型、高级团队的核心位置，负责整个流程：交付代理、衡量其质量，并使改进成为可重复的科学方法。这项工作很重要，因为代理在生产环境中运行，真实用户依赖它们而无需检查。
你将负责的工作
- 构建能够完整完成委托高管任务的代理，无需人工跟进。
- 设计并维护评估框架和指标，让团队有信心地知道代理做对的频率。
- 扩展代理可以处理的任务集，使用高管已有的工具，在发布前证明其准备就绪。
- 快速将生产故障转化为已验证的永久性解决方案。
- 使用可观测性和评估工具进行大规模检测，发现已知的失败模式以及团队尚未看到的长尾问题。
- 构建自我恢复机制，使代理在工具或外部服务出现故障时仍能保持任务执行。
- 使代理改进可测量：改变一个因素，运行现实的端到端模拟，保留有效的方法。
我们寻找的人选
- 5年以上数据科学、机器学习或分析经验，专注于生产AI的评估系统和质量度量。
- 有设计评估方法的经验：成功标准、数据集构建、指标选择，以及区分真实信号与噪声的能力。
- 具备生产级Python和SQL技能；在涉及Django、React和TypeScript时应感到舒适。
- 强大的统计和实验设计能力，包括抽样、方差、偏差检测和非确定性系统的显著性测试。
- 有LLM作为评判者系统、基于模型的评分器和评分器校准的经验。
- 能够跨提示、追踪、模型输出、代码、数据库和API进行分析调试，找到故障的真正原因。
- 有在生产环境中构建和运行具有多步骤执行和工具使用的代理系统的经验。
- 具备真实数据开发经验，包括标注指南、注释质量控制和数据集维护。
- 来自严谨学位项目的强大计算机科学或工程基础。
- 有从头到尾负责某项工作的记录，将其交付给真实用户，并展示所取得的成果。
查看英文原文
ABOUT THE ROLE
Join a small, senior team building autonomous AI agents for executive productivity. You will develop new agent capabilities and create reliable ways to evaluate them, helping ensure that important real-world tasks are completed accurately before new capabilities reach users.
WHAT YOU'LL DO
- Build agents that complete delegated work end to end without requiring the user to check each step.
- Develop trustworthy evaluations that measure how accurately agents perform different tasks.
- Ship new agent capabilities in the tools people already use, supported by evidence that they are ready.
- Investigate production failures and deliver durable, validated fixes.
- Build large-scale failure detection to identify known patterns and unexpected issues.
- Improve agent recovery when a tool or website fails.
- Create repeatable workflows so engineers can measure whether agent changes improve performance.
WHAT WE'RE LOOKING FOR
- Approximately 2 to 11 years of relevant experience, including a few years in one role with demonstrated progression.
- Experience building or evaluating production AI or machine learning systems, especially agents or LLM-powered products.
- Strong Python skills and familiarity with Django, React, and TypeScript.
- Experience with evaluation and monitoring tools such as Braintrust or Raindrop is relevant.
- Comfort working across implementation, evaluation, and iteration in a fast-moving environment.
COMPENSATION & BENEFITS
Salary range: $200,000 to $300,000 USD annually. Visa sponsorship is not available.
LOCATION
Hybrid, with four days per week in the office in Palo Alto, California, United States.