代理质量/评估工程师 1754
Agent Quality / Evals Engineer 1754
这是一个远程职位。
从头开始负责评估工具链和质量门禁。此职位取代了旧的后期“评估专家”模式，为可衡量的代理质量设立专职负责人。主要职责
· 构建并维护MVP评估工具链：黄金任务、异常任务、评分卡指标和回归测试包。
- 将评估集成到CI中，使质量退化导致构建和发布失败。
- 与产品和技术负责人定义并维护发布门禁阈值。
- 在不超出MVP范围的前提下，为后续对抗性测试和漂移测试扩展奠定基础。
要求
必须具备的资格
· 有评估ML、LLM或非确定性系统经验。
- 强大的测试和基准设计能力。
- 能够处理噪声指标、阈值和概率行为。
- 良好的脚本编写和自动化技能。
AI优先期望
· 使用AI生成候选评估案例和故障假设，但不会将生成的测试与验证的质量混淆。
- 将AI质量视为操作系统，而非QA的后续考虑。
前90天成功的表现 · 第一个参考代理拥有已发布的评分卡和受控评估路径。• 黄金和异常测试能自动运行。• 团队能够用可衡量的术语解释“足够好以发布”的含义。
最初发布于Himalayas
查看英文原文
This is a remote position.
Owns the eval harness and quality gate from the beginning. This role replaces the old late-stage “Evals Specialist” model with a standing owner for measurable agent quality. Key Responsibilities
· Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.
- Wire evals into CI so quality regressions fail builds and releases.
- Define and maintain release-gate thresholds with Product and the Tech Lead.
- Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.
Requirements
Must-Have Qualifications
· Experience evaluating ML, LLM, or non-deterministic systems.
- Strong test and benchmark design capability.
- Comfort working with noisy metrics, thresholds, and probabilistic behavior.
- Good scripting and automation skills.
AI-First Expectations
· Uses AI to generate candidate eval cases and failure hypotheses, but never confuses generated tests with validated quality.
- Approaches AI quality as an operating system, not a QA afterthought.
What Success Looks Like in the First 90 Days · The first reference agent has a published scorecard and gated eval path. • Golden and exception tests run automatically. • The team can explain what “good enough to ship” means in measurable terms.
Originally posted on Himalayas