Senior AI Backend Engineer - Agent Evaluation & Quality
关于角色 我们运行生产多代理系统，为大量用户处理实际工作。随着这些系统的增长，我们的最大限制是信心：我们需要知道代理商在出货之前表现得多好，捕捉退缩率，并在发布时保持质量稳定。这个角色拥有这一点。你将建立评估系统 - - 在我们的代理人身后 - 评审员，测试带和模拟器，告诉我们一个代理人在哪里工作以及它在哪里失败。目标是让我们更快地发送代理人，因为我们可以信任评估所说的话。评估是重点，但不会成为边界。由于你比任何人更了解代理人的故障模式，也将有机会为代理人本身的发展做出贡献，并与评估它们的系统一起构建和改进代理。责任 ·拥有评价堆。设计和建造 LLM 作为法官系统，对人类标签进行校准，使代理质量可测量以每个代理和每次失败模式。·让释放门成为真实的。建立每个人检验带和回归检测线程连接到 CI,因此质量是自动执行的，而不是通过手动传递。·构建用户模拟器在真正用户击中之前生成测试覆盖率和逆境情况。· 将生产信号转化为改进 - 管道真正的故障将重返到评估集中，以便系统随着时间的推移复合。· 与产品合作，使"看起来好什么"变成具体可测量的标准。· 发展成代理开发 - 从你最深入了解的组件开始构建和硬化代理本身。你已经使用 LLM （代理，RAG,工具/功能调用，配套框架）- 并理解它们如何行为和破解。· 一种测量思维方式。你考虑了指标，校准和实验；你想定量是否有什么东西工作，而不仅仅是运输它。· 生产经验。你在生产中运行了 LLM 系统并处理了可靠性，延迟，成本和可观测性。· 5年以上的软件工程，最近手动 LLM / 代理系统的工作。很好拥有 ·直接评估 LLM / 代工体系的经验。
查看英文原文
About the role
We run production multi-agent systems that handle real work for a large base of users. As those systems grow, our biggest constraint is confidence: we need to know how well the agents perform, catch regressions before they ship, and keep quality steady as we release. This role owns that.
You'll build the evaluation systems behind our agents - the judges, test harnesses, and simulators that tell us whether an agent is working and where it's failing. The goal is to let us ship agents faster because we can trust what the evaluation tells us.
Evaluation is the focus, but it won't be the boundary. Because you'll understand the agents' failure modes better than anyone there will also be opportunities to contribute to agent development itself, building and improving the agents alongside the systems that evaluate them.
Responsibilities
· Own the evaluation stack. Design and build LLM-as-judge systems, calibrate them against human labels, and make agent quality measurable per-agent and per-failure-mode.
- Make the release gate real. Build per-PR eval harnesses and regression detection wired into CI, so quality is enforced automatically, not by manual passes.
- Build user simulators to generate test coverage and adversarial cases before real users hit them.
- Turn production signal into improvement - pipe real failures back into evaluation sets so the system compounds over time.
- Partner with product to turn "what good looks like" into concrete, measurable criteria.
- Grow into agent development - contribute to building and hardening the agents themselves, starting with the components you know most deeply from evaluating them.
Requirements
- Strong software engineering fundamentals. Production Python or Typescript (or similar), clean API and system design, testing, CI/CD. You write code others build on - evaluation infrastructure is real engineering.
- Hands-on LLM/agent experience. You've built with LLMs - agents, RAG, tool/function calling, orchestration frameworks (LangGraph, LangChain, or equivalent) - and understand how they behave and break.
- A measurement mindset. You reason about metrics, calibration, and experiments; you want to quantify whether something works, not just ship it.
- Production experience. You've run LLM systems in production and dealt with reliability, latency, cost, and observability.
- 5+ years software engineering, with recent hands-on LLM/agent work.
Nice to have
- Direct experience evaluating LLM/agent systems - offline/online eval, LLM-as-judge, systematic regression testing.
- Observability tooling (Arize, LangSmith, or similar).
- Arabic language / NLP experience.
- E-commerce or merchant-facing product experience.
Originally posted on Himalayas