AI 质量工程师负责人
Lead Engineer, AI Quality
ABOUT US
Circle 正在打造全球领先的 AI 驱动的全方位数字业务平台。我们让创作者、教练、教育者和企业能够将他们的受众聚集在一起，进行互动讨论、直播、活动、聊天、课程和支付——所有内容都在一个地方，全部以他们自己的品牌呈现。
我们自豪地成为一家拥有约 270 名（仍在增长）员工的完全远程公司，来自全球 30 多个国家。我们寻找世界各地的优秀人才，让他们发挥出人生中最出色的工作成果，并在他们自己的生活中产生有意义的影响。我们不追踪工作时间，但我们会非常严格地管理高标准。我们跨时区协作，高度异步，喜欢大量文档记录。
每年两次，我们会在全球美丽的地方召集全体员工参加公司会议。到目前为止，我们已在土耳其、葡萄牙、墨西哥、泰国、哥伦比亚、意大利、爱尔兰等地举办过会议，还有更多地方即将加入！
查看我们的招聘页面 https://careers.circle.so/ 了解更多关于在 Circle 工作的信息。
ABOUT THE ROLE
Circle 的 AI 质量工程团队负责衡量、诊断和提升 Circle AI 功能质量的基础架构。该团队专注于构建用于衡量、诊断和改进生产 AI 系统的基础设施，而不是 ML 研究或模型训练。
我们正在寻找一位高级工程师，帮助我们构建评估框架、可观测性工具和诊断基础设施，以判断我们的 AI 代理是否正常运行，哪里需要改进，以及如何使其更快、更高效。
这是一个亲力亲为的教练型角色。你还将领导和管理 AI 质量工程团队，面对一个雄心勃勃且不断扩展的路线图。
如果你对让 AI 系统可靠、高效且可扩展地运行充满热情，那么这个职位适合你。
WHAT YOU'LL BE DOING
- 构建并主导我们的评估基础设施。设计 CI/CD 流水线、评分器和数据集，以判断 Circle 的 AI 代理（规划器、工具调用器和子代理）是否真正有效，从单个工具调用到完整的多轮对话。
- 精准诊断质量问题的根源。追踪代理流程中的失败情况，包括计划创建与执行、工具选择、复杂区域如工作流、网站构建器和分析中的工具轨迹。将你的发现转化为原型，解决该问题或确定优先级
查看英文原文
ABOUT US
Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses. We make it possible for creators, coaches, educators, and businesses to bring together their audience with engaging discussions, live streams, events, chat, courses, and payments — all in one place, all under their own brand.
We’re proud to be a fully remote company of around 270 (and growing!) team members from 30+ countries around the world. We seek exceptional individuals around the world, set them up to do the best work of their lives, and in turn, create a meaningful impact in their own lives. We don't track hours, but we do manage for high expectations very closely. We collaborate across time zones, are highly async, and like to document a lot.
Twice a year, we bring the whole company together in beautiful places around the world for our company offsites. So far, we’ve hosted offsites in Turkey, Portugal, Mexico, Thailand, Colombia, Italy, Ireland, and more, with still more to come!
Check out our Careers https://careers.circle.so/ page for more about working at Circle.
ABOUT THE ROLE
The AI Quality engineering team at Circle owns the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features. This team focuses on building the infrastructure to measure, diagnose, and improve production AI systems, rather than ML research or model training.
We're looking for a Lead Engineer to help us build out the evaluation frameworks, observability tooling, and diagnostic infrastructure that tell us whether our AI Agents are working well, where to improve them, and how to make them faster and more cost-efficient.
This is a hands-on player-coach role. You’ll also lead and manage the AI Quality engineering team, with an ambitious and growing roadmap.
If you're excited about making AI systems work reliably, efficiently, and at scale, this is for you.
WHAT YOU'LL BE DOING
- Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation.
- Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering can help.
- Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone.
- Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity.
- Lead and grow the AI Quality engineering team. Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself.
- Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.
WHAT YOU'LL NEED TO BE SUCCESSFUL
- 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product. You've worked on complex, tool-using systems (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants, and you can walk us through something you shipped and how you knew it was actually working.
- Comfortable in Ruby on Rails / Python or ready to pick them up quickly. Ruby on Rails is our production system and the foundation that Circle’s AI Agents are built on and proficiency in Python is a strong plus, especially for the data and evaluation side of the work.
- Experience building evaluation or observability infrastructure for ML/AI systems. You've built eval pipelines, scorers, dashboards, or CI/CD for evals before and have experience with evaluation frameworks like Braintrust, LangSmith, or similar.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation. You know how to turn raw examples into a dataset you can trust through human annotation, AI-generated data, or simulation.
- Learn fast, experiment aggressively, and thrive in a highly dynamic environment. You’ll need to be comfortable running a lot of experiments and letting the results settle the argument, including the ones that don't work (which will be many of them in the beginning).
- Comfortable in a fast-paced environment with ambiguity. You’ll need to learn and pick up new technologies when projects require it.
- Strong alignment with our company values https://circle.so/careers.
- You are proficient in English (spoken, written, and reading) at a CEFR Level C2 https://magoosh.com/english-speaking/english-proficiency-levels-a-guide-to-determining-your-level/ / ILR Level 5 https://www.govtilr.org/.
COMPENSATION & BENEFITS
Circle offers U.S.-benchmarked compensation globally, equity in the company with ongoing refresh grants, and 35 days of paid time off each year.
We’re a remote-only team that comes together twice a year for company retreats in incredible destinations around the world. Alongside incredible flexibility and autonomy, we offer a benefits package that supports health, wellbeing, and professional growth. Learn more in our Candidate Hub. https://circleco.notion.site/circle-candidate-hub
LEARN MORE
- Candidate Safety & Interview Process Notice https://app.notion.com/p/circleco/Circle-Candidate-Hub-374b655313b7803d9580fb7da4f51acf?source=copy_link#374b655313b780e5b681c4d56b962790
- Diversity, Equity & Inclusion https://app.notion.com/p/circleco/Circle-Candidate-Hub-374b655313b7803d9580fb7da4f51acf?source=copy_link#374b655313b7807a8e3ff5c25a1d93ba
- How We Use Candidate Data https://app.notion.com/p/circleco/Circle-Candidate-Hub-374b655313b7803d9580fb7da4f51acf?source=copy_link#374b655313b780e49e98f390eec6bb51
- Equal Employment Opportunity https://app.notion.com/p/circleco/Circle-Candidate-Hub-374b655313b7803d9580fb7da4f51acf?source=copy_link#374b655313b7807a967dea67e68f31f0
- Visit our Candidate Hub https://circleco.notion.site/circle-candidate-hub to learn more about working at Circle, our benefits, and our hiring process.