高级/资深机器学习运维工程师
Senior / Staff ML Ops Engineer
Waabi由人工智能先驱Raquel Urtasun创立，是物理AI领域的领导者。我们拥有世界级的团队，正在通过技术解锁自动驾驶运输的新时代，这些技术正在为商用自动驾驶卡车和机器人出租车提供支持。Waabi得到了人工智能、汽车、物流和深度科技领域领导企业的投资并与其合作。
在多伦多、旧金山、达拉斯和匹兹堡设有办公室，Waabi正在快速成长，寻找具有多样性、创新性和协作精神的候选人，希望以积极的方式影响世界。如需了解更多信息，请访问：www.waabi.ai
你将：
在Kubernetes上构建和演进我们的训练基础设施，包括基础设施——GPU调度、自动扩展、多节点分布式任务、容量策略，以及保持长时间训练可靠的运营商和工作流引擎。
塑造面向开发者的界面——命令行工具、SDK、任务提交、模板、便捷路径——这些设计是为实际使用它们的团队量身定制的。让常见情况只需一个命令，同时保持不常见情况的可行性。
缩短内部循环。首次训练运行时间、编辑到信号延迟、任务提交前的本地迭代、快速失败而非缓慢的神秘问题。测量它，发布它，不断降低它。
推广最佳实践工具和框架。跟踪生态系统中正在发布的工具，诚实评估，并通过工作原型和迁移路径进行论证——或者当一个新事物不值得切换成本时，明确说明。
加强数据和制品层。数据集版本控制、分片和大规模多模态传感器数据的高吞吐量加载，使任务能够充分利用GPU而不是等待I/O。
将一次性Python脚本转化为持久的工具——经过测试、文档齐全、可观察的库、命令行工具和带有合理默认值的服务，并在需要时删除过时的内容。
让实验变得清晰易懂，与实际生活在其中的团队一起：实验卫生、研究人员信任的仪表板、真正的模型注册表，以及从数据集到检查点再到模拟结果的完整溯源。
为模型、自主系统和模拟构建CI/CD，因此模型更改的验证方式应与代码更改相同。
在整个机器学习堆栈中建立可观测性——利用率、吞吐量、故障模式、队列时间、每次实验的成本。当任务在凌晨3点在节点47上失败时，研究人员应该能知道原因而不需要你。
将文档、入职和支援视为产品界面——黄金路径指南，使新研究员在第二天就能高效工作，通过办公时间将重复的问题转化为已发布的修复。
查看英文原文
Waabi, founded by AI visionary Raquel Urtasun, is the leader in Physical AI. With a world-class team, we're unlocking the next era of autonomous transportation with technology that's powering commercial autonomous trucks and robotaxis. Waabi is backed by and partners with world leaders in AI, automotive, logistics, and deep tech.
With offices in Toronto, San Francisco, Dallas, and Pittsburgh, Waabi is growing quickly and looking for diverse, innovative and collaborative candidates who want to impact the world in a positive way. To learn more visit: www.waabi.ai
You will..
Build and evolve our training infrastructure on Kubernetes with Infrastructure — GPU scheduling, autoscaling, multi-node distributed jobs, capacity strategy, and the operators and workflow engines that keep long-running training reliable.
Shape the developer-facing surface — CLIs, SDKs, job submission, templates, paved paths — designed with the teams who'll use them. Make the common case one command and keep the uncommon case possible.
Shorten the inner loop. Time to first training run, edit-to-signal latency, local iteration before a job hits the cluster, fast failure over slow mystery. Measure it, publish it, drive it down.
Evangelize best-in-class tooling and frameworks. Track what the ecosystem is shipping, evaluate honestly, and make the case with working prototypes and migration paths — or say plainly when a shiny thing isn't worth the switching cost.
Strengthen the data and artifact layer. Dataset versioning, sharding, and high-throughput loading of large multimodal sensor data, so jobs saturate GPUs instead of waiting on I/O.
Turn one-off Python into durable tooling — tested, documented, observable libraries, CLIs, and services with sane defaults, and deletions where they're overdue.
Make experiments legible, with the teams who live in them: experiment hygiene, dashboards researchers trust, a real model registry, and lineage from dataset to checkpoint to simulation result.
Ship CI/CD for models alongside autonomy and simulation, so a model change is validated the same way a code change is.
Build observability across the ML stack — utilization, throughput, failure modes, queue times, cost per experiment. When a job fails at 3am on node 47, the researcher should find out why without you.
Treat docs, onboarding, and support as product surface — golden-path guides, a new researcher productive on day two, office hours that turn repeat questions into shipped fixes.
Drive adoption, not just availability. Prototype with real users, watch them work, iterate. A tool nobody adopts didn't ship.
Make the platform boringly reliable — fewer failures, faster recovery, and none of the manual steps that quietly cost a team days.
Build guardrails that don't feel like walls, with Security, IT, and Infrastructure: access controls, data handling, and cost governance that hold up in an IP-sensitive environment while staying self-serve.
Qualifications:
5+ years of software or infrastructure engineering, including tools or platforms used by other engineers and operating ML or data-intensive production systems.
Hands-on Kubernetes expertise — GPU scheduling, autoscaling, Helm or equivalent, networking fundamentals, and the ability to debug a cluster under load rather than restart it.
Excellent Python, and a track record of designing APIs and CLIs other people enjoy using.
Practical AWS depth: object storage at scale, IAM, GPU compute, networking, cost management, and infrastructure as code (Terraform, Pulumi, or similar).
Distributed training in PyTorch (DDP, FSDP, or similar), plus experiment tracking and model registry tooling — from the perspective of someone who made them pleasant for others to use.
Fluency with containers, CI/CD, and modern build systems, including large monorepos.
The ability to influence without authority: evaluate a framework on its merits, pilot it credibly, and persuade skeptical senior engineers to change how they work.
A collaborative default — you'd rather co-own a system than draw a boundary around your part of it.
User empathy: you'd rather fix the third-most-interesting problem blocking ten people than the most interesting one blocking nobody.
Strong product instincts, strong writing, and comfort operating autonomously in ambiguous territory.
Passionate about self-driving technologies and frontier AI, and about what a small, world-class team can do with the right infrastructure.
Bonus/nice to have:
Internal developer platform, research platform, or DevEx work — with a story about a tool whose adoption you grew from zero.
Large-scale distributed GPU training: hundreds to thousands of accelerators, NCCL, high-performance cluster networking, collective communication tuning.
High-throughput loading of LiDAR or camera data, and formats such as Parquet or WebDataset.
Workflow and scheduling systems — Argo Workflows, Ray, Flyte, Kubeflow, or Slurm.
Build-system depth (Bazel or similar), including remote caching in a monorepo.
Simulation infrastructure or large-scale batch evaluation pipelines.
Background in ML, robotics, or autonomous systems infrastructure.
Security- and IP-sensitive production environments.
Open-source contributions to ML infrastructure or developer tools.
The US yearly salary range for this role is: $184,000 - $272,000 USD in addition to competitive perks & benefits. Waabi (US) Inc.’s yearly salary ranges are determined based on several factors in accordance with the Company’s compensation practices. The salary base range is reflective of the minimum and maximum target for new hire salaries for the position across all US locations. Note: The Company provides additional compensation for employees in this role, including equity incentive awards and an annual performance bonus.
Perks/Benefits:
Competitive compensation and equity awards.
Health and Wellness benefits encompassing Medical, Dental and Vision coverage (for full-time employees only).
Unlimited Vacation.
Flexible hours and Work from Home support.
Daily drinks, snacks and catered meals (when in office).
Regularly scheduled team building activities and social events both on-site, off-site & virtually.
As we grow, this list continues to evolve!
Waabi is a technology start-up building technologies to transform the way the world moves. Join our talented team to be a part of the future and to make an impact!
Waabi is an equal opportunity employer. We celebrate diversity and are committed to creating a supportive, inclusive, and accessible workplace for all our employees. We seek applicants of all backgrounds and identities, across race, color, ethnicity, national origin or ancestry, age, citizenship, religion, sex, sexual orientation, gender identity or expression, military or veteran status, marital status, pregnancy or parental status, caregiver status, disability, or any other characteristic protected by law. We make workplace accommodations for qualified individuals with disabilities as required by applicable law. If reasonable accommodation is needed to participate in the job application or interview process please let our recruiting team know.