技术员工，机器学习平台
Member of Technical Staff, ML Platform
我们正在通过融合艺术与科学，构建AI来模拟世界。
我们认为，世界模型是人工智能进步的前沿。仅靠语言模型无法解决世界上最困难的问题——机器人技术、疾病、科学发现。真正的进步需要能够体验世界并从错误中学习的模型，就像人类一样。而这种试错过程在模拟中可以被大幅加速，而不是在现实世界中。
世界模型为通用模拟提供了最清晰的路径，改变故事的讲述方式、科学进步的方式以及人类下一个前沿领域的探索方式。
我们的团队由富有创造力、开放思维、富有同理心和有抱负的人组成，他们决心改变世界。我们渴望不断创造不可能的事情，而我们实现这一目标的能力取决于打造一支卓越的团队。如果你也渴望做到这一点，我们很期待收到你的消息。
### **关于该职位**
我们正在寻找一名机器学习基础设施工程师，负责Runway中的模型评估工作，从端到端进行管理。我们对模型所做的每一个决定——保留哪个检查点继续训练、向数百万用户发布什么、哪种数据混合和架构最有前景——都依赖于评估。目前，这项工作分散在整个组织中。你将把它整合成一个平台。
你的工作将是设计和构建生成样本的系统，使用自动化指标和人工标注进行评分，跟踪不同检查点和版本的结果，并在几分钟而不是几天内将结果呈现给研究人员。你将定义“更好”的操作性含义——我们如何衡量它，我们有多确定，以及一个结果如何成为发布或不发布的决策。
你将作为ML Platform团队的一员，嵌入到研究团队中，并为公司范围内的评估设定技术方向。这是一个非常具有杠杆效应的角色：我们模型的质量受限于我们测量它们的能力。
### **我们技术栈的一瞥**
我们的推理和训练代码使用Python（PyTorch）编写，是云原生的。我们利用多个云服务，并使用Kubernetes原生和自研工具组合，以实现高效的作业编排。
我们大量使用代理式开发和运维。我们有一个强大的内部平台，并为最新模型和资源提供广泛访问权限。
你将可以使用一流的模型、代理、GPU、存储和云服务。
查看英文原文
We are building AI to simulate the world through merging art and science.
We believe that world models are at the frontier of progress in artificial intelligence. Language models alone won’t solve the world’s hardest problems – robotics, disease, scientific discovery. Real progress requires models that experience the world and learn from their mistakes, the same way that humans do. And this kind of trial and error can be massively accelerated when done in simulation, rather than in the real world.
World models offer the most clear path to general-purpose simulation, changing how stories are told, how scientific progress is made and how the next frontiers of humanity are reached.
Our team consists of creative, open minded, caring and ambitious people who are determined to change the world. We aspire to continuously build impossible things and our ability to do so relies on building an incredible team. If you are driven to do the same, we'd love to hear from you.
### **About the role**
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform.
Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.
You'll be embedded within research teams as a member of ML Platform, and you'll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them.
### **A peek at our technical stack**
Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration.
We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses.
You’ll have access to best-in-class models, agents, GPUs, storage and cloud services you need to drive a world class evaluation pipeline that helps us deliver frontier models.
### **What you'll do**
- Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale
- Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
- Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform
- Help set the standards for how we evaluate models at Runway: reproducibility, metric definitions, reporting formats, and when a result is trustworthy enough to act on
- Support broad adoption of the platform and its integration throughout Runway’s research efforts across training, production model serving and more
- Contribute broadly as a member of the ML Platform team to tools and systems that help Runway train and serve frontier models
### **What you'll need**
- 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
- Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
- Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
- Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
- Comfort building internal tools end to end, from the command line to the browser
- Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
- Familiarity with the full model development lifecycle: data, training, evaluation, serving
- Self-starter who can work embedded with research teams and move fast
- Strong systems thinking and pragmatic approach to production reliability
- Humility and open mindedness; at Runway we love to learn from one another
### **Nice to have**
- Track record of building evaluation suites for generative models
- Hands-on work with LLM- or VLM-as-judge pipelines
- Experience with online experimentation platforms and connecting offline metrics to product outcomes
- Prior work evaluating agents or robotics policies
Runway strives to recruit and retain exceptional talent from diverse backgrounds while ensuring pay equity for our team. Our salary ranges are based on competitive market rates for our size, stage and industry, and salary is just one part of the overall compensation package we provide.
There are many factors that go into salary determinations, including relevant experience, skill level and qualifications assessed during the interview process, and maintaining internal equity with peers on the team. The range shared below is a general expectation for the function as posted, but we are also open to considering candidates who may be more or less experienced than outlined in the job description. In this case, we will communicate any updates in the expected salary range.
Lastly, the provided range is the expected salary for candidates in the U.S. Outside of those regions, there may be a change in the range, which again, will be communicated to candidates.
#### **Working at Runway**
[**Great things come from great teams.**](https://www.youtube.com/watch?v=kwmj4ato2kw&ab_channel=Runway) **We’d love to hear from you.**
We’re committed to creating a space where our employees can bring their full selves to work and have equal opportunity to succeed. So regardless of race, gender identity or expression, sexual orientation, religion, origin, ability, age, veteran status, if joining this mission speaks to you, we encourage you to apply.
More about Runway
- [Universal World Simulator](https://runwayml.com/world-simulator.html)
- [GWM-1](https://runwayml.com/research/introducing-runway-gwm-1)
- [Gen-4.5](https://runwayml.com/research/introducing-runway-gen-4.5)
- [General World Models](https://runwayml.com/research/introducing-general-world-models)
- [Robotics SDK](https://runwayml.com/research/introducing-runway-gwm-1#robotics-section)
- [Conversational Real-time Agents](https://runwayml.com/research/introducing-runway-gwm-1#avatars-section)
- [Runway Studios](https://runwayml.com/studios)
We're excited to be recognized as a best place to work:
[Crain's](https://www.crainsnewyork.com/awards/best-places-work-2023) | [InHerSight](https://www.inhersight.com/companies/best/city/new-york-city-ny) | [BuiltIn NYC](https://builtin.com/awards/new-york-city/2024/best-places-to-work) | [INC](https://www.inc.com/best-workplaces/2024)