高级HPC集群管理员 - 深度学习框架基础设施
Senior HPC Cluster Administrator - Deep Learning Frameworks Infrastructure
NVIDIA的深度学习框架（DLFW）基础设施团队正在寻找一位技术能力深厚的高级HPC集群管理员，负责设计、部署和保障我们大规模GPU计算集群的可靠性。这些系统运行着行业内最苛刻的深度学习训练、推理和高性能计算工作负载——从DGX/HGX平台到突破性的Grace Blackwell系统。你将主导计算、网络和存储方面的架构决策，并与软件、研究和产品团队紧密合作，确保我们的基础设施始终领先于所支持的工作负载。
你将负责：
- 在异构Linux环境（DGX、HGX、嵌入式系统）中，负责GPU计算集群的全生命周期管理——采购、配置、配置管理、监控和退役
- 设计并扩展存储解决方案（NFS、Lustre、WekaFS或同等方案），并制定明确的容量和性能增长路线图
- 使用现代IaC工具（Ansible、Terraform）和CI/CD流水线（GitLab）实现基础设施自动化
- 通过Slurm管理并优化作业调度，包括公平共享策略、预留管理以及MIG/GPU分区策略
- 维护和改进可观测性堆栈（Prometheus、Grafana、DCGM），并推动硬件和软件事件的主动解决
- 与机器学习工程师和软件团队合作，对大规模分布式训练工作负载的集群配置进行调优
- 评估并引入新技术——网络结构（InfiniBand、NVLink、EFA/RDMA）、存储层级、容器运行时——以提升性能和可靠性
- 指导初级工程师，并为团队工程标准做出贡献
我们希望看到：
- 计算机科学、电子工程、计算机工程或相关领域的学士/硕士学历，或同等实际经验
- 5年以上部署和管理大规模HPC或机器学习训练集群的经验
- 在大规模Linux系统管理方面有深入的专业知识
- 熟练掌握Python和/或bash的脚本编写和自动化技能
- 有Slurm使用经验（调度、计费、cgroup配置）
- 熟悉配置管理和IaC（Ansible为必需；Terraform为加分项）
- 有容器技术经验（Docker、Apptainer/Singularity、Kubernetes）
- 对高速网络（InfiniBand、RoCE、RDMA、EFA）有扎实的理解
- 有分布式/并行文件系统和存储架构的经验
- 能够从头到尾独立解决问题并推进解决
查看英文原文
NVIDIA's Deep Learning Frameworks (DLFW) Infrastructure team is looking for a deeply technical Senior HPC Cluster Administrator to lead the design, deployment, and reliability of our large-scale GPU compute clusters. These systems run the most demanding deep learning training, inference, and high-performance computing workloads in the industry — from DGX/HGX platforms to ground-breaking Grace Blackwell systems. You will drive architectural decisions across compute, networking, and storage, and partner closely with software, research, and product teams to keep our infrastructure ahead of the workloads it supports.
What you'll be doing:
- Own the full lifecycle of GPU compute clusters — procurement, provisioning, configuration management, monitoring, and deprecation — across heterogeneous Linux environments (DGX, HGX, embedded systems)
- Design and scale storage solutions (NFS, Lustre, WekaFS, or equivalent) with a clear roadmap for capacity and performance growth
- Lead automation of infrastructure using modern IaC tools (Ansible, Terraform) and CI/CD pipelines (GitLab)
- Manage and optimize job scheduling via Slurm, including fair-share policies, reservation management, and MIG/GPU partitioning strategies
- Maintain and improve observability stacks (Prometheus, Grafana, DCGM) and drive proactive resolution of hardware and software incidents
- Collaborate with ML engineers and software teams to tune cluster configuration for large-scale distributed training workloads
- Evaluate and introduce new technologies — networking fabrics (InfiniBand, NVLink, EFA/RDMA), storage tiers, container runtimes — to improve performance and reliability
- Mentor junior engineers and contribute to team-wide engineering standards
What we need to see:
- BS/MS in CS, EE, CE, or equivalent hands-on experience
- 5+ years of experience deploying and administering large-scale HPC or ML training clusters
- Deep expertise in Linux systems administration at scale
- Strong scripting and automation skills in Python and/or bash
- Hands-on experience with Slurm (scheduling, accounting, cgroup configuration)
- Proficiency with configuration management and IaC (Ansible required; Terraform a plus)
- Experience with container technologies (Docker, Apptainer/Singularity, Kubernetes)
- Solid understanding of high-speed networking (InfiniBand, RoCE, RDMA, EFA)
- Experience with distributed/parallel filesystems and storage architecture
- Ability to own problems end-to-end and communicate clearly with engineering and management stakeholders
Ways to stand out from the crowd:
- Experience with NVIDIA GPU infrastructure tools (DCGM, nvidia-smi, MIG, NVSwitch diagnostics)
- Familiarity with cluster management platforms (Colossus, Bright Cluster Manager, xCAT, or similar)
- Experience supporting large-scale distributed deep learning workloads (PyTorch, JAX, Megatron)
- Knowledge of BMC/IPMI/Redfish for out-of-band management and hardware lifecycle
- Background in MLOps tooling or ML platform engineering
Join our team of world-class engineers and be part of the groundbreaking work we do at NVIDIA. We are committed to encouraging a collaborative and inclusive environment, where every team member has the opportunity to thrive and make a significant impact!
Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. For Poland: The base salary range is 221,250 PLN - 383,500 PLN for Level 3, and 292,500 PLN - 507,000 PLN for Level 4.Originally posted on Himalayas