高级数据平台工程师
Senior Data Platform Engineer
我们正在寻找一位技能娴熟的高级数据平台工程师加入我们的数据平台团队，负责设计、构建和演进公司的数据基础设施。
这是一项注重实践的工程岗位，专注于开发可扩展的数据平台、高吞吐量的数据摄入管道以及支撑分析、产品应用、AI工作负载和大规模模型执行的基础数据系统。您将帮助构建我们的数据湖，实现可扩展的可靠数据摄入能力，并塑造支持公司未来增长的核心基础设施。
这是一个与前沿技术合作并定义我们数据平台架构和演进的绝佳机会。
要求：
- 5年以上数据平台/基础设施工程师经验
- 在数据湖架构中构建和运营可扩展数据基础设施有成功案例
- 在Kubernetes上设计和运营大规模数据处理工作负载方面有实际经验，包括使用KEDA进行工作负载自动扩展
- 熟练掌握Python和SQL
- 有使用基础设施即代码（IaC）的经验，优先考虑Terraform
- 有管理数据编排平台（如Airflow或Prefect）的经验
- 有Snowflake使用经验
职责：
- 构建复杂、高吞吐量的批处理和近实时数据摄入服务，同时帮助提升平台的流式处理能力
- 在分布式微服务环境中为所有应用需求设计和实现数据解决方案
- 开发和优化适用于批处理和流式处理用例的大规模数据管道
- 确保数据质量、合规性和治理
- 管理和优化Kubernetes基础设施，利用KEDA动态扩展资源以应对大规模数据处理任务
- 使用Terraform和Kubernetes负责我们数据栈的全生命周期管理，从资源配置到应用级框架
- 为团队提供技术支持，推动基础设施、CI/CD、测试和系统设计方面的最佳实践
- 深入研究内存管理和性能调优，确保我们的大规模分布式系统在成本效益和速度上都表现优异
我们提供的福利：
· 舒适的工作环境、具有挑战性的任务和长期有趣的项目；
· 通过试用期后可享受调动预留（针对乌克兰同事）；
· 20天的带薪年假；
查看英文原文
We are looking for a highly skilled Senior Data Platform Engineer to join our Data Platform team and play a key role in designing, building, and evolving the company’s data infrastructure.
This is a hands-on engineering position focused on developing scalable data platforms, high-volume ingestion pipelines, and the foundational data systems that power analytics, product applications, AI workloads, and large-scale model execution. You will help architect our data lake, build reliable ingestion capabilities at scale, and shape the core infrastructure that supports the company’s future growth.
This is an exciting opportunity to work with cutting-edge technologies and have a defining impact on the architecture and evolution of our data platform.
Requirements:
- 5+ years of experience as a Data Platform/Infra Engineer
- Proven track record of building and operating scalable data infrastructure specifically within Data Lake architectures.
- Proven experience designing and operating large-scale data processing workloads on Kubernetes, including workload autoscaling with KEDA.
- Strong proficiency in Python and SQL.
- Hands-on experience with Infrastructure as Code, preferably Terraform.
- Experience with managing a data orchestration platform such as Airflow or Prefect
- Experience with Snowflake.
Responsibilities:
- Build complex, high-volume batch and near-real-time ingestion services while helping evolve the platform’s streaming capabilities.
- Design and implement data solutions for all application requirements in a distributed microservices environment
- Develop and optimize large-scale data pipelines for batch and streaming use cases.
- Ensure data quality, compliance, and governance.
- Manage and optimize Kubernetes infrastructure, utilizing KEDA to dynamically scale resources for large-scale data processing workloads.
- Own the full lifecycle of our data stack using Terraform and Kubernetes, from provisioning to application-level frameworks.
- Provide technical guidance for the team, drive best practices around infrastructure, CI/CD, testing, and system design.
- Deep-dive into memory management and performance tuning to ensure our large-scale distributed systems are both cost-effective and lightning-fast.
What we offer:
- Comfortable environment, challenging tasks and a long-term interesting project;
- Mobilization reservation available upon successful completion of the probation period (for Ukrainian colleagues).
- Covered 20 days of vacation;
- Working with top notch equipment;
- Bookkeeping by a professional accountant;
- Help and support from our caring HR-team;
Originally posted on Himalayas