高级无服务器 Spark 迁移工程师
Senior Serverless Spark Migration Engineer
Senior Platform Engineer, Cloud Infrastructure
类型：远程（巴西、墨西哥）
覆盖时间：太平洋时间（上午8:00 – 下午5:00 PST）
关于Virtasant
Virtasant是一家全球云计算和技术服务公司，帮助组织实现现代化、优化并大规模构建。我们与企业客户合作，开展复杂的云计算、数据、基础设施和AI项目，整合深厚的技术专业知识和实际交付能力。
关于该职位
我们正在寻找一名高级无服务器Spark迁移工程师，以帮助现代化大型企业数据平台。
您将领导将生产环境中的Apache Spark工作负载从本地Hadoop/Spark环境迁移到AWS和GCP的云原生和无服务器架构。这是一个涉及工作负载评估、架构设计、应用重构、迁移执行、性能优化、自动化和生产就绪性的实际工程岗位。
目标不仅仅是简单地进行上云迁移。您需要为每个工作负载确定合适的目标架构，并建立可重复的模式，最终支持在企业规模上的迁移。
您将负责
- 领导企业Spark工作负载从本地环境迁移到AWS和GCP。
- 评估Spark应用程序、集群、配置、依赖项、数据流和资源使用情况。
- 确定适合的迁移方法，包括重新托管、重新平台化、重构、现代化或退役。
- 在适当的情况下，将传统基于集群的工作负载现代化为无服务器Spark。
- 使用AWS EMR Serverless、S3、Glue、Lake Formation、GCP Dataproc Serverless、GCS和BigQuery等技术设计和实现架构。
- 重构遗留的PySpark/Scala/Spark SQL应用程序，以提高云可移植性、可扩展性和可靠性。
- 将工作负载从使用Hadoop、HDFS、YARN、Hive和本地Spark集群的环境中迁移出来。
- 排查和优化Spark工作负载，涉及分区、shuffle行为、连接、数据倾斜、执行计划、执行器配置、序列化和SQL执行。
- 基准测试性能，并优化无服务器工作负载以提高性能、可靠性和云成本。
- 构建可重用的迁移工具、自动化、模板和框架。
- 使用Terraform等工具实现CI/CD和基础设施即代码。
- 定义测试、验证、切换、回滚、可观测性和生产就绪性模式。
- 与数据工程、机器学习、云计算团队合作
查看英文原文
Senior Platform Engineer, Cloud Infrastructure
Type: Remote (Brazil, Mexico)
Coverage: Pacific Hours (8:00 AM – 5:00 PM PST)
About Virtasant
Virtasant is a global cloud and technology services company helping organizations modernize, optimize, and build at scale. We work with enterprise customers on complex cloud, data, infrastructure, and AI initiatives, bringing together deep technical expertise and hands-on delivery.
About the Role
We’re looking for a Senior Serverless Spark Migration Engineer to help modernize a large-scale enterprise data platform.
You’ll lead the migration of production Apache Spark workloads from on-premise Hadoop/Spark environments to cloud-native and serverless architectures across AWS and GCP. This is a hands-on engineering role spanning workload assessment, architecture, application refactoring, migration execution, performance optimization, automation, and production readiness.
The goal is not simply to lift and shift existing workloads. You’ll determine the right target architecture for each workload and establish repeatable patterns that can eventually support migration at significant enterprise scale.
What You’ll Do
- Lead migrations of enterprise Spark workloads from on-premise environments to AWS and GCP.
- Assess Spark applications, clusters, configurations, dependencies, data flows, and resource utilization.
- Determine the right migration approach across rehost, replatform, refactor, modernize, or retire.
- Modernize traditional cluster-based workloads for serverless Spark where appropriate.
- Design and implement architectures using technologies such as AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS, and BigQuery.
- Refactor legacy PySpark/Scala/Spark SQL applications for cloud portability, scalability, and reliability.
- Migrate workloads from environments using Hadoop, HDFS, YARN, Hive, and on-prem Spark clusters.
- Troubleshoot and optimize Spark workloads across partitioning, shuffle behavior, joins, data skew, execution plans, executor configuration, serialization, and SQL execution.
- Benchmark performance and optimize serverless workloads for performance, reliability, and cloud cost.
- Build reusable migration tooling, automation, templates, and frameworks.
- Implement CI/CD and Infrastructure as Code using tools such as Terraform.
- Define testing, validation, cutover, rollback, observability, and production-readiness patterns.
- Partner with Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps teams.
What We’re Looking For
- 8+ years of experience across data engineering, distributed systems, cloud engineering, or platform engineering.
- 5+ years of hands-on Apache Spark experience in enterprise environments.
- Strong PySpark and/or Scala development experience.
- Proven experience migrating large-scale Spark workloads between infrastructure platforms.
- Hands-on experience with both AWS and GCP.
- Experience with on-premise Hadoop/Spark ecosystems, including technologies such as HDFS, YARN, and Hive.
- Deep understanding of Spark internals and distributed processing.
- Strong SQL and data engineering fundamentals.
- Experience with cloud data lakes and object storage.
- Strong production troubleshooting and performance-tuning experience.
- Experience with CI/CD, Git, and Infrastructure as Code.
- Ability to own migration work end-to-end, from discovery and architecture through production cutover and optimization.
Nice to Have
Experience with EMR/EMR Serverless, Dataproc/Dataproc Serverless, Glue, Lake Formation, BigQuery, Delta Lake, Iceberg, Kafka, Airflow, Terraform, Docker, or Kubernetes is valuable.
What Success Looks Like
You can take ownership of the complete migration lifecycle:
Discover → Assess → Design → Refactor → Migrate → Validate → Optimize → Operate
You understand both the legacy Hadoop/Spark world and modern cloud-native data platforms, and can make pragmatic architecture decisions based on workload characteristics rather than simply reproducing an existing environment in the cloud.
Originally posted on Himalayas