Data Platform Engineer (102-08SENG-01)
这一角色将一个大型 Azure 数据工厂的产业转化为 AWS 上的 Databricks 工作流程。第一年范围：2,094个 ADF 管道迁移 作为可重复使用模板而不是单独构建 9,859个管道活动进行翻译 （一些地图直接，其他需要重新编写为 Lambda 或步骤函数）,471个 Spark 数据流向 AWS 上的 Databricks 工作并保留 4个 Databricks 的工作空间 (Dev,QA,Pre-prod,Prod).这是一种受监管环境，因此与源系统之间的迁移数据是定义的一部分，而不是后期思考。要求你会做什么？· 将 Azure Data Factory 管道转换为 AWS 上的数据工作流，而不是一次迁移可重复使用模板。· Rehost Databricks 工作空间进入 AWS 并将 ADLS Gen2存储迁至 S3. · 重写 ADF Web 活动作为 Lambda 函数或步骤函数任务，并用本土 Databricks 机制取代 ADF 特定规模化。· 为生产数据量构建和调整 PySpark 变革。· 用 Databricks 仓库替换 Azure Synapse SQL Server. · 根据定义进行源系统的协调。· 强大的 Spark 和 PySpark 经验，用于实际数据量，包括调整。· 生产级的 Python. · 建立或迁移 Azure Data Factory 管道，并有对 ADF 活动模式的坚实的了解。· AWS 数据服务：S3,Glue,Athena,Lambda 和 Step Functions. · 高端 SQL,包括阅读和考虑存储程序。·专业的书面和语音英语。有 Delta Lake,Unity Catalog,Azure Synapse,Terraform,Airflow/MWAA,dbt,Kafka,Databricks 认证，数据建模，大期待，SAS/分析平台集成，CRM 数据。参与细节 · 全职 · 100%远程开放 · 从所有 LATAM 亮点来申请 数据，PySpark,Python,Azure Data Factory,AWS (S3,Glue,Athena,Lambda, Step Functions),SQL,Delta Lake,团结目录，Azure Synapse,Terraform,Airflow/MWAA,dbt,卡夫卡 原始发布在喜马拉雅山
查看英文原文
This role converts a large Azure Data Factory estate into Databricks workflows on AWS. The scope for year one: 2,094 ADF pipelines to migrate — built as reusable templates rather than one-by-one — 9,859 pipeline activities to translate (some map directly, others need rewriting as Lambda or Step Functions), 471 Spark dataflows to move onto Databricks on AWS, and 4 Databricks workspaces (Dev, QA, Pre-prod, Prod) to rehost, including notebook paths and Unity Catalog rewiring. This is a regulated environment, so reconciling migrated data against source systems is part of the definition of done, not an afterthought.
Requirements
What you will do
- Convert Azure Data Factory pipelines into Databricks workflows on AWS, building reusable templates rather than migrating one at a time.
- Rehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3.
- Rewrite ADF Web Activities as Lambda functions or Step Functions tasks, and replace ADF-specific scaling with native Databricks mechanisms.
- Build and tune PySpark transformations for production data volumes.
- Replace Azure Synapse Serverless with Databricks SQL Warehouse.
- Reconcile migrated data against source systems as part of the definition of done.
Required
- Production experience with Databricks: workspaces, jobs and workflows. The central skill for this role.
- Strong Spark and PySpark experience for real data volumes, including tuning.
- Production-grade Python.
- Experience building or migrating Azure Data Factory pipelines, with a solid understanding of the ADF activity model.
- AWS data services: S3, Glue, Athena, Lambda and Step Functions.
- Advanced SQL, including reading and reasoning about stored procedures.
- Professional written and spoken English.
Nice to have
Delta Lake, Unity Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka, Databricks certification, data modeling, Great Expectations, SAS/analytics platform integration, CRM data.
Engagement details
- Full-time
- 100% remote
- Open to candidate from all LATAM
Highlights
Databricks, PySpark, Python, Azure Data Factory, AWS (S3, Glue, Athena, Lambda, Step Functions), SQL, Delta Lake, Unity Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka
Originally posted on Himalayas