高级数据工程师（Databricks & PySpark）
Senior Data Engineer (Databricks & PySpark)
Senior Data Engineer (Databricks & PySpark)
地点：100% 远程（波兰华沙）
薪资：最高 38 EUR/小时
工作模式：全职（每周40小时）
合作方：Optiveum（B2B 合作协议）
职位简介：
我们代表客户寻找一位经验丰富的 Senior Data Engineer，负责设计、构建和运营其可持续发展数据基础架构（SDF）。该 SDF 通过基于 Databricks 的现代数据平台，支持可信、可扩展且可审计的可持续发展报告和分析。
您将加入一个专注于可靠性、运营卓越和实用解决方案的可持续发展数据与 AI 团队，而非不必要的复杂性。该职位的核心是交付能够实现监管合规和高级分析的数据产品和管道。
主要职责：
- 数据工程与产品开发：使用 Databricks、Delta Lake 和 PySpark 设计并实现可扩展的数据产品。构建和维护从 ERP、采购、可持续发展和外部系统处理复杂数据集的数据管道。
- 性能优化：优化工作负载以提高性能、可扩展性和成本效率，构建可重用的工程模式。
- 数据质量和治理：在整个数据生命周期中实施自动化数据质量控制。主动识别并解决数据问题，防止其影响报告。
- 平台工程与 DevOps：实现 CI/CD 管道、自动化部署、测试框架和基础设施即代码（IaC）。支持平台安全控制和访问管理。
- 利益相关者协作：与可持续发展专家、业务分析师和报告团队合作，支持需求收集、方案设计和生产发布。
任职要求：
- 5–7 年设计、开发和运营数据平台和管道的经验。
- 在生产环境中具有丰富的 Databricks（包括 Azure Databricks 和 Databricks Workflows）实战经验。
- 对 PySpark、Python、Spark SQL、数据建模和数据管道设计具有专家级熟练程度。
- 具有为数据工程工作负载实施 CI/CD 管道、基于 Git 的开发和版本控制的经验。
- 对数据血缘、元数据管理、治理、审计和验证框架有扎实的理解。
- 英语流利（书面和口语）。
- 主动、自主的心态，具备强大的故障排除和根本原因分析能力。
加分项：
- 具有...
查看英文原文
Senior Data Engineer (Databricks & PySpark)
Location: 100% Remote (Warsaw, PL)
Rate: Up to 38 EUR / hour
Mode of work: Full-time (40 hours/week)
Contracting party: Optiveum (B2B Cooperation Agreement)
About the role:
We are seeking an experienced Senior Data Engineer on behalf of our client to design, build, and operate their Sustainability Data Foundation (SDF). The SDF supports trusted, scalable, and auditable sustainability reporting and analytics through a modern data platform built on Databricks.
You will join a Sustainability Data & AI team that values reliability, operational excellence, and practical solutions over unnecessary complexity. The focus of this role is to deliver data products and pipelines that enable regulatory compliance and advanced analytics.
Key Responsibilities:
- Data Engineering & Product Development: Design and implement scalable data products using Databricks, Delta Lake, and PySpark. Build and maintain data pipelines processing complex datasets from ERP, procurement, sustainability, and external systems.
- Performance Optimization: Optimize workloads for performance, scalability, and cost efficiency, building reusable engineering patterns.
- Data Quality & Governance: Implement automated data quality controls throughout the data lifecycle. Proactively identify and resolve data issues before they impact reporting.
- Platform Engineering & DevOps: Implement CI/CD pipelines, automated deployments, testing frameworks, and Infrastructure as Code (IaC). Support platform security controls and access management.
- Stakeholder Collaboration: Partner with sustainability experts, business analysts, and reporting teams to support requirements gathering, solution design, and production releases.
Required Qualifications:
- 5–7 years of experience designing, developing, and operating data platforms and pipelines.
- Extensive hands-on experience with Databricks (including Azure Databricks and Databricks Workflows) in production environments.
- Expert-level proficiency in PySpark, Python, Spark SQL, Data Modelling, and Data Pipeline Design.
- Experience implementing CI/CD pipelines for data engineering workloads, Git-based development, and version control.
- Solid understanding of data lineage, metadata management, governance, auditing, and validation frameworks.
- Fluent English (both written and spoken).
- Proactive, self-driven mindset with strong troubleshooting and root-cause analysis skills.
Nice to have:
- Experience with sustainability reporting, ESG data, and regulatory reporting requirements.
- Knowledge of procurement, supplier, finance, and ERP data domains.
- Experience developing Databricks applications, dashboards, or user-facing data tools.
Please note: Optiveum acts as the recruitment partner and the direct contracting party for this position. The selected candidate will sign a cooperation agreement directly with Optiveum.
Originally posted on Himalayas