la26.006 - 数据工程负责人
la26.006 - Data Engineering Lead
Tiger Analytics 是一家领先的 AI 和高级分析咨询公司，致力于为全球财富 100 强企业提供有影响力的基于数据的解决方案。我们专注于通过创新的数据工程、机器学习和 AI 技术帮助组织解决复杂的业务挑战。我们正在寻找一位经验丰富的数据工程负责人，负责管理和领导复杂的数据工程项目，推动交付工作的卓越与创新
要求
• 我们正在寻找一位具有四年以上现代数据栈技术经验的资深数据工程师，特别关注 Azure Databricks 和包含 Bronze、Silver 和 Gold 处理层的 Medallion 架构
• 具备设计、开发和优化可扩展的云数据管道的经验是必要的
• 管道开发与编排：负责使用 Azure Databricks、PySpark 和 Azure Data Factory (ADF) 创建、部署和维护生产级数据管道
• Medallion 框架管理：负责在 Bronze、Silver 和 Gold 层上开发和管理完整的 Lakehouse 管道架构
• Delta Lake 优化：具备应用 Delta Lake 高级功能（如 Z-Ordering、液体聚类、清理和分区技术）的专业知识，以减少查询成本并提高读写效率
• 基础设施即代码与 CI/CD：参与使用 Terraform、Git 和 Azure DevOps 管理和部署云基础设施，自动化数据管道的部署
• 数据治理与安全：实施企业级数据治理框架，执行访问控制，并使用 Unity Catalog 管理数据血缘追踪
• 性能调优：通过优化集群大小、内存分配和执行计划来监控、分析和解决分布式 Spark 集群中的性能瓶颈
• 需要具备英语口语和书面沟通能力
• 工作时间从 PST 时间上午 8:00 到下午 5:00
福利
- 协作和创新的工作环境
- 与全球客户合作的有影响力项目的机遇
- 针对中高级专业人员的具有竞争力的薪资和福利
- 持续学习和职业发展机会
- 基于墨西哥的灵活工作安排
最初发布于 Himalayas
查看英文原文
Tiger Analytics is a leading AI and advanced analytics consulting firm dedicated to delivering impactful data-driven solutions to Fortune 100 companies worldwide. We specialize in helping organizations solve complex business challenges through innovative Data Engineering, Machine Learning, and AI techniques. We are currently seeking a highly skilled Data Engineering Lead to manage and lead complex data engineering projects, driving excellence and innovation in our delivery.
Requirements
• We are seeking a highly experienced Data Engineer with over four years of expertise in modern data stack technologies, specifically focusing on Azure Databricks and the Medallion Architecture encompassing the Bronze, Silver, and Gold processing layers.
• Proven experience in designing, developing, and optimizing scalable, cloud-based data pipelines is essential.
• Pipeline Development & Orchestration: Responsible for creating, deploying, and maintaining production-grade data pipelines utilizing Azure Databricks, PySpark, and Azure Data Factory (ADF).
• Medallion Framework Management: Accountable for developing and managing the comprehensive Lakehouse pipeline architecture across the Bronze, Silver, and Gold layers.
• Delta Lake Optimization: Expertise in applying advanced Delta Lake features such as Z-Ordering, liquid clustering, vacuuming, and partitioning techniques to minimize query costs and improve read/write efficiency.
• Infrastructure as Code & CI/CD: Collaborate in managing and deploying cloud infrastructure using Terraform, Git, and Azure DevOps, automating the deployment of data pipelines.
• Data Governance & Security: Implement enterprise-level data governance frameworks, enforce access controls, and manage data lineage tracking utilizing Unity Catalog.
• Performance Tuning: Monitor, analyze, and address performance bottlenecks in distributed Spark clusters by optimizing cluster sizing, memory allocation, and execution plans.
• Proficiency in English, both spoken and written, is required for effective communication.
• Working hours are from 8:00 AM to 5:00 PM PST.
Benefits
- A collaborative and innovative work environment.
- Opportunities to work on impactful projects with global clients.
- Competitive salary and benefits tailored to mid to senior professionals.
- Continuous learning and career development opportunities.
- Flexible working arrangements based in Mexico.
Originally posted on Himalayas