Azure Data Integration Engineer
该职位涉及构建和管理数据管道，解决故障问题，并确保在 Azure Synapse Analytics,Azure Data Lake Gen2和 SQL 环境等各种平台上的数据准确性。此职位需要广泛的 SQL 经验和 PySpark 开发方面的强背景。责任数据工程：·与 Azure Synopse Pipelines 和 Py Spark 合作进行数据转型和管道管理。·在 Delta Lake 环境中执行数据集成和方案更新，以确保流动的数据流量和精确的报告。·与我们的 Azure DevOps 团队一起工作 CI/CD 进程部署基础设施作为代码 (IaC) 和工作空间文物。· 开发客户定义的定制解决方案，并帮助随时间而来改善我们的数据解决方案模式。文档：· 记录票分辨率，测试协议和数据验证流程。· 与其他利益相关者合作提供规格和报价，以满足客户要求的改进。门票管理：·监控 Jira 门票队列，并在提升时响应门票问题。·理解门票的问题，利用广泛的 SQL,synapse 分析和其他工具来解决这些问题。· 与提升票的客户用户有效沟通，并根据需要与其他团队 （例如 FinOps,Databricks）合作解决问题。故障处理和支持：·处理有关 ETL 管道故障，多尔达湖处理或数据不一致性的问题在 Synapse Analytics. ·为数据管道和验证问题的迅速解决，确保数据完整性和性能。所需技能 & 要求寻找具有5年以上 dynamics 365生态系统经验和强大的 PySpark 开发背景的候选人。虽然各种配置可能适用，但我们高度重视一个强有力的个人组织合适应。我们的理想候选人拥有以下属性和资格：· 广泛的 SQL 经验，包括 Azure SQL,Synapse Analytics 和 Delta Lake 环境中的查询编写和故障解决。· 在实施和支持 ETL 流程，数据湖泊和数据工程解决方案方面有强大的理解和经验。· 使用 Azure Synapse Analytics 的能力，包括工作空间管理，管道创建和数据流量管理。· 用 PySpark 进行数据处理和自动化实践经验。•使用 VPN,MFA,RDP,跳板/跳跃主机的能力。
查看英文原文
The role involves building and managing data pipelines, troubleshooting issues, and ensuring data accuracy across various platforms such as Azure Synapse Analytics, Azure Data Lake Gen2, and SQL environments.
This position requires extensive SQL experience and a strong background in PySpark development.
Responsibilities
Data Engineering:
- Work with Azure Synapse Pipelines and PySpark for data transformation and pipeline management.
- Perform data integration and schema updates in Delta Lake environments, ensuring smooth data flow and accurate reporting.
- Work with our Azure DevOps team on CI/CD processes for deployment of Infrastructure as Code (IaC) and Workspace artifacts.
- Develop custom solutions for our customers defined by our Data Architect and assist in improving our data solution patterns over time.
Documentation:
- Document ticket resolutions, testing protocols, and data validation processes.
- Collaborate with other stakeholders to provide specifications and quotations for enhancements requested by customers.
Ticket Management:
- Monitor the Jira ticket queue and respond to tickets as they are raised.
- Understand ticket issues, utilizing extensive SQL, Synapse Analytics, and other tools to troubleshoot them.
- Communicate effectively with customer users who raised the tickets and collaborate with other teams (e.g., FinOps, Databricks) as needed to resolve issues.
Troubleshooting and Support:
- Handle issues related to ETL pipeline failures, Delta Lake processing, or data inconsistencies in Synapse Analytics.
- Provide prompt resolution to data pipeline and validation issues, ensuring data integrity and performance.
Desired Skills & Requirements
Seeking a candidate with 5+ years of Dynamics 365 ecosystem experience with a strong PySpark development background. While various profiles may apply, we highly value a strong person-organization fit.
Our ideal candidate possesses the following attributes and qualifications:
- Extensive experience with SQL, including query writing and troubleshooting in Azure SQL, Synapse Analytics, and Delta Lake environments.
- Strong understanding and experience in implementing and supporting ETL processes, Data Lakes, and data engineering solutions.
- Proficiency in using Azure Synapse Analytics, including workspace management, pipeline creation, and data flow management.
- Hands-on experience with PySpark for data processing and automation.
- Ability to use VPNs, MFA, RDP, jump boxes/jump hosts, etc., to operate within the customers secure environments.
- Some experience with Azure DevOps CI/CD IaC and release pipelines.
- Ability to communicate effectively both verbally and in writing, with strong problem-solving and analytical skills.
- Understanding of the operation and underlying data structure of D365 Finance and Operations, Business Central, and Customer Engagement.
- Experience with Data Engineering in Microsoft Fabric
- Experience with Delta Lake and Azure data engineering concepts (e.g., ADLS, ADF, Synapse, AAD, Databricks).
- Certifications in Azure Data Engineering.
Why Join Us?
- Opportunity to work with innovative technologies in a dynamic environment where progressive work culture with a global perspective where your ideas truly matter, and growth opportunities are endless.
- Work with the latest Microsoft Technologies alongside Dynamics professionals committed to driving customer success.
- Enjoy the flexibility to work from anywhere
- Work-life balance that suits your lifestyle.
- Competitive salary and comprehensive benefits package.
- Career growth and professional development opportunities.
- A collaborative and inclusive work culture.
Originally posted on Himalayas