RQ08913 - 软件开发工程师 - ETL - 高级
RQ08913 - Software Developer - ETL - Senior
职责：
- 为数据采集和存储到我们集中的数据仓库设计技术解决方案。
- 开发ELT脚本，设计数据驱动逻辑并进行单元测试。
- 进行数据库建模和设计以提升整体性能。
- 生成设计文档和制品，以便未来对已实现的解决方案进行支持。
- 调查和解决事件，并确定问题是由于数据加载代码还是来自数据提供方的错误数据导致。
- 执行与常规和临时数据加载相关的服务请求。
- 提供数据质量检查并报告数据质量问题。
技能、经验与技能要求
技术技能
10年以上经验：
· 设计和开发可扩展的Medallion数据湖架构。
· 精通使用Delta Lake和Databricks进行数据摄取、转换和整理。
· 具有将结构化和非结构化数据源集成到星型/雪花型模式的经验。
· 使用Azure Data Factory (ADF)、Databricks (PySpark、SQL、Delta Live Tables) 和 dbt 构建、自动化和优化复杂的ETL/ELT流水线。
· 在Azure环境中实现协调的工作流和作业调度。
· 熟悉关系型（SQL Server、Synapse、PostgreSQL）和维度建模。
· 高级SQL查询优化、索引、分区和数据复制策略。
· 具有在Azure Databricks中使用Apache Spark、Delta Lake和分布式计算框架的经验。
· 使用Parquet、ORC和JSON格式进行优化存储和检索。
· 深入了解Azure Data Lake Storage (ADLS)、Azure Synapse Analytics、Azure SQL、Event Hubs和Azure Functions。
· 对云安全、RBAC和数据治理有深刻理解。
· 精通Python（PySpark）、SQL和PowerShell用于数据工程工作流。
· 具有使用CI/CD自动化（Azure DevOps、GitHub Actions）用于数据流水线的经验。
· 实现数据血缘、目录管理、元数据管理和数据质量框架。
· 具有使用Unity Catalog在Databricks环境中管理权限的经验。
· 精通Power BI（DAX、数据建模、性能调优）。
· 具有将Power BI与Azure Synapse和Databricks SQL数据仓库集成的经验。
· 熟悉MLflow、AutoML，并在数据流水线中嵌入AI驱动的洞察。
50分
核心技能和经验
· 10年以上技术系统经验
查看英文原文
Responsibilities:
- Design technical solutions for data acquisition and storage into our centralized data repository.
- Develop ELT scripts, design data-driven logic and conduct unit testing.
- Conduct database modeling and design as to improve overall performance.
- Produce design artifacts and documentation which will allow future support of the implemented solutions.
- Investigate and resolve incidents and identify whether the problem is caused by the data loading code or is due to bad data received from the data provider.
- Execute service requests related to routine and ad-hoc data loads.
- Provide the data quality check and report on the data quality issue.
SkillsExperience and Skill Set Requirements
Technical Skills
10+ years experience in:
· Designing and developing scalable Medallion Data Lakehouse architectures.
· Expertise in data ingestion, transformation, and curation using Delta Lake and Databricks.
· Experience integrating structured and unstructured data sources into star/snowflake schemas.
· Building, automating, and optimizing complex ETL/ELT pipelines using Azure Data Factory (ADF), Databricks (PySpark, SQL, Delta Live Tables), and dbt.
· Implementing orchestrated workflows and job scheduling in Azure environments.
· Strong knowledge of relational (SQL Server, Synapse, PostgreSQL) and dimensional modeling.
· Advanced SQL query optimization, indexing, partitioning, and data replication strategies.
· Experience with Apache Spark, Delta Lake, and distributed computing frameworks in Azure Databricks.
· Working with Parquet, ORC, and JSON formats for optimized storage and retrieval.
· Deep expertise in Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure SQL, Event Hubs, and Azure Functions.
· Strong understanding of cloud security, RBAC, and data governance.
· Proficiency in Python (PySpark), SQL, and PowerShell for data engineering workflows.
· Experience with CI/CD automation (Azure DevOps, GitHub Actions) for data pipelines.
· Implementing data lineage, cataloging, metadata management, and data quality frameworks.
· Experience with Unity Catalog for managing permissions in Databricks environments.
· Expertise in Power BI (DAX, data modeling, performance tuning).
· Experience in integrating Power BI with Azure Synapse and Databricks SQL Warehouses.
· Familiarity with MLflow, AutoML, and embedding AI-driven insights into data pipelines.
50 points
Core Skill and Experience
· 10+ years of experience with technical systems specifications and translating them into working, tested applications for large, complex, mission critical applications.
· 10+ years of experience in technical analysis, program code, detailed programming and reports specifications, program design, writing and /or generating code, and conducting unit tests.
· 10+ years of experience in software in various computing platforms, operating systems, database technology, communication protocols, middleware and gateways.
· 10+ years of experience in developing and maintaining system design models, technical documentation, and specifications.
· 5+ years of experience in conducting technical evaluation and assessment of options for technical design issues, application figuration aspects and integration capabilities, related tools and utilities, gap analysis of integration components to technical requirements / specifications / documentation.
· SDLC end-to-end.
30 points
General Skills
· Demonstrated strong leadership and people management skills.
· Proven technical leadership skills with ability to identify areas for improvement, and recommend solutions.
· Exceptional analytical, problem solving and decision-making skills.
· Demonstrated strong interpersonal, verbal and written communication, and presentation skills.
· Proven troubleshooting and critical thinking experience
· Demonstrated ability to apply strong listening skills to facilitate issue resolution.
· Effective consulting skills to engage with all stakeholders with proven track record for building strong working relationships.
· Strong interpersonal, facilitation and negotiation skills with ability to build rapport with stakeholders and drive negotiations to a successful outcome.
· Excellent customer service skills, including tact and diplomacy to ensure client needs are managed effectively.
· A motivated, flexible, detail-oriented and creative team player with perseverance, excellent organization and multi-tasking abilities, and a proven track record for meeting strict deadlines.
15 points
Public Sector/Healthcare Experience
· Knowledge of Public Sector Enterprise Architecture artifacts (or similar), processes and practices, and ability to produce technical documentation that comply with industry standard practices.
· Knowledge of Project Management Institute (PMI) and Public Sector I&IT project management methodologies.
· Knowledge and understanding of Ministry policy and IT project approval processes and requirements.
· Experience with large complex IT Health-related projects.
5 points
MUST HAVES:
10+ years experience in:
· Designing and developing scalable Medallion Data Lakehouse architectures.
· Expertise in data ingestion, transformation, and curation using Delta Lake and Databricks.
· Experience integrating structured and unstructured data sources into star/snowflake schemas.
· Building, automating, and optimizing complex ETL/ELT pipelines using Azure Data Factory (ADF), Databricks (PySpark, SQL, Delta Live Tables), and dbt.
· Implementing orchestrated workflows and job scheduling in Azure environments.
· Strong knowledge of relational (SQL Server, Synapse, PostgreSQL) and dimensional modeling.
· Advanced SQL query optimization, indexing, partitioning, and data replication strategies.
· Experience with Apache Spark, Delta Lake, and distributed computing frameworks in Azure Databricks.
· Working with Parquet, ORC, and JSON formats for optimized storage and retrieval.
· Deep expertise in Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure SQL, Event Hubs, and Azure Functions.
· Strong understanding of cloud security, RBAC, and data governance.
· Proficiency in Python (PySpark), SQL, and PowerShell for data engineering workflows.
· Experience with CI/CD automation (Azure DevOps, GitHub Actions) for data pipelines.
· Implementing data lineage, cataloging, metadata management, and data quality frameworks.
· Experience with Unity Catalog for managing permissions in Databricks environments.
· Expertise in Power BI (DAX, data modeling, performance tuning).
· Experience in integrating Power BI with Azure Synapse and Databricks SQL Warehouses.
Originally posted on Himalayas