高级数据工程师（印度）
Senior Data Engineer (India)
首席数据工程师将设计、构建并实现可扩展的数据解决方案，以支持企业分析和AI/ML计划。该职位需要精通Databricks、Azure Fabric、PySpark、SQL和Azure生态系统，具备在数据仓库、数据湖和实时集成方面的丰富经验。首席数据工程师将使用行业标准工具构建端到端的管道，推动自动化，并有效地将解决方案投入生产。该职位需确保符合数据治理要求（包括GxP和HIPAA/GDPR），同时构建可重用、集成的管道和分析模型，以促进自助式分析。该职位为团队提供技术领导力，指导初级工程师，并与业务利益相关者合作，确保数据工程与组织目标一致。
职位描述
- 数据架构与工程
- 使用Azure Databricks、PySpark、Azure Data Factory和Azure SQL，设计、构建和实现端到端的数据解决方案。
- 为特定用例从数据源通过集成到消费设计、构建和维护数据管道。
- 在数据湖的青铜层、银层和金层中实施稳健的数据建模标准。
- 根据需要开发数据模型（概念型、逻辑型和/或物理型）。
- 优化Spark和SQL工作负载，以提高性能、可扩展性和成本效率。
- 使用数据准备、集成和AI驱动的工具和技术管理元数据。
- 数据集成与自动化
- 推动数据集成的自动化；推荐并领导实施自动化重复数据准备和集成任务的技术。
- 构建基于API的集成（REST/JSON）和实时摄入框架。
- 使用Azure DevOps管道和基于Git的CI/CD实践自动化数据工作流。
- 实现参数化、可重用的管道模板用于摄入和转换。
- 为数据作业开发自动化的单元测试、回归测试和集成测试框架。
- 分析与数据赋能
- 准备和整理高质量的数据集用于BI、报告和高级分析。
- 与使用Power BI、Tableau或类似平台的分析团队合作，定义语义模型和KPI。
- 实现性能优化的数据模型以支持自助式分析。
- 偶尔为终端用户提供数据可视化解决方案的使用支持。
利益相关者参与与沟通
查看英文原文
The Lead Data Engineer will design, build, and operationalize scalable data solutions to support enterprise analytics and AI/ML initiatives. This role requires expert-level proficiency in Databricks, Azure Fabric, PySpark, SQL, and the Azure ecosystem, with deep experience across data warehouses, data lakes, and real-time integration. The Lead Data Engineer will architect end-to-end pipelines using industry-standard tools, drive automation, and move solutions effectively into production. The incumbent will ensure compliance with data governance requirements (including GxP and HIPAA/GDPR) while building reusable, integrated pipelines and analytical models that promote self-service analytics. This role provides technical leadership across the team, mentors junior engineers, and partners with business stakeholders to align data engineering with organizational objectives.
About the Role
. Data Architecture & Engineering
- Architect, design, and implement end-to-end data solutions using Azure Databricks, PySpark, Azure Data Factory, and Azure SQL.
- Design, build, and maintain data pipelines from data sources through integration to consumption for specific use cases.
- Implement robust data modeling standards across bronze, silver, and gold layers in the data lake.
- Develop data models (conceptual, logical, and/or physical) as required.
- Optimize Spark and SQL workloads for performance, scalability, and cost efficiency.
- Manage metadata using data preparation, integration, and AI-enabled tools and techniques.
. Data Integration & Automation
- Drive automation in data integration; recommend and lead implementation of techniques to automate repeatable data preparation and integration tasks.
- Build API-based integrations (REST/JSON) and real-time ingestion frameworks.
- Automate data workflows using Azure DevOps pipelines and Git-based CI/CD practices.
- Implement parameterized, reusable pipeline templates for ingestion and transformation.
- Develop automated unit, regression, and integration testing frameworks for data jobs.
. Analytics & Data Enablement
- Prepare and curate high-quality datasets for BI, reporting, and advanced analytics.
- Partner with analytics teams using Power BI, Tableau, or similar platforms to define semantic models and KPIs.
- Implement performance-optimized data models for self-service analytics.
- Will occasionally provide support to end users on the use of data visualization solutions.
Stakeholder Engagement & Leadership
- Lead technical design reviews, mentor junior engineers, and promote best practices.
- Assist cross-functional groups, business analysts, and stakeholders to gather, define, and refine data requirements.
- Collaborate with business and IT stakeholders to align data engineering with organizational objectives.
- Propose innovative data ingestion, preparation, and integration techniques to address stakeholder requirements.
- Contribute to architectural roadmaps and technology evaluations for the data platform.
- In collaboration with functional leaders, identify inefficiencies and recommend improvements to the executive team.
About You
Job Experience & Education Requirements:
Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field (Master’s preferred)
And
5–8 years of experience designing and developing enterprise-scale data solutions (data warehouses, data lakes, operational databases)
Other:
- Expert-level proficiency in Databricks, Azure Fabric, PySpark, SQL, and Azure DevOps.
- Proven experience with Azure Data Factory, ADLS Gen2, and Azure SQL Server.
- Strong experience with Microsoft Azure data management architectures including Data Warehouse, Data Lake, and Data Catalogue, and supporting processes such as Data Integration, Governance, and Metadata Management.
- Experience with Power BI required; Tableau or Looker a plus.
- Working knowledge of CI/CD automation, version control (Git), and infrastructure as code (ARM, Bicep, or Terraform).
- Experience in life sciences or healthcare industries is a strong plus.
- Good understanding of GxP, GDPR/HIPAA, and applicable CFR/CTR/CTD regulations.
- Demonstrated success working with both IT and business stakeholders while integrating analytics and data science output into business processes and workflows.
- Must have excellent written and verbal communication skills.
- Proven ability to work independently and as part of a team and meet important deadlines.
- Statistical analysis skills are an asset.
Originally posted on Himalayas