数据工程师 - 高级
Data Engineer - Senior
主导数据和分析平台的设计、开发、部署和维护。开发可靠、可扩展且高效的数据显示管道和数据处理解决方案，使数据能够被有效处理、存储、管理，并提供给分析师和其他数据使用者。与业务利益相关者、IT专家、数据科学家、架构师和领域专家合作，交付符合业务和技术需求的企业级数据和分析解决方案。
关键职责
· 使用来自关系型、事件驱动型、半结构化和非结构化来源的数据，设计、开发并自动化分布式数据采集和转换解决方案。
- 使用适当的工具、技术和脚本语言构建可靠、可扩展且高效的ETL/ELT数据管道。
- 设计并实现数据质量、验证、监控和警报框架，以识别和解决数据完整性问题。
- 实施涵盖元数据、数据访问、保留、合规性和安全性的数据治理实践。
- 设计并实现物理数据模型，包括数据库结构、索引和表关系，以支持性能和可扩展性。
- 在云和分布式数据平台上开发和运营大规模数据存储和处理解决方案，包括数据湖、数据仓库和湖仓一体环境。
- 优化数据管道、Spark工作负载、数据库和云基础设施的性能、可靠性、可扩展性和成本效率。
- 集成来自各种企业应用和源系统的数据，并支持实时和事件驱动的数据处理。
- 开发常见且重复的数据准备、集成、部署和平台管理活动的自动化，以减少人工和易出错的流程。
- 实施CI/CD和DevOps实践，以支持自动化部署、测试和发布管理。
- 参与数据管道和平台解决方案的故障排除、测试、验证和持续改进。
- 确保数据平台和解决方案符合相关的质量、治理、安全、合规和监管要求。
- 与数据科学家、分析师、架构师、IT团队和业务利益相关者协作，理解需求并交付有效的数据解决方案。
- 文档化数据解决方案、流程、设计和技术信息，以支持知识传递和操作效率。
查看英文原文
Leads the design, development, deployment, and maintenance of data and analytics platforms. Develops reliable, scalable, and efficient data pipelines and data processing solutions that enable data to be effectively processed, stored, governed, and made available to analysts and other data consumers. Collaborates with business stakeholders, IT experts, data scientists, architects, and subject-matter experts to deliver enterprise data and analytics solutions aligned with business and technical requirements.
Key Responsibilities
· Design, develop, and automate distributed data ingestion and transformation solutions using data from relational, event-based, semi-structured, and unstructured sources.
- Build reliable, scalable, and efficient ETL/ELT data pipelines using appropriate tools, technologies, and scripting languages.
- Design and implement data quality, validation, monitoring, and alerting frameworks to identify and resolve data integrity issues.
- Implement data governance practices covering metadata, data access, retention, compliance, and security.
- Design and implement physical data models, including database structures, indexing, and table relationships, to support performance and scalability.
- Develop and operate large-scale data storage and processing solutions across cloud and distributed data platforms, including data lakes, warehouses, and lakehouse environments.
- Optimize data pipelines, Spark workloads, databases, and cloud infrastructure for performance, reliability, scalability, and cost efficiency.
- Integrate data from a variety of enterprise applications and source systems and support real-time and event-driven data processing.
- Develop automation for common and repeatable data preparation, integration, deployment, and platform-management activities to minimize manual and error-prone processes.
- Implement CI/CD and DevOps practices to support automated deployment, testing, and release management.
- Participate in troubleshooting, testing, validation, and continuous improvement of data pipelines and platform solutions.
- Ensure data platforms and solutions meet applicable quality, governance, security, compliance, and regulatory requirements.
- Collaborate with data scientists, analysts, architects, IT teams, and business stakeholders to understand requirements and deliver effective data solutions.
- Document data solutions, processes, designs, and technical information to support knowledge transfer and operational effectiveness.
- Apply Agile development methodologies such as Scrum and Kanban to deliver data engineering initiatives.
- Provide technical leadership and mentor less experienced team members, promoting engineering excellence and collaboration.
Originally posted on Himalayas