Data Engineer
工作简介：作为数据工程师，你将负责设计，实施和维护可扩展的数据管道和系统来支持数据处理，分析和视觉化。你将与跨功能团队密切合作了解数据需求，优化数据工作流程，确保数据质量和可靠性。责任：1.数据管道开发：设计，构建和优化数据管道以从各种来源中提取，转换和加载 (ETL) 数据到数据仓库或数据湖，确保可扩展性，可靠性和效率。2.数据模型：开发和维护数据模型和方案，以支持业务分析，报告和机器学习应用程序，确保数据完整性，一致性和性能。3. 数据集成：从不同来源的数据集成，包括数据库，API,流媒体平台和第三方系统，为分析和报告目的提供统一视图。4. 数据质量保证：实施数据质量检查和监测过程来识别并纠正数据异常，不一致性和错误，确保高数据质量和可靠性。5. 绩效优化：6. 基础设施管理：管理和维护数据基础设施组件，包括数据库，数据仓库，ETL 管道和数据处理集群，确保可用性，扩展性和安全。7. 合作与文档：与数据分析师，数据科学家和其他利益相关者合作了解数据需求并提供满足业务需求的解决方案。文件数据管道，工作流程和系统配置以共享知识和参考未来。要求：·计算机科学，工程或相关领域的学士学位。优先获得士学位。· 经验证明4年作为数据工程师或类似职位，具有强有力的设计和实施数据管道和系统的记录。· 熟练掌握 Python,Java 或 Scala 等编程语言以及使用 Apache Spark,Apache Flink 或 Hadoop 等数据处理框架的经验。· 认真了解 SQL 和 NoSQL 数据库 （例如 PostgreSQL,MySQL,MongoDB,Cassandra）等数据库技术。·
查看英文原文
Job Description:
As a Data Engineer, you will be responsible for designing, implementing, and maintaining scalable data pipelines and systems to support data processing, analysis, and visualization. You will collaborate closely with cross-functional teams to understand data requirements, optimize data workflows, and ensure data quality and reliability.
Responsibilities:
1. Data Pipeline Development: Design, build, and optimize data pipelines to extract, transform, and load (ETL) data from various sources into data warehouses or data lakes, ensuring scalability, reliability, and efficiency.
2. Data Modeling: Develop and maintain data models and schemas to support business analytics, reporting, and machine learning applications, ensuring data integrity, consistency, and performance.
3. Data Integration: Integrate data from disparate sources, including databases, APIs, streaming platforms, and third-party systems, to provide unified views of data for analysis and reporting purposes.
4. Data Quality Assurance: Implement data quality checks and monitoring processes to identify and rectify data anomalies, inconsistencies, and errors, ensuring high data quality and reliability.
5. Performance Optimization: Optimize data processing and query performance through indexing, partitioning, and caching techniques, leveraging distributed computing frameworks and database technologies.
6. Infrastructure Management: Manage and maintain data infrastructure components, including databases, data warehouses, ETL pipelines, and data processing clusters, ensuring availability, scalability, and security.
7. Collaboration and Documentation: Collaborate with data analysts, data scientists, and other stakeholders to understand data requirements and deliver solutions that meet business needs. Document data pipelines, workflows, and system configurations for knowledge sharing and future reference.
Requirements:
- Bachelor's degree in Computer Science, Engineering, or related field. Master's degree preferred.
- Proven experience 4 years as a Data Engineer or similar role, with a strong track record of designing and implementing data pipelines and systems.
- Proficiency in programming languages such as Python, Java, or Scala, and experience with data processing frameworks such as Apache Spark, Apache Flink, or Hadoop.
- Solid understanding of database technologies, including SQL and NoSQL databases (e.g., PostgreSQL, MySQL, MongoDB, Cassandra).
- Familiarity with cloud platforms such as AWS, Azure, or GCP, and experience with cloud-based data services (e.g., AWS Redshift, Azure SQL Data Warehouse, Google BigQuery).
- Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues and optimize data workflows for performance and scalability.
- Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to non-technical stakeholders.
Originally posted on Himalayas