大数据开发工程师
Big Data Developer
L3 大数据开发工程师（7-10年经验）
1. 使用 Hadoop 生态系统技术如 HBase、Hive、Kudu 和 Spark 设计、开发和实现高可扩展性和分布式大数据解决方案。
2. 架构 HBase 模式和数据模型以满足不断变化的业务需求，并确保数据存储和检索操作的最优性能。
3. 开发复杂的 Hive 查询和数据处理流程，将原始数据转换为适合分析和报告的结构化格式。
4. 使用 Spark Streaming 和 Spark SQL 实现数据摄入流程，对流数据源进行实时处理，确保高吞吐量和低延迟。
5. 优化 Spark 应用程序的性能和资源利用率，包括调整 RDD 转换、优化数据分区策略以及利用内存缓存。
6. 利用 Spark MLlib 的高级功能进行机器学习任务，如分类、回归、聚类和协同过滤。
7. 设计和部署 Kudu 表以支持快速分析查询和实时分析，利用 Kudu 快速分析和快速数据摄入的独特组合。
8. 与数据科学家合作，将机器学习模型集成到 Spark 工作流中，并将其产品化用于实时预测和分析。
9. 排查大数据应用程序和基础设施中的性能瓶颈、数据质量问题和系统故障，并实施解决方案加以解决。
10. 跟踪大数据处理和分析领域的新技术和最佳实践，并评估其对我们架构和解决方案的潜在影响。
最初发布于 Himalayas
查看英文原文
L3 Big datadeveloper ( 7-10 Years)
1. Design, develop, and implement highly scalable and distributed big datasolutions using Hadoop ecosystem technologies such as HBase, Hive, Kudu, andSpark.
2. Architect HBase schemas and data models to accommodate evolving businessrequirements and ensure optimal performance for data storage and retrievaloperations.
3. Develop complex Hive queries and data processing pipelines to transform rawdata into structured formats suitable for analysis and reporting.
4. Implement data ingestion pipelines using Spark Streaming and Spark SQL forreal-time processing of streaming data sources, ensuring high throughput andlow latency.
5. Optimize Spark applications for performance and resource utilization,including tuning RDD transformations, optimizing data partitioning strategies,and leveraging in-memory caching.
6. Utilize advanced features of Spark MLlib for machine learning tasks such asclassification, regression, clustering, and collaborative filtering.
7. Design and deploy Kudu tables for fast analytical queries and real-timeanalytics, leveraging Kudus unique combination of fast analytics and fast dataingestion.
8. Collaborate with data scientists to integrate machine learning models intoSpark workflows and productionize them for real-time predictions and analytics.
9. Troubleshoot performance bottlenecks, data quality issues, and systemfailures in big data applications and infrastructure, and implement solutionsto address them.
10. Stay abreast of emerging technologies and best practices in big data processingand analytics, and evaluate their potential impact on our architecture andsolutions.
Originally posted on Himalayas