Data Engineer with IRS MBI Clearance
工作称号：数据工程师位置：远程。持续时间：全职。需要 IRSMBI 许可与 IRSGFE 职位概述：我们正在寻找一个经验丰富的数据工程师，以现代大数据技术设计，构建和维护可扩展的数据处理管道。理想候选人将拥有强大的开发技能和经验分布式计算框架。所需技能和资格：计算机科学，工程或相关技术领域学士学位 5年以上软件开发经验在 Python,Java 或 Scala 编程语言中强有能力；广泛使用 Apache Spark SQL,包括：· Spark SQL · 闪电流及其数据来源 KAFKA,CDC 源等 · 性能优化和调整 · 数据转换和处理 手掌经验与基于云的闪电平台 (Databricks, AWS EMR, AWS Glue) 有关蜂巢，团结目录，粘合目录有着强大的理解 工作经验与任何数据质量框架/工具 有关观察性和监测数据处理管道的工作经验 分布计算概念的强大了解版本控制系统的技能 (Github/GitLab/Bitbucket) 技術要求：构建和维护高容量的数据处理管线 数据建模和 ETL/ELT 最佳实践知识熟悉 SQL 和 NoSQL 数据库理解数据仓库概念和维度建模优先资格：实时数据处理和流媒体架构的经验 德尔塔湖或类似数据湖泊技术的知识 与 CI/CD 管道的经验云平台专业知识 (AWS/Azure/GCP) 对开源项目贡献关键责任：设计和实施可扩展的数据管道优化现有的数据工作流量以实现性能和可靠性与数据科学家和分析师合作支持他们的数据需求 执行数据质量检查和监测 保持数据流程和架构的文档软技能：强大的解决问题能力优秀沟通技巧团队协作能力独立工作能力 专注于细节理想候选人应具有生产级数据管道开发和维护的经验记录，并对大规模数据系统的工作有热情。原始地发布在喜马拉雅山
查看英文原文
Job Title: Data Engineer
Location: Remote.
Duration: Full-Time.
NEED IRS MBI Clearance with IRS GFE
Position Overview:
We are seeking an experienced Data Engineer to design, build, and maintain scalable data processing pipelines using modern big data technologies. The ideal candidate will have strong development skills and experience with distributed computing frameworks.
Required Skills & Qualifications:Bachelor's degree in Computer Science, Engineering, or related technical field
5+ years of software development experience
Strong proficiency in Python, Java or Scala programming languages
Extensive hands-on experience with Apache Spark, including:
- Spark SQL
- Spark Streaming and its data sources KAFKA, CDC feed, etc
- Performance optimization and tuning
- Data transformation and processing
Hands on Experience working with cloud-based Spark platforms (Databricks, AWS EMR, AWS Glue)
Strong understanding of Hive, Unity catalog, Glue catalog
Hands on Experience working with any data quality framework/tools
Hands on Experience working with Observability and monitoring for data processing pipelines
Strong understanding of distributed computing concepts
Proficiency in version control systems (Github/GitLab/Bitbucket)
Technical Requirements:
Experience building and maintaining high-volume data processing pipelines
Knowledge of data modeling and ETL / ELT best practices
Familiarity with SQL and NoSQL databases
Understanding of data warehouse concepts and dimensional modeling
Preferred Qualifications:
Experience with real-time data processing and streaming architectures
Knowledge of Delta Lake or similar data lakehouse technologies
Experience with CI/CD pipelines
Cloud platform expertise (AWS/Azure/GCP)
Contributions to open-source projects
Key Responsibilities:
Design and implement scalable data pipelines
Optimize existing data workflows for performance and reliability
Collaborate with data scientists and analysts to support their data needs
Implement data quality checks and monitoring
Maintain documentation for data processes and architectures
Soft Skills:
Strong problem-solving abilities
Excellent communication skills
Team collaboration
Ability to work independently
Strong attention to detail
The ideal candidate should have a proven track record of developing and maintaining production-grade data pipelines and be passionate about working with large-scale data systems.
Originally posted on Himalayas