Databricks 数据工程师
Databricks Data Engineer
职责
Databricks 数据工程师将负责政府拥有的 Databricks 工作区的实操搭建，以及整合和接入数据的工作，以整合机构业务系统数据。这包括从核心企业系统设计和实现数据管道，实施 Unity Catalog 进行数据治理，配置 MLflow 用于机器学习工作流，并开发全面的文档以支持试点后的持续维护。
此职位为 100% 远程。
其他职责包括：
- 在现有 Databricks 环境中搭建和配置专用的政府工作区
- 从核心机构业务系统（包括财务、人力资源、客户关系管理、IT 服务管理）设计和实现数据采集管道
- 在源系统支持的情况下，利用原生/内置连接器；为遗留系统设计自定义集成方案
- 在 Databricks 中对采集的数据进行标准化和准备，供下游可视化/报告工具使用
- 实施 Databricks Unity Catalog 实现集中式数据治理、元数据管理、主动审计和端到端血缘追踪
- 审查当前配置，评估 CUI/PII/PHI/财务数据的安全控制措施，并实施改进
- 开发全面的“实际构建”文档，包括物理/逻辑架构图、自动化数据字典和标准操作程序
- 记录数据源、集成方法和数据湖架构决策，以支持持续维护
资格要求
必备资格：
- 本科/学士学位至少 8 年，硕士/硕士至少 6 年，博士至少 3 年
- 美国公民身份
- 有效的国防部机密级安全许可
- 至少 2 年在 Databricks 平台上工作的经验
- 精通 PySpark、Spark SQL 和 Python，用于大规模数据处理和管道开发
- 具有 Unity Catalog 管理的实际经验，包括元存储管理、访问策略和数据血缘
- 具有管道编排工具（Airflow、Databricks Workflows）的经验
- 具有 Delta Lake、模式演变、时间旅行和优化技术的经验
- 具有整合异构企业系统的经验，包括遗留系统/定制集成
- 熟悉云平台（Azure/AWS）和基础设施即代码实践
- 了解数据治理原则、合规框架和 CUI/PII 处理实践
- 熟悉财务、人力资源、客户关系管理或 IT 服务管理的数据结构
查看英文原文
Responsibilities
The Databricks Data Engineer will be responsible for the hands-on build-out of a Government-owned Databricks workspace and the data ingestion/integration work needed to consolidate agency business system data. This includes designing and implementing data pipelines from core enterprise systems, implementing Unity Catalog for data governance, configuring MLflow for machine learning workflows, and developing comprehensive documentation to support sustainment beyond the pilot.
This role is 100% Remote.
Other Responsibilities Include:
- Build out and configure a dedicated Government workspace within the existing Databricks environment
- Design and implement data ingestion pipelines from core agency business systems including financial, HR, CRM, and ITSM systems
- Leverage native/built-in connectors where source systems support them; design custom integration approaches for legacy systems
- Normalize and prepare ingested data within Databricks for consumption by downstream visualization/reporting tools
- Implement Databricks Unity Catalog for centralized data governance, metadata management, active auditing, and end-to-end lineage tracking
- Review current configuration, assess security controls for CUI/PII/PHI/financial data, and implement improvements
- Develop comprehensive "as-built" documentation including physical/logical architecture diagrams, automated data dictionaries, and SOPs
- Document data sources, integration methods, and data lake architecture decisions to support sustainment
Qualifications
Required Qualifications:
- Min 8 years with BS/BA, Min 6 years with MS/MA, Min 3 years with PhD
- US Citizenship
- Active DoD Secret clearance
- 2 years working on the Databricks platform
- Strong proficiency in PySpark, Spark SQL, and Python for large-scale data processing and pipeline development
- Hands-on experience with Unity Catalog administration, including metastore management, access policies, and data lineage
- Experience with pipeline orchestration tools (Airflow, Databricks Workflows)
- Experience with Delta Lake, schema evolution, time travel, and optimization techniques
- Experience integrating heterogeneous enterprise systems, including legacy/custom integrations
- Familiarity with cloud platforms (Azure/AWS) and infrastructure-as-code practices
- Understanding of data governance principles, compliance frameworks, and CUI/PII handling practices
- Familiarity with financial, HR, CRM, or ITSM data structures
- Git/CI-CD pipeline experience
- DoD 8570 certification
Preferred Qualifications:
· Databricks Certified Data Engineer Associate or Professional certification
Peraton Overview
Peraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the world’s leading mission capability integrator and transformative enterprise IT provider, we deliver trusted, highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land, sea, space, air, and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Each day, our employees do the can’t be done by solving the most daunting challenges facing our customers. Visit peraton.com to learn how we’re keeping people around the world safe and secure.
Target Salary Range
$112,000 - $179,000. This represents the typical salary range for this position. Salary is determined by various factors, including but not limited to, the scope and responsibilities of the position, the individual’s experience, education, knowledge, skills, and competencies, as well as geographic location and business and contract considerations. Depending on the position, employees may be eligible for overtime, shift differential, and a discretionary bonus in addition to base pay.EEO
EEO: Equal opportunity employer, including disability and protected veterans, or other characteristics protected by law.Originally posted on Himalayas