数据工程师 - 中级和高级水平

This is a remote position.
Role Overview
We are looking for Data Engineers at Senior and Mid-Level to join our team in building a privacy-preserving data platform where data engineering meets production-grade software engineering.
You will work on designing, developing, and maintaining reliable data pipelines that transform operational data into high-quality, secure, and AI-ready datasets.
Key Responsibilities
· Build and maintain production-grade data pipelines on AWS.

• Extract, transform, validate, and curate large-scale Parquet datasets.

• Implement data de-identification, masking, and privacy-preserving transformations.

• Design and maintain data pipeline orchestration, scheduling, retries, and backfill mechanisms.

• Implement comprehensive data quality checks, monitoring, and alerting.

• Work with workflow orchestration tools such as Airflow, Dagster, or AWS Step Functions.

• Contribute to CI/CD pipelines and Infrastructure as Code (IaC) practices.

• Manage schema evolution and schema drift across data sources and pipelines.

• Provide production support, troubleshooting, and root cause analysis for data pipeline issues.

• Maintain data catalogs, metadata, and data lineage.

• Follow software engineering best practices including Git, code reviews, automated testing, and maintainable code.

• Build reliable and idempotent data pipelines capable of handling retries and large-scale backfills.

Required Skills & Experience
· Strong proficiency in Python and SQL.

• Hands-on experience with AWS data services and production data pipelines.

• Experience with Apache Spark or equivalent distributed data processing technologies.

• Practical experience with Airflow, Dagster, AWS Step Functions, or similar orchestration tools.

• Strong understanding of data pipeline architecture, ETL/ELT, and data transformation.

• Experience working with Parquet and large-scale datasets.

• Understanding of data quality, schema management, monitoring, and alerting.

• Strong software engineering practices including:
• Git and version control

• Code reviews

• Automated testing

• Idempotency

• Error handling

• Retries and backfills

• Experience supporting and troubleshooting production data pipelines.

• Ability to work effectively with cross-functional engineering and data teams.

Requirements
Required Skills
Python | SQL | AWS | Data Engineering | Data Pipelines | ETL/ELT | Apache Spark | Parquet | Airflow | Dagster | AWS Step Functions | Data Orchestration | Data Quality | Schema Management | Data Transformation | Production Support | Git | CI/CD | Automated Testing | Data De-identification | Data Lineage | Data Catalog
Good to Have
Debezium | AWS DMS | Apache Iceberg | Delta Lake | Apache Hudi | Data Masking | Data Tokenization | Terraform | CloudFormation | ML/AI Training Data | Privacy-Preserving Data

Originally posted on Himalayas