Technical Lead - Data Engineer (Data&AI)
我们正在寻找一个领先的数据工程师，他将实践多平台专业知识与数据架构，管道和 CI/CD 方面的强有力的领导力结合在一起。这个职位需要具有现代数据平台 （如 Databricks,Snowflake,AWS 和 Azure）的深入技术技能，了解 MLOps/DevOps 实践以及指导高性能团队在建立可扩展，生产式的数据解决方案方面的能力。你不会仅限于单个平台，而是会利用各种工具包来解决复杂的数据挑战。关键责任 ·管道 & 建筑：领导可扩展的 ETL/ELT 管道，数据模型和集成框架的实践开发来处理大量 （数十亿份记录）的结构化和非结构化的零售数据。·多平台工程：设计，开发和优化跨多平台的数据处理应用程序，包括 Databricks (Spark/Delta Lake),雪，AWS 或 Azure. ·数据集成与配套：使用 Apache Airflow 进行配套并管理稳健的数据管道，用于无的数据集成和移动。·数据处理：设计和实施对变化数据捕获 (CDC) ,大规模批量处理，低延迟实时/流媒体数据处理的强大的解决方案。·API 管理：广泛与外部 API 进行数据摄入工作，以及为下游应用程序和用户提供数据设计，创建和管理内部 REST API. ·AI 增强的交付工具：积极利用人工智能助理来概念化，设计和加速数据管道和日常工程任务的开发。·DevOps & CI / CD:拥有并发展 CI / CD 管道 （Git 工作流程，自动化，释放周期，测试秘密管理，文件）. 引导使用 Dockerized 应用，Kubernetes 编排和监控/记录工具 (Splunk, Datadog, Dynatrace) 的 DevOps 导向部署。·MLOps 配套：与数据科学家合作对 ML 项目的数据准备性，确保与 ML 生命周期阶段 （数据准备，功能工程，模型部署）的协调。·政府和领导力：建立并执行数据治理，数据质量，元数据和安全方面的最佳实践。通过同行评价，知识分享和技术领导方式指导团队成员。·创新：保持业趋势前沿 MLOps,可观察性和 GenAI,引入相关的做法和要求。
查看英文原文
Lead Data Engineer
Overview
We are looking for a Lead Data Engineer who combines hands-on multi-platform expertise with strong leadership in data architecture, pipelines, and CI/CD. This role requires a versatile engineer with deep technical skills across modern data platforms (such as Databricks, Snowflake, AWS, and Azure), an understanding of MLOps/DevOps practices, and the ability to guide a high-performing team in building scalable, production-ready data solutions. You will not be limited to a single platform but will leverage a diverse toolkit to solve complex data challenges.
Key Responsibilities
·Pipeline & Architecture: Lead hands-on development of scalable ETL/ELT pipelines, data models, and integration frameworks to process high-volume (billions of records) structured and unstructured retail data.
·Multi-Platform Engineering: Design, develop, and optimize data processing applications across multiple platforms, including Databricks (Spark/Delta Lake), Snowflake, AWS, or Azure.
·Data Integration & Orchestration: Build and manage robust data pipelines using Apache Airflow for orchestration and Airbyte for seamless data integration and movement.
·Data Processing: Architect and implement robust solutions for Change Data Capture (CDC), large-scale batch processing, and low-latency real-time/streaming data processing.
·API Management: Work extensively with external APIs for data ingestion, as well as design, create, and manage internal REST APIs to serve data to downstream applications and users.
·AI-Augmented Deliverables: Actively leverage AI assistants to conceptualize, design, and accelerate the development of data pipelines and everyday engineering tasks.
·DevOps & CI/CD: Own and evolve CI/CD pipelines (Git workflows, automated testing, release cycles, secrets management, documentation). Guide DevOps-oriented deployments utilizing Dockerized applications, Kubernetes orchestration, and monitoring/logging tools (Splunk, Datadog, Dynatrace).
·MLOps Alignment: Collaborate with Data Scientists on data readiness for ML projects and ensure alignment with ML lifecycle stages (data prep, feature engineering, model deployment).
·Governance & Leadership: Establish and enforce best practices in data governance, data quality, metadata, and security. Mentor team members through peer reviews, knowledge sharing, and technical leadership.
·Innovation: Stay ahead of industry trends in MLOps, observability, and GenAI, introducing relevant tools and practices.
Required Skills & Experience
·Experience: 5+ years of experience in Data Engineering.
·Data Lakes & Warehouses: Mandatory expertise in designing, building, and managing large-scale Data Warehouses and Data Lakes from the ground up.
·Data Processing Paradigms: Extensive, hands-on experience working with Change Data Capture (CDC) mechanisms, complex batch processing, and real-time/streaming data processing.
·Platform Expertise: Proven expertise in more than one major cloud data platform/ecosystem (e.g., Databricks, Snowflake, AWS Analytics, Azure Data Engineering).
·SQL Mastery: Advanced proficiency in writing, optimizing, and debugging complex SQL queries for large-scale data processing and analytics.
·Programming: Strong programming skills in Python (async, threading, decorators, advanced I/O).
·APIs: Strong proficiency in interacting with third-party APIs and hands-on experience creating and managing REST APIs (using frameworks like FastAPI, Flask, or similar).
·Tooling: Deep hands-on experience with workflow orchestration (Apache Airflow) and data integration platforms (Airbyte).
·AI-Assisted Engineering: Mandatory capability to use AI coding assistants and tools to design pipelines, write code, and enhance day-to-day productivity.
·Data Architecture: Experience with data modeling (e.g., Delta Lake or Snowflake architecture) and scalable ETL/ELT design.
·DevOps/CI/CD: Hands-on experience with Git-based CI/CD (GitLab preferred) and a working knowledge of Docker & Kubernetes for deployment and scaling.
·MLOps: Understanding of MLOps concepts including data preparation, model lifecycle, registries, and monitoring.
·Soft Skills: Strong problem-solving skills with the ability to design for scale and performance, coupled with excellent collaboration, communication, and leadership skills.
Good to Have
·Customer Data Platform (CDP): Experience working with, building, or implementing CDPs to unify customer data across systems.
·Experience in the retail domain or other large-scale data-heavy environments.
·Familiarity with streaming frameworks (Kafka, Spark Streaming, etc.).
·Agentic Pipeline Development: Experience or strong interest in building agentic pipelines using LLMs for dynamic data orchestration and automation.
·Knowledge of model observability tools and ML deployment pipelines.
·Exposure to GenAI concepts (vector embeddings, vector databases, RAG).
Originally posted on Himalayas