Senior Data Engineer
角色：我们正在寻找一名高级数据工程师，从架构到生产，可靠地将大量的数据从供应商 API,Webhooks,文件和网页缩转化为我们的产品的可信托记录。今天，我们的丰富和提取工作流程主要在容器内进行序列运行。你将领导向支持并发处理，事件驱动接入和配套批量工作流程的平台迈进。责任：·多源接入：建立提供商 API,webhooks,档案和抓获的同时摄入，以率限制和处理失败。·批量和事件驱动执行：编程计划的工作流程和历史后填充，使用 Airflow,Dagster 或 Prefect 等工具。用 Pub/Sub 或云任务创建事件驱动的输入，包括重复试验，死字队列以及未完成工作的控制重播。· 原始到关闭数据：在 Google Cloud Storage (GCS) 中保存原来的有效载荷，包括提供者 JSON,然后将其转换为 dbt.建立对原始，标准化和关闭数据的奖牌层。·通过恢复进行正确性：实现无异性的安全升级和反复试验；重播和历史后补充以避免重复记录或数据腐败。使数据可追溯到源头并可返回数据质量和质量。· 数据：定义数据契约，建立自动验证和管道测试，监控新鲜性和故障。设计和优化 PostgreSQL 方案，查询和装载模式，以便下游产品获得可靠的数据。· 提取和丰富：集成供应商丰富和基于 LLM 结构化的文本从平台中提取。要求：· 从初始架构和管道实施到部署和持续运行，从零到一组建立多源生产数据平台的经验。· 使用 Airflow,Dagster,Prefect 或 Temporal 等工作流配套工具进行制作经验。经验：• 经验在数据流程中进行操作。· 与 dbt 和 GCP 的实践生产经验。· 经验建立以事件驱动系统，包括异步处理和故障恢复等信息或任务队列。· 强大的 SQL 和 PostgreSQL 技能，包括数据建模，查询优化和可靠增量加载。有了数据契约，原始和精选存储，保留政策以及自动质量检查的经验。
查看英文原文
The Role:
We’re looking for a senior data engineer to own our data platform from architecture through production, reliably turning high volumes of data from provider APIs, webhooks, files, and web scraping into trusted records for our products.
Today, our enrichment and extraction workflows largely run serially inside a container. You’ll lead the move to a platform that supports concurrent processing, event-driven intake, and orchestrated batch workflows.
Responsibilities:
- Multi-source intake: Build concurrent ingestion for provider APIs, webhooks, files, and scraping, with rate limits and failure handling.
- Batch and event-driven execution: Orchestrate scheduled workflows and historical backfills using tools such as Airflow, Dagster, or Prefect. Build event-driven intake with Pub/Sub or Cloud Tasks, including retries, dead-letter queues, and controlled replay of failed work.
- Raw-to-curated data: Preserve original payloads, including provider JSON, in Google Cloud Storage (GCS) before transforming them with dbt. Establish medallion layers for raw, standardized, and curated data.
- Correctness through recovery: Implement idempotent upserts and safe retries, replay, and historical backfills that avoid duplicate records or data corruption. Make data traceable to its source and safe to reprocess.
- Data quality and delivery: Define data contracts, build automated validation and pipeline tests, and monitor freshness and failures. Design and optimize PostgreSQL schemas, queries, and loading patterns so downstream products receive dependable data.
- Extraction and enrichment: Integrate provider enrichment and LLM-based structured extraction from text into the platform.
Requirements:
- Experience building multi-source production data platforms from zero to one, from initial architecture and pipeline implementation through deployment and ongoing operation.
- Production experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal, including dependencies, retries, and failure recovery. Airflow experience is preferred.
- Hands-on production experience with dbt and GCP.
- Experience building event-driven systems with messaging or task queues, including asynchronous processing and failure recovery.
- Strong SQL and PostgreSQL skills, including data modeling, query optimization, and reliable incremental loading. Experience with data contracts, raw and curated storage, retention policies, and automated quality checks.
- Strong Python skills, including asynchronous HTTP and concurrent processing with tools such as asyncio, aiohttp, or HTTPX.
- Hands-on web scraping experience with tools such as Scrapy, Playwright, Selenium, or Beautiful Soup, including concurrent scraping, rate limits, and failure handling.
Nice-to-Have:
- Experience working in startups or small engineering teams, with broad ownership and changing priorities.
- Entity resolution experience is especially valuable: matching company and people records across sources and reconciling duplicates into consistent entities.
- LLM-based structured extraction or company and people data enrichment.
- Familiarity with BigQuery and Dataform.
Originally posted on Himalayas