Senior Data Engineer
加入一个数据基础设施和分析团队，建立了一个平台，将来自许多来源的信息转化为可靠的当前公司智能。你将拥有这个平台从架构到摄入到生产运营，帮助推进下游产品使用可信赖的数据。你会做什么？- 设计和建造一个多源数据平台用于同时摄入，事件驱动处理和配套批量工作流程。- 从提供商 API,Webhooks,文件和网络扫描中摄取数据，以限制速度和强大的处理失败。- 使用 Airflow,Dagster 或 Prefect 等工具来编辑计划的工作流程和历史后填补。- 建立回复，死字母队列和控制重播的消息和任务排行工作流程。- 在 Google 云存储中保存原始有效载荷并将 dbt 数据转化为原料，标准化和策划层。- 开发无效的升级，数据契约，自动质量检查，管道测试以及新鲜性和故障监控。- 为可靠下游使用设计和优化 PostgreSQL 方案和加载模式。- 将数据丰富和基于 LLM 的结构式提取集成到平台中。我们正在寻找什么？- 5至10年或以上的经验，包括从架构到部署和持续运营的多源数据平台生产所有权。- 强大的实践经验与 Airflow 或类似编排工具，dbt 和 Google 云平台。- 用消息或任务队列以及异步处理建立事件驱动系统的经验。- 具有强大的 SQL 和 PostgreSQL 技能，包括数据建模，查询优化和增量加载。- Python 和同步或异步处理的熟练能力，如 asyncio,aiohttp 或 HTTPX. - 使用 Scrapy,Playwright 或 Selenium 等工具进行实际网络扫描经验。- 经验数据契约，存储层和自动化数据质量检查。实体分辨率是有价值的；大 Query,数据形式和基于 LLM 的结构性提取有帮助。- 在一个初创公司或小型团队中工作的舒适性与广泛的所有权和可适应能力。补偿和福利签证赞助商不提供。位置远程。
查看英文原文
ABOUT THE ROLE
Join a data infrastructure and analytics team building a platform that turns information from many sources into reliable, current company intelligence. You will own the platform end to end, from architecture and ingestion through production operations, helping power downstream products with trusted data.
WHAT YOU'LL DO
- Design and build a multi-source data platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
- Ingest data from provider APIs, webhooks, files, and web scraping, with rate limiting and robust failure handling.
- Orchestrate scheduled workflows and historical backfills using tools such as Airflow, Dagster, or Prefect.
- Build messaging and task queue workflows with retries, dead-letter queues, and controlled replay.
- Preserve original payloads in Google Cloud Storage and transform data with dbt into raw, standardized, and curated layers.
- Develop idempotent upserts, data contracts, automated quality checks, pipeline tests, and monitoring for freshness and failures.
- Design and optimize PostgreSQL schemas and loading patterns for reliable downstream use.
- Integrate data enrichment and LLM-based structured extraction into the platform.
WHAT WE'RE LOOKING FOR
- 5 to 10 or more years of experience, including production ownership of multi-source data platforms from architecture through deployment and ongoing operation.
- Strong hands-on experience with Airflow or similar orchestration tools, dbt, and Google Cloud Platform.
- Experience building event-driven systems with messaging or task queues and asynchronous processing.
- Strong SQL and PostgreSQL skills, including data modeling, query optimization, and incremental loading.
- Proficiency in Python and concurrent or asynchronous processing, such as asyncio, aiohttp, or HTTPX.
- Practical web scraping experience with tools such as Scrapy, Playwright, or Selenium.
- Experience with data contracts, storage layers, and automated data quality checks. Entity resolution is valuable; BigQuery, Dataform, and LLM-based structured extraction are helpful.
- Comfort working in a startup or small team with broad ownership and adaptability.
COMPENSATION & BENEFITS
Visa sponsorship is not available.
LOCATION
Remote.