数据抓取 1743
Data Scraping 1743
这是一个远程职位。
我们正在寻找一位数据抓取工程师，帮助从公共和政府来源收集、整理并标准化数据，以统一的结构化格式呈现。该职位专注于解决复杂的数据采集挑战，研究不熟悉的来源，从网站和数据源中提取信息，并将其转换为可被下游系统使用的预定义格式。理想的候选人喜欢处理杂乱的数据集，研究网站和数据源的结构，并创建可重复使用且能持续产出一致结果的解决方案。该职位需要强大的问题解决能力、注重细节，以及独立工作的能力，同时能够清晰地记录发现和流程。
时间安排：周一至周五 - 中午12:00 – 晚上8:00 CST
职责：
研究并识别公共和政府数据来源。从网站、API、数据流和在线存储库中提取并标准化数据。构建可重复使用、可维护且可重新运行的脚本和抓取流程。以预定义格式交付结构化输出。在处理更大数据集之前提供样本输出供审核。记录数据来源、提取方法、遇到的挑战以及重新运行的流程。在提取过程中捕获并报告任何相关发现的信息，包括不一致之处、修订内容、生效日期、废止说明或相关元数据。排查数据采集问题并在需要时提出替代方案。通过定期检查和书面沟通与相关方协作。维护版本控制的代码仓库并遵循标准开发实践。
要求
具备网页抓取和数据提取的丰富经验。具有使用Python或其他类似脚本语言的实际编程经验。有处理HTML解析、API、HTTP请求、FTP来源以及结构化或非结构化数据的经验。能够评估、调试和改进抓取解决方案。具备较强的分析和解决问题的能力。有构建可重复使用自动化流程的经验，而非一次性脚本。熟悉关系型数据库（优先考虑PostgreSQL）并熟悉标准的Git工作流程。具备良好的文档编写和沟通能力。能够独立工作并承担技术挑战的责任。对细节高度关注并致力于数据准确性。加分项：有与政府、监管、合规或公众数据相关的工作经验。
查看英文原文
This is a remote position.
We are seeking a Data Scraping to help collect, organize, and normalize data from public and government sources into a consistent, structured format. This role focuses on solving complex data acquisition challenges, researching unfamiliar sources, extracting information from websites and feeds, and transforming it into predefined formats that can be consumed by downstream systems. The ideal candidate enjoys working with messy datasets, investigating how websites and data sources are structured, and creating reusable solutions that can be executed repeatedly with consistent results. This position requires strong problem-solving skills, attention to detail, and the ability to work independently while documenting findings and processes clearly.
Schedule: Monday to Friday - 12:00 PM – 8:00 PM CST
Responsibilities:
Research and identify public and government data sources. Extract and normalize data from websites, APIs, feeds, and online repositories. Build reusable, maintainable, and re-runnable scripts and scraping workflows. Deliver structured outputs in predefined formats. Provide sample outputs for review before processing larger datasets. Document data sources, extraction methodologies, challenges encountered, and re-run procedures. Capture and report any relevant information discovered during extraction, including inconsistencies, amendments, effective dates, repeal notes, or related metadata. Troubleshoot data acquisition issues and propose alternative approaches when needed. Collaborate with stakeholders through regular check-ins and written communication. Maintain version-controlled code repositories and follow standard development practices.
Requisitos
Strong experience with web scraping and data extraction. Practical programming experience using Python or similar scripting languages. Experience working with HTML parsing, APIs, HTTP requests, FTP sources, and structured or unstructured data. Ability to evaluate, debug, and improve scraping solutions. Strong analytical and problem-solving skills. Experience building reusable automation workflows rather than one-off scripts. Familiarity with relational databases (PostgreSQL preferred) and a normal Git workflow. Strong documentation and communication skills. Ability to work independently and take ownership of technical challenges. High attention to detail and commitment to data accuracy. Nice to Have: Experience working with government, regulatory, compliance, or public-sector datasets. Experience with Playwright, Selenium, Puppeteer, Scrapy, or similar scraping frameworks. Experience with data versioning, change detection, or document lineage. Familiarity with AI-assisted development tools and workflows.
Originally posted on Himalayas