Web Scraping Specialist
职位概述：我们正在寻找一个高技能的网页剪辑架构师加入我们的团队。成功的候选人将负责设计，实施和维护网页剪译流程以有效地和准确地收集各种在线来源的数据。作为 Web Scraping 专家，你将在竞争对手分析和其他商业情报目的收集数据方面发挥关键作用。责任：·可扩展性/性能：领导并提供规模化电子商务市场中的剪辑专业知识。· 数据源识别：确定相关网站和需要从哪些网络来源中取消数据。合作了解数据需求和目标。· 网页剪辑设计：开发并实施有效的网页剪写策略，以从目标网站中提取数据。这包括选择适当的工具，图书馆或框架来完成任务。· 数据摘录：创建和维护 web 扫描脚本或程序，以提取所需的数据。确保代码优化，可靠，并且能够处理网站结构中的变化。· 数字清洁和验证：清理和验证收集的数据以消除错误，不一致性和重复。在整个过程中确保数据完整和准确性。·监控和保养：持续监控和维护网页选过程。解决网站变化，数据格式修改或反选机制造成的任何问题。·可扩展性和性能：优化网页取程序以提高效率和可扩展度，特别是处理大量数据或多个数据来源时。· 合规性和法律考虑：保持与网页？? 选相关的法律和伦理方面的最新信息，包括网站服务条款，版权和隐私法规。· 文档：维护网頁选流程，数据源和方法论的详细文件。建立明确和简洁的指示，让其他人遵循。·协作：与数据分析师，开发人员和业务利益相关者等其他团队合作，以有效地了解数据需求并提供见解。·安全：在整个网页扫描过程中实施保密措施和保护敏感数据。要求：·经过4年以上网络扫描专家或类似职位的验证经验，具有成功的网页扫黑记录。
查看英文原文
Position Overview:
We are seeking a highly skilled Web Scraping Architect to join our team. The successful candidate will be responsible for designing, implementing, and maintaining web scraping processes to gather data from various online sources efficiently and accurately. As a Web Scraping Specialist, you will play a crucial role in collecting data for competitor analysis, and other business intelligence purposes.
Responsibilities:
- Scalability/Performance: Lead and provide expertise in scraping at scale e-commerce marketplaces.
- Data Source Identification: Identify relevant websites and online sources from which data needs to be scraped. Collaborate with the team to understand data requirements and objectives.
- Web Scraping Design: Develop and implement effective web scraping strategies to extract data from targeted websites. This includes selecting appropriate tools, libraries, or frameworks for the task.
- Data Extraction: Create and maintain web scraping scripts or programs to extract the required data. Ensure the code is optimized, reliable, and can handle changes in the website's structure.
- Data Cleansing and Validation: Cleanse and validate the collected data to eliminate errors, inconsistencies, and duplicates. Ensure data integrity and accuracy throughout the process.
- Monitoring and Maintenance: Continuously monitor and maintain the web scraping processes. Address any issues that arise due to website changes, data format modifications, or anti-scraping mechanisms.
- Scalability and Performance: Optimize web scraping procedures for efficiency and scalability, especially when dealing with a large volume of data or multiple data sources.
- Compliance and Legal Considerations: Stay up-to-date with legal and ethical considerations related to web scraping, including website terms of service, copyright, and privacy regulations.
- Documentation: Maintain detailed documentation of web scraping processes, data sources, and methodologies. Create clear and concise instructions for others to follow.
- Collaboration: Collaborate with other teams such as data analysts, developers, and business stakeholders to understand data requirements and deliver insights effectively.
- Security: Implement security measures to ensure the confidentiality and protection of sensitive data throughout the scraping process.
Requirements:
- Proven experience of 4+ years as a Web Scraping Specialist or similar role, with a track record of successful web scraping projects.
- Expertise in handling dynamic content, user-agent rotation, bypass CAPTCHAs, rate limits, and utilization of proxy services.
- Knowledge on browser fingerprinting
- Has leadership experience.
- Proficiency in programming languages commonly used for web scraping, such as Python, BeautifulSoup, Scrapy, or Selenium.
- Strong knowledge of HTML, CSS, XPath, and other web technologies relevant to web scraping and Coding.
- Knowledge and experience in best of class data storage and retrieval of large volume of scraped data.
- Understanding of web scraping best practices, including handling dynamic content, user-agent rotation, and IP address management.
- Attention to detail and the ability to handle and process large volumes of data accurately.
- Familiarity with data cleansing techniques and data validation processes.
- Good communication skills and the ability to collaborate effectively with cross-functional teams.
- Knowledge of web scraping ethics, legal considerations, and compliance with website terms of service.
- Strong problem-solving skills and the ability to adapt to changing web environments
Preferred Qualifications:
- Bachelor's degree in Computer Science, Data Science, Information Technology, or related fields.
- Experience with cloud-based solutions and distributed web scraping systems.
- Familiarity with APIs and data extraction from non-public sources.
- Knowledge of machine learning techniques for data extraction and natural language processing is desired but not mandatory
- Prior experience in handling large-scale data projects and working with big data frameworks.
- Understanding of various data formats such as JSON, XML, CSV, etc.
- Experience with version control systems like Git.
This is a Work from Home position
Originally posted on Himalayas