article · SoftwareX
RCrawler is an open-source R package developed for domain-based web crawling and data scraping. It provides the first parallel web crawling framework inside the R programming environment. The software allows users to crawl websites, parse and store web pages, extract text and structural content, and generate datasets ready for web content mining tasks. Key technical capabilities include multi-threaded execution, duplicate content detection, URL and content-type filtering, depth level controls, and robots.txt parsing. The architecture is optimised to handle specific runtime limitations within R, delivering high-throughput downloads of many pages per second while maintaining resilience against spider traps and system crashes. Beyond standard web mining, the tool features a flexible design that can be adapted to support diverse data gathering applications.
Web data extraction is a foundational step in web content mining, but processing large volumes of online material requires fast, parallel execution. By providing an integrated multi-threaded crawler within R, this tool allows data scientists and analysts to collect, filter, and parse large online datasets directly in their existing statistical programming workflows without relying on external scraping pipelines.
RCrawler serves as a ready-to-use software tool for data scientists, market researchers, and developers requiring structured web data collection. Because it exists as an implemented and experimentally tested R package, the technology is immediately usable for web content mining workflows. Potential commercial applications include automated competitive intelligence gathering, social web monitoring, and market research, provided users adapt its filtering and parsing features to their operational needs.
AI-generated from the published abstract. Always read the original work before citing.
RCrawler is a contributed R package for domain-based web crawling and content scraping. As the first implementation of a parallel web crawler in the R environment, RCrawler can crawl, parse, store pages, extract contents, and produce data that can be directly employed for web content mining applications. However, it is also flexible, and could be adapted to other applications. The main features of RCrawler are multi-threaded crawling, content extraction, and duplicate content detection. In addition, it includes functionalities such as URL and content-type filtering, depth level controlling, and a robot.txt parser. Our crawler has a highly optimized system, and can download a large number of pages per second while being robust against certain crashes and spider traps. In this paper, we describe the design and functionality of RCrawler, and report on our experience of implementing it in an R environment, including different optimizations that handle the limitations of R. Finally, we discuss our experimental results.
This page summarises published work. The authoritative version sits with the publisher.
DOI: 10.1016/j.softx.2017.04.004
Is something wrong with this record? Report it or request removal.
Discussion
Have you built on this work, tried to replicate it, or seen it applied in practice? Share what you know. Verified researchers and MARATTO™ domain experts can open a discussion, and any member can reply. Contributions are reviewed before they appear.
No discussion yet. Open the first thread.
New to MARATTO™? Create a free account.
