Web crawler that finds hidden web directories without brute force
Distributed Crawler Management Framework Based on Scrapy
Descubre archivos, rutas escondidas realizando busquedas avanzadas
Task automation software for accessing and manipulating website data.
Easy Spider is a distributed Perl Web Crawler Project from 2006
A service daemon to run Scrapy spiders
Headless Chrome crawler for collecting URLs for vulnerability scans
Free Extracts Emails, Phones and custom text from Web using JAVA Regex
Using industrial automation techniques for creating web scraping tools
Web crawler for archiving and backing up sites into WARC archives
Simple Python framework for building multithreaded web crawlers
Automated mobile app crawler and testing tool built on Appium
Instagram profile crawler that extracts posts, tags, and stats
Gospider - Fast web spider written in Go
Async Python framework for fast and flexible web scraping spiders
Guide and resources for accessing and using the U3C3 BitTorrent site
Proxy crawler that aggregates, tests, and serves usable proxy nodes
Collection of Python ecommerce and website crawler examples projects
Ever wanted to download only a part of a Git repository.
Open source web crawler for Java
Android app for saving webpages for offline reading
Finds all the security information for a given domain name
anonymous tools [uncontinued]
Compile your mobile web pages into mobile aps via build.phonegap.com
A web crawler oriented to information security.