Search Results for "python web crawler"
Sort By:
Showing 33 open source projects for "python web crawler"
View related business solutions-
AestheticsPro Medical Spa SoftwareAestheticsPro is the most complete Aesthetics Software on the market today. HIPAA Cloud Compliant with electronic charting, integrated POS, targeted marketing and results driven reporting; AestheticsPro delivers the tools you need to manage your medical spa business. It is our mission To Provide an All-in-One Cutting Edge Software to the Aesthetics Industry.
-
Rezku Point of SaleRezku is an all-inclusive ordering platform and management solution for all types of restaurant and bar concepts. You can now get a fully custom branded downloadable smartphone ordering app for your restaurant exclusively from Rezku.
-
1
IPRanges
Daily updated lists of cloud, bot, and service IP ranges
ipranges is an open source repository that provides continuously updated lists of IP address ranges associated with major cloud providers, search engine crawlers, and online services. ipranges collects IP ranges from publicly available sources and organizes them into structured files that can be easily used in security, networking, and automation workflows. It includes address ranges from providers such as Google Cloud, Amazon AWS, Microsoft, Oracle Cloud, and DigitalOcean, as well as well... -
2
fsociety
Modular CLI framework for managing penetration testing tools
...Instead of implementing its own security scanners, the framework acts as a wrapper and orchestrator that helps users discover, install, and execute tools from various GitHub repositories. Its modular architecture organizes tools into categories such as information gathering, networking, web application security, and password testing. This structure allows users to quickly navigate through different security tasks while maintaining a consistent interface. fsociety can automatically clone and manage required tools, reducing the manual effort typically needed to set up a penetration testing toolkit. fsociety is distributed as a Python package. -
3
Robin
AI-powered tool for dark web OSINT search and investigation
Robin is an AI-powered open source tool designed to assist investigators and researchers in conducting dark web OSINT (Open Source Intelligence) investigations. It combines automated dark web search capabilities with large language models (LLMs) to analyze and summarize information discovered across hidden services and Tor-based search engines. The tool helps refine investigative queries, collect results from multiple dark web sources, and filter relevant intelligence using AI-driven... -
4
WhatsApp Beacon
OSINT tool for tracking WhatsApp online status via Web automation
WhatsApp Beacon is an open source OSINT tool designed to monitor and analyze the online activity status of WhatsApp users through WhatsApp Web. It uses Selenium automation to interact with the web interface and detect when a target account goes online or offline. By continuously monitoring these changes, WhatsApp Beacon records connectivity patterns and builds a historical dataset of activity sessions. The collected information is stored in logs and a local database, allowing users to review... -
Loan management software that makes it easy.Bryt Software is ideal for lending professionals who are looking for a feature rich loan management system that is intuitive and easy to use. We are 100% cloud-based, software as a service. We believe in providing our customers with fair and honest pricing. Our monthly fees are based on your number of users and we have a minimal implementation charge.
-
5
Argus
Python toolkit for OSINT and reconnaissance with 135+ modules
Argus is a Python-based open source toolkit designed to simplify information gathering and reconnaissance tasks in cybersecurity. It provides an integrated command-line environment that consolidates numerous reconnaissance utilities into a single framework. The tool enables users to collect data about networks, domains, web applications, and infrastructure in an organized and efficient manner. -
6
Tookie-OSINT
Username OSINT tool for discovering accounts across many websites
Tookie-OSINT is an open source intelligence tool designed to help security researchers, ethical hackers, and investigators discover online accounts associated with a specific username. It automates the process of searching for usernames across multiple websites, making it easier to identify a person's presence on different platforms. By entering a target username, Tookie-OSINT scans a list of supported sites and checks whether the username exists on those platforms. This approach removes the... -
7
reNgine
Automated framework for web application reconnaissance and scanning
reNgine is an automated reconnaissance framework designed to simplify and enhance the process of gathering information about web applications during security assessments. It provides a streamlined workflow for penetration testers, bug bounty hunters, and security teams who need to perform reconnaissance efficiently and at scale. The platform integrates multiple open-source reconnaissance tools into a unified environment with a configurable scanning engine and an intuitive web interface.... -
8
OWASP Maryam
Modular OSINT framework for automated open-source intelligence gatheri
Maryam is an open source intelligence (OSINT) framework designed to automate the process of gathering and analyzing publicly available information from the internet. It provides a modular environment that enables users to collect data from search engines, open data sources, and various online services for reconnaissance and investigative purposes. Written in Python, Maryam is built to provide a flexible and extensible framework for harvesting information quickly and efficiently from open... -
9
discover
Automation framework for reconnaissance and penetration testing tasks
Discover is a collection of custom Bash scripts designed to automate many common tasks involved in penetration testing workflows. The project brings together a variety of security testing functions into a single framework that simplifies reconnaissance, scanning, and enumeration processes. It provides a menu-driven interface that allows security professionals to quickly launch different tools and scripts without manually executing each command. The framework helps streamline activities such... -
Failed Payment Recovery for Subscription BusinessesFlexPay’s innovative platform uses multiple technologies to achieve the highest number of retained customers, resulting in reduced involuntary churn, longer life span after recovery, and higher revenue. Leading brands like LegalZoom, Hooked on Phonics, and ClinicSense trust FlexPay to recover failed payments, reduce churn, and increase customer lifetime value.
-
10
IVRE
Open source framework for large scale network reconnaissance and analy
...The framework integrates with well known security and scanning tools such as Nmap, Masscan, ZGrab2, ZDNS, and Zeek to gather large amounts of network intelligence. IVRE stores the collected data in a database and offers multiple ways to explore and analyze it, including a web interface, command line tools, and a Python API. This allows security professionals to query scan results, inspect network flows, and identify patterns across large datasets. The project can be used to build self hosted alternatives to internet scanning services such as Shodan or Censys, giving organizations full control over their own reconnaissance infrastructure. -
11
BBOT
The recursive internet scanner for hackers
BBOT is an advanced open-source reconnaissance automation framework designed to streamline large-scale OSINT and attack surface discovery workflows. It operates as a modular and recursive scanning tool that can enumerate subdomains, perform port scans, gather metadata, and collect web intelligence through a unified command-line interface. The project emphasizes extensibility, allowing users to create or integrate custom modules that expand the scope of reconnaissance tasks without modifying... -
12
IntelOwl
Centralized platform for automated threat intelligence analysis
IntelOwl is an open source platform designed to manage and enrich threat intelligence data at scale. It provides a centralized environment where security analysts can gather information about suspicious files and observables such as IP addresses, domains, URLs, or hashes using a single API request. The platform integrates numerous online intelligence sources and advanced malware analysis tools, enabling users to obtain comprehensive threat intelligence without manually querying multiple... -
13
dnstwist
Detects phishing and lookalike domains using DNS fuzzing techniques
dnstwist is an open source cybersecurity tool designed to identify malicious or suspicious domain names that imitate legitimate websites. It works by generating a large set of domain name permutations based on a target domain and analyzing whether any of those variants are actively registered or used. These permutations simulate common techniques used in phishing attacks, typosquatting, and brand impersonation campaigns. Security teams can use the tool to discover potential threats where... -
14
Raccoon
High-performance reconnaissance and vulnerability scanning tool
Raccoon is a high-performance offensive security tool designed to assist with reconnaissance and vulnerability scanning during penetration testing and security assessments. It automates several common reconnaissance tasks, allowing security professionals to quickly gather information about a target system or web application. The tool combines multiple scanning techniques into a single workflow, helping users identify potential weaknesses, exposed services, and accessible resources on a... -
15
YoungerSibling
YoungerSibling: Cross-platform OSINT tool for quick data gathering.
YoungerSibling is a Python-based terminal utility script designed for educational purposes. It provides a set of useful tools to perform tasks like searching the web, performing lookups (Google search, IP lookup, username lookup, etc.), and extracting metadata from images, directly from the terminal. This project aims to help students, developers, and hobbyists learn about web scraping, API usage, and terminal interaction with Python. -
16
OnionSearch
Search multiple Tor .onion engines at once and collect hidden links.
...It also offers flexible command-line options that allow users to limit results, choose which engines to query, and export collected data. By automating searches across several dark web search engines, OnionSearch simplifies the process of discovering information on hidden services. -
17
TorBot
Dark Web OSINT Tool
Contributions to this project are always welcome. To add a new feature fork the dev branch and give a pull request when your new feature is tested and complete. If its a new module, it should be put inside the modules directory. The branch name should be your new feature name in the format <Feature_featurename_version(optional)>. On Linux platforms, you can make an executable for TorBot by using the install.sh script. You will need to give the script the correct permissions using chmod +x... -
18
NExfil
Fast OSINT tool for discovering web profiles by username
NExfil is an open source OSINT (Open Source Intelligence) tool designed to locate user profiles across the web based on a given username. Developed in Python, the tool automates the process of checking hundreds of websites to determine whether a specific username exists on those platforms. By performing automated queries across numerous services, NExfil helps investigators, researchers, and security professionals quickly identify potential accounts associated with a particular username. ... -
19
linkedin2username
Generate probable usernames from LinkedIn company employee lists
linkedin2username is an open source OSINT (Open Source Intelligence) tool designed to generate lists of potential usernames by scraping employee information from a company’s LinkedIn page. It logs into LinkedIn using valid user credentials and collects publicly visible employee names associated with a specified organization. Using these names, it automatically generates multiple possible username formats that organizations commonly use for accounts or email addresses. This process helps... -
20
paramspider
Mine parameterized URLs from web archives for security testing
ParamSpider is an open source command-line tool designed to discover URLs that contain parameters by mining historical data from web archives such as the Wayback Machine. It helps security researchers, penetration testers, and bug bounty hunters collect potential attack surfaces by automatically gathering archived URLs related to a specific domain. Instead of returning every discovered URL, the tool intelligently filters results to highlight parameterized endpoints that are more useful for... -
21
Yark
Simple OSINT tool for archiving and browsing YouTube channels offline
Yark is an open source command-line tool designed to simplify the process of archiving YouTube channels for research, analysis, or personal preservation. The project focuses on OSINT (Open Source Intelligence) workflows by allowing users to collect and store videos, metadata, and thumbnails from a YouTube channel in a structured local archive. Instead of simply downloading individual videos, Yark creates a self-contained archive directory that includes metadata files and organized folders... -
22
Moriarty Project
Web-based OSINT tool for investigating phone number information
Moriarty Project is an open source web-based investigation tool designed to gather publicly available information about phone numbers. It allows users to input a phone number and analyze various details related to that number through multiple investigation features. It performs information gathering by scraping data from online sources to retrieve insights such as owner information, spam risk, and related web references. Users can select specific investigation features to run individually or... -
23
cloud_enum
Multi-cloud OSINT tool for discovering public cloud resources
cloud_enum is an open source reconnaissance and OSINT tool designed to discover publicly accessible cloud resources across major cloud providers. It focuses on enumerating assets in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform using keyword-based discovery techniques. It works by taking user-provided keywords and generating variations through mutation wordlists, then testing these combinations against common cloud service naming patterns. cloud_enum performs both... -
24
ReconSpider
Most Advanced Open Source Intelligence (OSINT) Framework
...Reconnaissance is a mission to obtain information by various detection methods, about the activities and resources of an enemy or potential enemy, or geographic characteristics of a particular area. A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). -
25
HostHunter
OSINT reconnaissance tool for discovering hostnames from IP addresses
HostHunter is an open source reconnaissance tool designed to discover and extract hostnames associated with a large set of IPv4 or IPv6 addresses. It helps security professionals map IP addresses to virtual hostnames using a combination of OSINT data sources and active reconnaissance techniques. This approach enables users to identify hidden or additional services that may be hosted behind a single IP address. By correlating hostname information from certificates, APIs, HTTP headers, and...
