Search Results for "python web crawler"
Sort By:
63 projects for "python web crawler" with 2 filters applied:
-
CloudZero: The Cloud Cost Optimization PlatformCloudZero is the leader in proactive cloud cost efficiency. We enable engineers to build cost-efficient software without slowing down innovation. CloudZero's next-generation cloud cost optimization platform automates the collection, allocation, and analysis of cloud costs to uncover savings opportunities and improve unit economics. We are the only platform that enables companies to understand 100% of their operational cloud spend and take an engineering-led approach to optimizing that spend. CloudZero is used by industry leaders worldwide, such as Coinbase, Klaviyo, Miro, Nubank, and Rapid7.
-
CompanyCam is a photo-based solution created for contractors, by contractors.Take unlimited photos—which are location and time-stamped, sent to the cloud, and stored securely. Every photo is organized by project and instantly available to your team, allowing you to see what’s going on anytime, anywhere. Annotate photos with drawings, arrows, comments, tags, and voice notes, and create project timelines, photo galleries, reports, and transformation photos through the app. Sharing photos with customers and insurance adjusters has never been easier, and keeping your entire process organized has never been simpler.
-
1
IPRanges
Daily updated lists of cloud, bot, and service IP ranges
ipranges is an open source repository that provides continuously updated lists of IP address ranges associated with major cloud providers, search engine crawlers, and online services. ipranges collects IP ranges from publicly available sources and organizes them into structured files that can be easily used in security, networking, and automation workflows. It includes address ranges from providers such as Google Cloud, Amazon AWS, Microsoft, Oracle Cloud, and DigitalOcean, as well as well... -
2
Robin
AI-powered tool for dark web OSINT search and investigation
Robin is an AI-powered open source tool designed to assist investigators and researchers in conducting dark web OSINT (Open Source Intelligence) investigations. It combines automated dark web search capabilities with large language models (LLMs) to analyze and summarize information discovered across hidden services and Tor-based search engines. The tool helps refine investigative queries, collect results from multiple dark web sources, and filter relevant intelligence using AI-driven... -
3
fsociety
Modular CLI framework for managing penetration testing tools
...Instead of implementing its own security scanners, the framework acts as a wrapper and orchestrator that helps users discover, install, and execute tools from various GitHub repositories. Its modular architecture organizes tools into categories such as information gathering, networking, web application security, and password testing. This structure allows users to quickly navigate through different security tasks while maintaining a consistent interface. fsociety can automatically clone and manage required tools, reducing the manual effort typically needed to set up a penetration testing toolkit. fsociety is distributed as a Python package. -
4
WhatsApp Beacon
OSINT tool for tracking WhatsApp online status via Web automation
WhatsApp Beacon is an open source OSINT tool designed to monitor and analyze the online activity status of WhatsApp users through WhatsApp Web. It uses Selenium automation to interact with the web interface and detect when a target account goes online or offline. By continuously monitoring these changes, WhatsApp Beacon records connectivity patterns and builds a historical dataset of activity sessions. The collected information is stored in logs and a local database, allowing users to review... -
Haystack is a modern, engaging, and intuitive intranet platform that employees actually use.With customizable iOS and Android mobile apps, Slack and Microsoft Teams integrations, and an intuitive design employees love, Haystack brings an outstanding digital employee experience to your entire workforce, no matter where their work takes them.
-
5
Argus
Python toolkit for OSINT and reconnaissance with 135+ modules
Argus is a Python-based open source toolkit designed to simplify information gathering and reconnaissance tasks in cybersecurity. It provides an integrated command-line environment that consolidates numerous reconnaissance utilities into a single framework. The tool enables users to collect data about networks, domains, web applications, and infrastructure in an organized and efficient manner. -
6
discover
Automation framework for reconnaissance and penetration testing tasks
Discover is a collection of custom Bash scripts designed to automate many common tasks involved in penetration testing workflows. The project brings together a variety of security testing functions into a single framework that simplifies reconnaissance, scanning, and enumeration processes. It provides a menu-driven interface that allows security professionals to quickly launch different tools and scripts without manually executing each command. The framework helps streamline activities such... -
7
IntelOwl
Centralized platform for automated threat intelligence analysis
IntelOwl is an open source platform designed to manage and enrich threat intelligence data at scale. It provides a centralized environment where security analysts can gather information about suspicious files and observables such as IP addresses, domains, URLs, or hashes using a single API request. The platform integrates numerous online intelligence sources and advanced malware analysis tools, enabling users to obtain comprehensive threat intelligence without manually querying multiple... -
8
dnstwist
Detects phishing and lookalike domains using DNS fuzzing techniques
dnstwist is an open source cybersecurity tool designed to identify malicious or suspicious domain names that imitate legitimate websites. It works by generating a large set of domain name permutations based on a target domain and analyzing whether any of those variants are actively registered or used. These permutations simulate common techniques used in phishing attacks, typosquatting, and brand impersonation campaigns. Security teams can use the tool to discover potential threats where... -
9
BBOT
The recursive internet scanner for hackers
BBOT is an advanced open-source reconnaissance automation framework designed to streamline large-scale OSINT and attack surface discovery workflows. It operates as a modular and recursive scanning tool that can enumerate subdomains, perform port scans, gather metadata, and collect web intelligence through a unified command-line interface. The project emphasizes extensibility, allowing users to create or integrate custom modules that expand the scope of reconnaissance tasks without modifying... -
White Labeled Fintech Software Solutions | CentrexThe Centrex products include: CRM, loan origination, loan and advance servicing software, syndication management, white labeled mobile app, money manager, underwriting, Esign, and website smart app builder. The Centrex services include: fintech software consulting, admin retainer services, and managed data cloud.
-
10
Raccoon
High-performance reconnaissance and vulnerability scanning tool
Raccoon is a high-performance offensive security tool designed to assist with reconnaissance and vulnerability scanning during penetration testing and security assessments. It automates several common reconnaissance tasks, allowing security professionals to quickly gather information about a target system or web application. The tool combines multiple scanning techniques into a single workflow, helping users identify potential weaknesses, exposed services, and accessible resources on a... -
11
Wapiti
Wapiti is a web-application vulnerability scanner
Wapiti is a vulnerability scanner for web applications. It currently search vulnerabilities like XSS, SQL and XPath injections, file inclusions, command execution, XXE injections, CRLF injections, Server Side Request Forgery, Open Redirects... It use the Python 3 programming language. -
12
YoungerSibling
YoungerSibling: Cross-platform OSINT tool for quick data gathering.
YoungerSibling is a Python-based terminal utility script designed for educational purposes. It provides a set of useful tools to perform tasks like searching the web, performing lookups (Google search, IP lookup, username lookup, etc.), and extracting metadata from images, directly from the terminal. This project aims to help students, developers, and hobbyists learn about web scraping, API usage, and terminal interaction with Python. -
13
ufonet
UFONet - Denial of Service Toolkit
UFONet - Is a set of hacktivist tools that allow launching coordinated DDoS and DoS attacks and combine both in a single offensive. It also works as an encrypted DarkNET to publish and receive content by creating a global client/server network based on a direct-connect P2P architecture. + FAQ: https://ufonet.03c8.net/FAQ.html -------------------------------------------- -> UFONet-v1.8 [DPh] "DarK-PhAnT0m!" (.zip) -> md5 = [ c8ab016f6370c8391e2e6f9a7cbe990a ] -> UFONet-v1.8... -
14
OnionSearch
Search multiple Tor .onion engines at once and collect hidden links.
...It also offers flexible command-line options that allow users to limit results, choose which engines to query, and export collected data. By automating searches across several dark web search engines, OnionSearch simplifies the process of discovering information on hidden services. -
15
NExfil
Fast OSINT tool for discovering web profiles by username
NExfil is an open source OSINT (Open Source Intelligence) tool designed to locate user profiles across the web based on a given username. Developed in Python, the tool automates the process of checking hundreds of websites to determine whether a specific username exists on those platforms. By performing automated queries across numerous services, NExfil helps investigators, researchers, and security professionals quickly identify potential accounts associated with a particular username. ... -
16
linkedin2username
Generate probable usernames from LinkedIn company employee lists
linkedin2username is an open source OSINT (Open Source Intelligence) tool designed to generate lists of potential usernames by scraping employee information from a company’s LinkedIn page. It logs into LinkedIn using valid user credentials and collects publicly visible employee names associated with a specified organization. Using these names, it automatically generates multiple possible username formats that organizations commonly use for accounts or email addresses. This process helps... -
17
paramspider
Mine parameterized URLs from web archives for security testing
ParamSpider is an open source command-line tool designed to discover URLs that contain parameters by mining historical data from web archives such as the Wayback Machine. It helps security researchers, penetration testers, and bug bounty hunters collect potential attack surfaces by automatically gathering archived URLs related to a specific domain. Instead of returning every discovered URL, the tool intelligently filters results to highlight parameterized endpoints that are more useful for... -
18
Yark
Simple OSINT tool for archiving and browsing YouTube channels offline
Yark is an open source command-line tool designed to simplify the process of archiving YouTube channels for research, analysis, or personal preservation. The project focuses on OSINT (Open Source Intelligence) workflows by allowing users to collect and store videos, metadata, and thumbnails from a YouTube channel in a structured local archive. Instead of simply downloading individual videos, Yark creates a self-contained archive directory that includes metadata files and organized folders... -
19
Hakrawler
Fast Go web crawler for discovering URLs and web app endpoints
hakrawler is a lightweight command-line web crawler built in Go that is designed to quickly discover URLs, endpoints, and assets within web applications. It is primarily used during the reconnaissance phase of security testing, bug bounty hunting, and penetration testing. It works by automatically crawling web pages and extracting links, JavaScript file locations, and other resources that may reveal additional attack surface or hidden functionality. hakrawler is implemented as a simple and efficient crawler using the Gocolly library, which allows it to perform fast and concurrent crawling of web pages. ... -
20
cloud_enum
Multi-cloud OSINT tool for discovering public cloud resources
cloud_enum is an open source reconnaissance and OSINT tool designed to discover publicly accessible cloud resources across major cloud providers. It focuses on enumerating assets in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform using keyword-based discovery techniques. It works by taking user-provided keywords and generating variations through mutation wordlists, then testing these combinations against common cloud service naming patterns. cloud_enum performs both... -
21
Trape
OSINT tool for tracking users and analyzing browser data online
Trape is an open source OSINT analysis and research tool designed to track and analyze users on the internet in real time. The project focuses on demonstrating how web browsers can reveal sensitive information about users while interacting with websites and online services. It provides researchers, security professionals, and organizations with a platform for studying how attackers could gather intelligence through social engineering techniques. The tool can clone websites and monitor... -
22
phpsploit
Full-featured C2 framework which silently persists on webserver
Full-featured C2 framework which silently persists on webserver via polymorphic PHP oneliner. The obfuscated communication is accomplished using HTTP headers under standard client requests and web server's relative responses, tunneled through a tiny polymorphic backdoor. Detailed help for any option (help command) Cross-platform on both client and server. CLI supports auto-completion & multi-command. Session saving/loading feature & persistent history. Multi-request support for large... -
23
PHP mini vulnerability suite
Multiple server/webapp vulnerability scanner
github: https://github.com/samedog/phpmvs -
24
cintruder
CIntruder - OCR Bruteforcing Toolkit
Captcha Intruder is an automatic pentesting tool to bypass captchas. -> CIntruder-v0.4 (.zip) -> md5 = 6326ab514e329e4ccd5e1533d5d53967 -> CIntruder-v0.4 (.tar.gz) ->md5 = 2256fccac505064f3b84ee2c43921a68 -------------------------------------------- -
25
AttackSurfaceMapper
Automated tool for mapping & expanding organization’s attack surface
AttackSurfaceMapper (ASM) is a reconnaissance and attack surface discovery tool designed to automate the process of mapping potential targets within an organization's infrastructure. It combines open source intelligence (OSINT) with selective active reconnaissance techniques to expand and analyze a target’s external attack surface. Users can supply domains, subdomains, or IP addresses as input, and applies multiple discovery methods to identify additional related assets such as new...
