Search Results for "python web crawler"
Sort By:
Showing 56 open source projects for "python web crawler"
View related business solutions-
Automate Proposals with AI in Microsoft Word.Automate your best practices, processes, and standards to guide your proposal writers, sales teams, and subject experts. And don’t worry, it’s so easy to use they will use it. We would love the opportunity to help you quantify the impact your business can expect from investing in Expedience Software. Click here to request a Return on Investment (ROI) calculation. In this 15-minute session, we will ask 20 simple questions to assess and grade your current proposal quality and scalability. Manual proposal processes are likely costing you far more than you realize. These models waste time and kill the productivity of proposal writers, sales team members, senior staff, and subject experts.
-
Native Teams: Payments and Employment for International TeamsWith Native Teams’ Employer of Record (EOR) service, you can compliantly hire in 85+ countries without setting up a legal entity. From dedicated employee support and localised benefits to tax optimisation, we help you build a global team that feels truly cared for.
-
1
GPT Crawler
Crawl a site to generate knowledge files to create your own custom GPT
GPT Crawler is an open-source tool designed to automatically crawl websites and generate structured knowledge that can be used to build AI assistants and retrieval systems. It focuses on extracting high-quality textual content from web pages and preparing it in formats suitable for embedding, indexing, or fine-tuning workflows. The project is especially useful for teams that want to turn documentation sites or knowledge bases into conversational AI backends without building custom scrapers from scratch. ... -
2
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. DouyinCrawler supports both automated scraping and batch operations to process multiple targets efficiently. It also integrates with the Aria2 download utility to enable large-scale downloading of videos and images associated with collected content. ... -
3
watercrawl
AI-ready web crawler that extracts and structures website content
WaterCrawl is an open source web crawling and data extraction platform designed to transform website content into structured data suitable for machine learning and AI workflows. It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website.... -
4
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
Cloudbrink Personal SASE serviceCloudbrink’s Personal SASE is a high-performance connectivity and security service that delivers a lightning-fast, in-office experience to the modern hybrid workforce anywhere. Combining high-performance ZTNA with Automated Moving Target Defense (AMTD), and Personal SD-WAN all connections are ultra-secure.
-
5
Stlite
n-browser Streamlit
Stlite is a WebAssembly-powered framework that enables Streamlit applications to run entirely in the browser without requiring a Python backend server. It achieves this by using Pyodide, a WebAssembly-based Python runtime, to execute Python code directly within the browser environment. This allows developers to build interactive data applications using familiar Streamlit APIs while eliminating the need for server-side infrastructure. Stlite supports a wide range of Python libraries and enables real-time interactivity, making it suitable for dashboards, data visualization tools, and educational applications. ... -
6
MDCx
Movie metadata scraper and organizer for media libraries and NFO
MDCx is an open source media metadata scraping and organization tool designed to automate the process of collecting detailed information for movie files. It retrieves metadata from multiple online sources and applies it to local media collections, helping users maintain structured and well-organized libraries. MDCx can download information such as titles, cast data, artwork, and other metadata, then generate standardized NFO files compatible with media management systems. It also supports... -
7
Search with Lepton
Lightweight demo to build a conversational AI search engine quickly
...It includes both a backend service written in Python and a web interface that allows users to interact with the search engine in a conversational format. Developers can configure different search providers and language models through environment variables, making it flexible for experimentation and prototyping. -
8
LaVague
Framework for building AI agents that automate complex web tasks
LaVague is an open source framework designed to help developers build AI-powered web agents capable of automating tasks across websites and web applications. It implements the concept of a Large Action Model framework, allowing agents to interpret a user-provided objective and translate it into a sequence of actions performed in a browser. These agents can navigate web pages, retrieve information, fill out forms, and execute multi-step workflows automatically. LaVague is centered around a... -
9
Open Notebook
An Open Source implementation of Notebook LM with more flexibility
Open Notebook is an open-source, privacy-focused alternative to Google’s Notebook LM that gives users full control over their research and AI workflows. Designed to be self-hosted, it ensures complete data sovereignty by keeping your content local or within your own infrastructure. The platform supports 16+ AI providers—including OpenAI, Anthropic, Ollama, Google, and LM Studio—allowing flexible model choice and cost optimization. Open Notebook enables users to organize and analyze... -
A privacy-first API that predicts global consumer preferencesThrough our API, we provide contextualized personalization and insights based on a deep understanding of consumer behavior and more than 575 million people, places, and things.
-
10
Visual Studio Code
Modern IDE and code editor from Microsoft for Mac, Windows, and Linux
Visual Studio Code combines the simplicity of a code editor with what developers need for their core edit-build-debug cycle. It provides comprehensive code editing, navigation, and understanding support along with lightweight debugging, a rich extensibility model, and lightweight integration with existing tools. Visual Studio Code is a distribution of the Code - OSS repository with Microsoft-specific customizations released under a traditional Microsoft product license. Visual Studio Code is... -
11
Quadratic
Data science spreadsheet with Python & SQL
Quadratic enables your team to work together on data analysis to deliver better results, faster. You already know how to use a spreadsheet, but you’ve never had this much power before. Quadratic is a Web-based spreadsheet application that runs in the browser and as a native app (via Electron). Our goal is to build a spreadsheet that enables you to pull your data from its source (SaaS, Database, CSV, API, etc) and then work with that data using the most popular data science tools today (Python, Pandas, SQL, JS, Excel Formulas, etc). ... -
12
Harbor LLM
Run a full local LLM stack with one command using Docker
Harbor is an open source, containerized toolkit designed to simplify running local large language model (LLM) environments. It combines a CLI and companion app to launch backends, frontends, and supporting services with minimal setup. With a single command, users can start preconfigured tools like Ollama and Open WebUI, enabling chat, workflows, and integrations immediately. Harbor supports multiple inference engines, including llama.cpp and vLLM, and connects them seamlessly to user... -
13
Every Code
Local AI coding agent CLI with multi-agent orchestration tools
Every Code (often referred to simply as Code) is a fast, local AI-powered coding agent designed to run directly in the terminal environment. It is a community-driven fork of the Codex CLI, with a strong emphasis on improving real-world developer ergonomics and workflows. Every Code enhances the traditional coding assistant model by introducing multi-agent orchestration, allowing multiple AI agents to collaborate, compare solutions, and refine outputs in parallel. It supports integration with... -
14
Xianyu Intelligent Monitor Bot
AI tool for real-time monitoring and analysis of Goofish listings
ai-goofish-monitor is an open source automation tool designed to monitor listings on the Goofish second-hand marketplace and analyze them using artificial intelligence. It combines browser automation with AI-based analysis to automatically search, collect, and evaluate newly posted items that match a user’s purchase criteria. It uses Playwright to simulate real user interactions with the marketplace, allowing the system to retrieve product data and track updates in near real time.... -
15
OpenHands
Open-source autonomous AI software engineer
Welcome to OpenHands (formerly OpenDevin), an open-source autonomous AI software engineer who is capable of executing complex engineering tasks and collaborating actively with users on software development projects. Use AI to tackle the toil in your backlog, so you can focus on what matters: hard problems, creative challenges, and over-engineering your dotfiles We believe agentic technology is too important to be controlled by a few corporations. So we're building all our agents in the... -
16
pyTelegramBotAPI
Python Telegram bot api.
TeleBot is the synchronous and asynchronous implementation of Telegram Bot API. -
17
Fragments by E2B
Open source template for AI-powered code generation apps w/ sandboxes
...It supports multiple programming frameworks including Python interpreters, Next.js, Vue.js, Streamlit, and Gradio, allowing generated projects to span from simple scripts to full web applications. It integrates with a wide range of large language model providers and supports streaming responses. -
18
Eigent
The Open Source Cowork Desktop to Unlock Your Exceptional Productivity
Eigent is an open-source cowork desktop application designed to help you build, manage, and deploy a custom AI workforce. It enables multiple specialized AI agents to collaborate in parallel, turning complex workflows into automated, end-to-end tasks. Built on the CAMEL-AI multi-agent framework, Eigent emphasizes productivity, flexibility, and transparent system design. You can run Eigent fully locally for maximum privacy and data control, or choose a cloud-connected experience for quick... -
19
JupyterLite
Wasm powered Jupyter running in the browser
JupyterLite is a browser-based distribution of the Jupyter ecosystem that enables interactive computing entirely within a web browser without requiring a backend server. Built using JupyterLab components and powered by WebAssembly technologies, it allows users to run Python and other language kernels directly in the browser through tools like Pyodide or Xeus. This architecture eliminates the need for installation or server infrastructure, making it highly accessible for education, demonstrations, and lightweight data science workflows. ... -
20
Theia
Cloud & Desktop IDE Platform
Eclipse Theia is an open source extensible platform for developing multi-language Cloud and Desktop IDEs with state-of-the-art web technologies. If you’re not sure if you need a web version or a desktop version or both, Theia is the best solution. It can develop one IDE for both browsers and native desktop applications from a single source. Theia has a highly flexible architecture, allowing extenders and adopters to customize and extend every aspect of it. It supports VS code extensions,... -
21
CapRover
Scalable PaaS (automated Docker+nginx), aka Heroku on Steroids
CapRover is an extremely easy-to-use app/database deployment & web server manager for your NodeJS, Python, PHP, ASP.NET, Ruby, MySQL, MongoDB, Postgres, WordPress (and etc...) applications! It's blazingly fast and very robust as it uses Docker, Nginx, LetsEncrypt and NetData under the hood behind its simple-to-use interface. For a developer who does not like spending hours and days setting up a server, building tools, sending code to the server, building it, getting an SSL certificate, installing it, update nginx over and over again. ... -
22
AstronRPA
Agent-ready RPA suite with visual workflow automation tools engine
Astron RPA is an enterprise-grade robotic process automation platform designed to help organizations and developers build automated workflows for desktop and web applications. It provides a visual workflow designer that supports low-code and no-code development, allowing users to create automation processes through a drag-and-drop interface instead of writing extensive code. It enables automation of common desktop software and browser-based tasks, making it suitable for repetitive business... -
23
freeCodeCamp
freeCodeCamp.org's open-source codebase and curriculum
freeCodeCamp is a nonprofit educational platform that offers a self-paced curriculum for learning web development, programming, data visualization, APIs, and algorithms. It features interactive coding challenges, real-world projects, and guided progress through topic modules, culminating in certificates for completed tracks. A key aspect is that students contribute to open-source projects for nonprofits or internal tooling as part of their learning, reinforcing both technical and... -
24
CoPaw
Your Personal AI Assistant; easy to install, deploy on local or coud
CoPaw is a personal AI assistant designed to run on your own machine or in the cloud, giving you full control over memory, models, and data. Built by the AgentScope team, it connects to multiple chat platforms—including DingTalk, Feishu, QQ, Discord, iMessage, and more—through a single unified assistant. CoPaw supports both cloud-based LLM providers and fully local models such as llama.cpp, MLX, and Ollama, allowing you to operate without API keys if preferred. It includes a browser-based... -
25
Universal Commerce Protocol (UCP)
The common language for platforms, agents and businesses.
Universal Commerce Protocol (UCP) is an open standard designed to unify how platforms, businesses, and payment providers interact across the modern commerce ecosystem. It provides a common language that eliminates fragmented, custom integrations and enables seamless interoperability between diverse commerce systems. Built for an increasingly agentic web, UCP supports AI-driven platforms that can discover products, manage carts, and complete transactions securely on a user’s behalf. Its...
