Search Results for "web crawler source code"
Sort By:
Showing 1721 open source projects for "web crawler source code"
View related business solutions-
Skillfully - The future of skills based hiringSkillfully transforms hiring through AI-powered skill simulations that show you how candidates actually perform before you hire them. Our platform helps companies cut through AI-generated resumes and rehearsed interviews by validating real capabilities in action. Through dynamic job specific simulations and skill-based assessments, companies like Bloomberg and McKinsey have cut screening time by 50% while dramatically improving hire quality.
-
Create stunning, professional email signatures in minutesCreate, assign and manage all your employees’ email signatures and add targeted marketing banners. Stop getting worked up about your signatures! Leverage a centralized interface to easily create and manage the email signatures of all your employees. Take advantage of each email to broadcast and amplify your brand. Letsignit helps you regain control over your digital identity. Harmonize 100% of your employee’s email signatures in just a few clicks! 121 professional emails are received and 40 are sent every day by an employee. With Letsignit, turn every email into a powerful communication opportunity: send the right message to the right person at the right time! Innovative more than tech, inspiring more than following. Authentic more than overrated, close more than "think big", trustworthy more than doubtful. Hands-on more than complex, available but yet premium, fun but yet expert.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
SiteOne Crawler
SiteOne Crawler is a website analyzer and exporter
SiteOne Crawler is a very useful and easy-to-use tool you'll ♥ as a Dev/DevOps, website owner or consultant. Works on all popular platforms - Windows, macOS, and Linux (x64 and arm64 too). It will crawl your entire website in depth, analyze and report problems, show useful statistics and reports, generate an offline version of the website, generate sitemaps, or send reports via email. Watch a detailed video with a sample report for Astro. build website. This crawler can be used as a... -
3
WebMagic
A scalable web crawler framework for Java
WebMagic is a scalable crawler framework. It covers the whole lifecycle of crawler, downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler. WebMagic is a simple but scalable crawler framework. You can develop a crawler easily based on it. WebMagic has a simple core with high flexibility, a simple API for html extracting. It also provides annotation with POJO to customize a crawler, and no configuration is needed. Some other... -
4
Heritrix
Internet Archive's open-source, web-scale, web crawler project
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. -
The CI/CD Platform built for Mobile DevOpsSave time, money, and developer frustration with fast, flexible, and scalable mobile CI/CD that just works. Whether you swear by native or would rather go cross-platform, we have you covered. From Swift to Objective-C, Java to Kotlin, as well as Xamarin, Cordova, Ionic, React Native, and Flutter: Whatever you choose, we will automatically configure your initial workflows and have you building in minutes.
-
5
Phoenix Code Editor
Phoenix is a modern open-source Code Editor for the web
Phoenix is a modern open-source and free software code editor for the web, built for the browser. -
6
Visual Studio Code
Modern IDE and code editor from Microsoft for Mac, Windows, and Linux
Visual Studio Code combines the simplicity of a code editor with what developers need for their core edit-build-debug cycle. It provides comprehensive code editing, navigation, and understanding support along with lightweight debugging, a rich extensibility model, and lightweight integration with existing tools. Visual Studio Code is a distribution of the Code - OSS repository with Microsoft-specific customizations released under a traditional Microsoft product license. Visual Studio Code is... -
7
Yii Web Programming Framework
Yii PHP Framework 1.1.x
Yii, a high-performance component-based PHP framework. Yii is a fast, secure, and efficient PHP framework. Flexible yet pragmatic. Works right out of the box. Has reasonable defaults. Yii gives you the maximum functionality by adding the least possible overhead. Sane defaults and built-in tools helps you write solid and secure code. Write more code in less time with simple, yet powerful APIs and code generation. The minimum requirement by Yii is that your Web server supports PHP 5.1.0 or... -
8
Laravel Web Tinker
Tinker in your browser
Artisan's tinker command is a great way to tinker with your application in the terminal. Unfortunately running a few lines of code, making edits, and copy/pasting code can be bothersome. Wouldn't it be great to tinker in the browser? This package will add a route to your application where you can tinker to your heart's content. In case light hurts your eyes, there's a dark mode too. -
9
claude-code-transcripts
Tools for publishing transcripts for Claude Code sessions
claude-code-transcripts is a command-line utility that takes session files exported from Claude Code (in JSON or JSONL format) and turns them into clean, navigable HTML transcripts that can be viewed in any modern web browser. It is designed to make the often dense and verbose outputs from AI coding sessions easier to read, share, and archive by breaking conversations into paginated, annotated pages with navigable timelines of prompts and responses. Users can run this tool locally or fetch... -
The Receptionist for iPad | the Original Visitor Management SystemThe Receptionist for iPad is visitor management software that allows users to calm the chaos of the front office. Our digital check-in solution is customizable to your needs; from your company branding, to configurable buttons and drag-and-drop-design badge printing. Effectively manage and track everyone who comes to your workspace and store the information securely in the cloud: no more paper visitor log!
-
10
Web Experience Toolkit
Open source code library for building innovative websites
Web Experience Toolkit (WET): Open source code library for building innovative websites that are accessible, usable, interoperable, mobile-friendly and multilingual. This collaborative open source project is led by the Government of Canada. A collection of flexible and themeable templates and reusable components. A collaborative open source project led by the Government of Canada. -
11
The Falcon Web Framework
The no-nonsense REST API and microservices framework
Falcon is a minimalist WSGI library for building speedy web APIs and app backends. We like to think of Falcon as the Dieter Rams of web frameworks. When it comes to building HTTP APIs, other frameworks weigh you down with tons of dependencies and unnecessary abstractions. Falcon cuts to the chase with a clean design that embraces HTTP and the REST architectural style. Highly optimized, extensible code base. Easy access to headers and bodies through request and response objects. DRY request... -
12
Lightning Web Components
A Blazing Fast, Enterprise-Grade Web Components Foundation
Lean, lightweight runtime optimized for performance, with minimal boilerplate code. Lightning Web Components uses standard HTML, modern JavaScript (ES6+), and the best of native Web Components. Leave abstractions behind and build custom elements with HTML, JavaScript, and CSS. Lightning web components are custom elements built using HTML and modern JavaScript. Ready to learn more? Check out the Developer Guide and write code in the Playground. You’ll be up and running in no time. Create... -
13
IPFS Web UI
A frontend for an IPFS node
A web interface to IPFS, shipped with Kubo, and ipfs-desktop. Check on your node stats, explore the IPLD powered merkle forest, see peers around the world and manage your files, without needing to touch the CLI. When working on the code, run an ipfs daemon, the local dev server, the unit tests, and the storybook component viewer and see the results of your changes as you save files. If you need to run IPFS in a Docker container, you can still have Web UI available by exposing both the... -
14
QR Code generator library
High-quality QR Code generator library in Java, TypeScript/JavaScript
This project aims to be the best, clearest library for generating QR Codes. My primary goals are flexible options and absolute correctness. The secondary goals are compact implementation size and good documentation comments. This work is an independent implementation based on reading the official ISO specification documents. I believe that my library has a more intuitive API and shorter code length than competing libraries out there. The library is designed first in Java and then ported to... -
15
The Apache Struts web framework
Mirror of Apache Struts
The Apache Struts web framework is a free open-source solution for creating Java web applications. Web applications differ from conventional websites in that web applications can create a dynamic response. Many websites deliver only static pages. A web application can interact with databases and business logic engines to customize a response. Web applications based on JavaServer Pages sometimes commingle database code, page design code, and control flow code. ... -
16
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
17
web-extension-starter
Web Extension starter to build "Write Once Run on Any Browser"
Web Extension starter to build "Write Once Run on Any Browser" extension. Cross Browser Support (Web-Extensions API). Browser Tailored Manifest generation. Automatic build on code changes. Auto packs browser-specific build files. SASS styling. TypeScript by default. ES6 modules support. React UI Library by default. Smart reload. Used by extensions in production that has over 100,000+ users. -
18
Jupyter Notebook
Jupyter Interactive Notebook
The notebook extends the console-based approach to interactive computing in a qualitatively new direction, providing a web-based application suitable for capturing the whole computation process: developing, documenting, and executing code, as well as communicating the results. The Jupyter notebook combines two components. A web application, which is a browser-based tool for interactive authoring of documents which combine explanatory text, mathematics, computations and their rich media... -
19
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe... -
20
Tencent Cloud Code Analysis
Static code analysis
Tencent Cloud Code Analysis (TCA for short, used internally by the R&D code CodeDog ) is a cloud-native, distributed, high-performance comprehensive code analysis and tracking platform that integrates many analysis tools, including server, web and client The three components have integrated a number of self-developed tools, and also support the dynamic integration of analysis tools of various programming languages in the industry. Obtain the Tencent Cloud code analysis platform by... -
21
Bot Framework Web Chat
A highly-customizable web-based client for Azure Bot Services
This repository contains code for the Bot Framework Web Chat component. The Bot Framework Web Chat component is a highly-customizable web-based client for the Bot Framework V4 SDK. The Bot Framework SDK v4 enables developers to model conversation and build sophisticated bot applications. This repo is part of the Microsoft Bot Framework, a comprehensive framework for building enterprise-grade conversational AI experiences. -
22
miniblink49
Lighter, faster browser kernel of blink to integrate HTML UI in apps
miniblink is an open source, one file, small browser widget based on chromium. By using C interface, you can create a browser with just some line code. miniblink is an open source, single-file, and currently the smallest known chromium-based browser control. Through its exported pure C interface, a browser control can be created in a few lines of code. C++, C#, Delphi and other language calls (support C++, C#, Delphi language to call). -
23
AWS Toolkit for Visual Studio Code
Local Lambda debug, CodeWhisperer, SAM/CFN syntax, etc.
The AWS Toolkit extension for Visual Studio Code enables you to interact with Amazon Web Services (AWS). Try the AWS Code Sample Catalog to start coding with the AWS SDK. The AWS Explorer provides access to the AWS services that you can work with when using the Toolkit. To see the AWS Explorer, choose the AWS icon in the Activity bar. The Developer Tools panel is a section for developer-focused tooling curated for working in an IDE. The Developer Tools panel can be found underneath the AWS... -
24
IntelliJ Community Edition
IntelliJ IDEA & IntelliJ Platform
IntelliJ Community is the open source upstream of JetBrains’ IntelliJ IDEA, forming the core of a powerful, extensible, and intelligent development environment. It provides foundational features like a robust editor with code completion, syntax highlighting, refactoring tools, version control integrations, terminal, debugger, and plugin architecture. Since it’s open, community developers can contribute to language supports, UI tweaks, and platform enhancements. -
25
mslearn-tailspin-spacegame-web
Code used in Microsoft Learn modules to support Azure DevOps
The Tailspin Space Game Web project is a sample application created by Microsoft as part of its learning resources. It’s a web-based game application used in Microsoft Learn modules and documentation to demonstrate concepts such as Azure App Services, continuous integration and delivery (CI/CD) pipelines, and DevOps practices with GitHub Actions and Azure Pipelines. The project is intentionally lightweight and easy to deploy so learners can quickly experiment with cloud deployment, testing,...
