Showing 156 open source projects for "web crawler source code"

View related business solutions
  • The #1 solution for profitable resource management Icon
    The #1 solution for profitable resource management

    Designed to give Operations and Finance leaders the insight and foresight they need to achieve profitable delivery at scale.

    Unlike spreadsheets or clunky PSAs, Float offers a clear, centralized view to schedule teams, plan capacity, estimate work, and track margins in real-time so that you can keep your people and profits on track.
    Learn More
  • Award-winning proxy networks, AI-powered web scrapers, and business-ready datasets for download.
 Icon
    Award-winning proxy networks, AI-powered web scrapers, and business-ready datasets for download.


    How the world collects public web data

    Bright Data is a leading data collection platform, enabling businesses to collect crucial structured and unstructured data from millions of websites through our proprietary technology. Our proxy networks give you access to sophisticated target sites using precise geo-targeting. You can also use our tools to unblock tough target sites, accomplish SERP-specific data collection tasks, manage and optimize your proxy performance as well as automating all of your data collection needs.
    Learn More
  • 1
    Spatie Crawler

    Spatie Crawler

    An easy to use, powerful crawler implemented in PHP

    Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites.
    Downloads: 6 This Week
    Last Update:
    See Project
  • 2
    Heritrix

    Heritrix

    Internet Archive's open-source, web-scale, web crawler project

    Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.
    Downloads: 6 This Week
    Last Update:
    See Project
  • 3
    claude-code-transcripts

    claude-code-transcripts

    Tools for publishing transcripts for Claude Code sessions

    claude-code-transcripts is a command-line utility that takes session files exported from Claude Code (in JSON or JSONL format) and turns them into clean, navigable HTML transcripts that can be viewed in any modern web browser. It is designed to make the often dense and verbose outputs from AI coding sessions easier to read, share, and archive by breaking conversations into paginated, annotated pages with navigable timelines of prompts and responses. Users can run this tool locally or fetch...
    Downloads: 5 This Week
    Last Update:
    See Project
  • 4
    QR Code generator library

    QR Code generator library

    High-quality QR Code generator library in Java, TypeScript/JavaScript

    This project aims to be the best, clearest library for generating QR Codes. My primary goals are flexible options and absolute correctness. The secondary goals are compact implementation size and good documentation comments. This work is an independent implementation based on reading the official ISO specification documents. I believe that my library has a more intuitive API and shorter code length than competing libraries out there. The library is designed first in Java and then ported to...
    Downloads: 11 This Week
    Last Update:
    See Project
  • The #1 AI-Powered eLearning Platform Icon
    The #1 AI-Powered eLearning Platform

    For users seeking a platform to generate online courses using AI

    Transform your content into engaging eLearning experiences with Coursebox, the #1 AI-powered eLearning authoring tool. Our platform automates the course creation process, allowing you to design a structured course in seconds. Simply make edits, add any missing elements, and your course is ready to go. Whether you want to publish privately, share publicly, sell your course, or export it to your LMS, Coursebox has you covered.
    Learn More
  • 5
    X-Crawl

    X-Crawl

    Flexible Node.js AI-assisted crawler library

    A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction.
    Downloads: 4 This Week
    Last Update:
    See Project
  • 6
    Python-Spider

    Python-Spider

    Python3 web crawler practice

    Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 7
    WWWBasic

    WWWBasic

    wwwBASIC is an implementation of BASIC that runs on Node.js & the Web

    wwwBASIC is a JavaScript-based implementation of the classic BASIC programming language designed to run seamlessly in web browsers and Node.js environments. Created by Google, it allows developers and enthusiasts to write and execute BASIC programs directly within HTML pages or via command-line tools. The interpreter compiles BASIC source code into JavaScript at load time, enabling efficient execution within modern web environments without requiring external emulators or plugins. ...
    Downloads: 2 This Week
    Last Update:
    See Project
  • 8
    Layui UI

    Layui UI

    An open source Web UI component library

    layui is a set of open-source Web UI component library, which adopts its own classic modularization specifications and follows the development method of native HTML/CSS/JS. It is very easy to use and ready to use. Its style is simple and light, while its components are elegant and rich. Every detail from source code to usage method has been carefully crafted, which is very suitable for the rapid development of web interface.
    Downloads: 5 This Week
    Last Update:
    See Project
  • 9
    Sycamore

    Sycamore

    A library for creating reactive web apps in Rust and WebAssembly

    A reactive library for creating web apps in Rust and WebAssembly. Sycamore harnesses the full power of Rust via WebAssembly, giving you full control over performance. Write code that feels natural. Everything is built on reactive primitives without a cumbersome virtual DOM. No JavaScript. Had enough of JavaScript? So have we. Create apps using Sycamore without touching a single line of JS.
    Downloads: 1 This Week
    Last Update:
    See Project
  • Streamline Hiring with Skill Assessments Icon
    Streamline Hiring with Skill Assessments

    Say goodbye to hiring guesswork. Use Canditech’s job simulation tests to assess real-world skills and make data-driven decisions.

    Canditech offers innovative, cheat-proof skill assessments and job simulations to transform your hiring process. From technical skills to soft skills, we help you assess candidates on actual job performance. With over 500 customizable tests and powerful video interview features, you can evaluate real-world capabilities, streamline your hiring, and reduce biases. Whether you’re hiring for remote roles, mass hiring, or looking to expand your diversity pool, Canditech’s data-driven platform ensures the right candidates are chosen for the job every time.
    Get a Free Demo
  • 10
    Testcontainers node

    Testcontainers node

    Supports tests, providing lightweight, throwaway instances of database

    Testcontainers is an open source library for providing throwaway, lightweight instances of databases, message brokers, web browsers, or just about anything that can run in a Docker container. No more need for mocks or complicated environment configurations. Define your test dependencies as code, then simply run your tests and containers will be created and then deleted.
    Downloads: 4 This Week
    Last Update:
    See Project
  • 11
    Prism

    Prism

    Lightweight, robust, elegant syntax highlighting

    Prism is a lightweight, extensible syntax highlighter, built with modern web standards in mind. It’s used in millions of websites, including some of those you visit daily.
    Downloads: 4 This Week
    Last Update:
    See Project
  • 12
    Coraza

    Coraza

    OWASP Coraza WAF is a golang modsecurity compatible firewall library

    ...Coraza runs the OWASP Core Rule Set (CRS) to protect your web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. CRS protects from many common attack categories including: SQL Injection (SQLi), Cross Site Scripting (XSS), PHP & Java Code Injection, HTTPoxy, Shellshock, Scripting/Scanner/Bot Detection & Metadata & Error Leakages. Coraza is a library at its core, with many integrations to deploy on-premise Web Application Firewall instances.
    Downloads: 6 This Week
    Last Update:
    See Project
  • 13
    Tamagui

    Tamagui

    Style React fast with 100% parity on React Native

    Tamagui is a cross-platform UI framework and styling system for React and React Native that enables developers to build high-performance applications with a single codebase shared across web and mobile platforms. It combines a universal style system with a powerful optimizing compiler that transforms styled components into efficient, platform-specific output, significantly improving runtime performance. The framework provides a superset of the React Native styling API, allowing developers to...
    Downloads: 19 This Week
    Last Update:
    See Project
  • 14
    multiOTP open source

    multiOTP open source

    PHP strong authentication library, web interface & CLI, OATH certified

    multiOTP is a PHP class, a powerful command line utility and a web interface developed by SysCo systèmes de communication sa in order to provide a completely free and easy operating system independent server side implementation for strong two factors authentication solution. multiOTP supports hardware and software tokens with different One-Time Password algorithms like OATH/HOTP, OATH/TOTP and mOTP (Mobile-OTP). QRcode generation is also embedded in order to support provisioning of Google...
    Downloads: 54 This Week
    Last Update:
    See Project
  • 15
    SQLite compiled to JavaScript

    SQLite compiled to JavaScript

    A javascript library to run SQLite on the web

    sql.js is a javascript SQL database. It allows you to create a relational database and query it entirely in the browser. You can try it in this online demo. It uses a virtual database file stored in memory, and thus doesn't persist the changes made to the database. However, it allows you to import any existing sqlite file, and to export the created database as a JavaScript typed array. sql.js uses emscripten to compile SQLite to webassembly (or to javascript code for compatibility with older...
    Downloads: 7 This Week
    Last Update:
    See Project
  • 16
    MudBlazor

    MudBlazor

    Do more with Blazor, utilizing CSS and keeping Javascript to a minimum

    Trusted by thousands of users, from hobby developers to large enterprises. Use MudBlazor to rapidly build amazing web applications without leaving your loved C# language and toolchain. We bring together everything that's required to build amazing Blazor applications that scale from desktop to mobile. Apart from the library itself we also provide templates, a learning platform, theme manager, demo and example projects as well as an online code editor integrated with our documentation and...
    Downloads: 5 This Week
    Last Update:
    See Project
  • 17
    JsStore

    JsStore

    A complete IndexedDB wrapper with SQL like syntax

    Execute Database operation in browsers with JsStore. JsStore is an indexedDB wrapper. It makes IndexedDB super easy with its SQL-like APIs. Harness the power of JsStore to streamline database operations in your web applications. With its SQL-like API, JsStore simplifies IndexedDB interactions, enabling developers to easily query, filter, and manipulate data with familiar syntax and efficiency.
    Downloads: 5 This Week
    Last Update:
    See Project
  • 18
    Hutool

    Hutool

    A set of tools that keep Java sweet

    Hutool is a small but comprehensive Java tool class library. Through static method encapsulation, it reduces the learning cost of related APIs, improves work efficiency, makes Java as elegant as a functional language, and makes the Java language "sweet". The tools and methods in Hutool come from each user's meticulous attention to detail. It covers all aspects of the underlying code of Java development. It is not only a sharp tool to solve small problems in large-scale project development,...
    Downloads: 7 This Week
    Last Update:
    See Project
  • 19
    tabler icons

    tabler icons

    1950 free MIT-licensed high-quality SVG icons

    A set of over 1950 free MIT-licensed high-quality SVG icons for you to use in your web projects. Over 1950 pixel-perfect icons for web design. Free and open-source icons designed to make your website or app attractive, visually consistent, and simply beautiful. Each icon is designed on a 24x24 grid and a 2px stroke. All icons are built with SVG, so you can place them as <img>, background-image and inline in HTML code.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 20
    Hello ImGui

    Hello ImGui

    Hello, Dear ImGui: unleash your creativity in app development

    hello_imgui is a cross-platform framework that simplifies the development of graphical applications using Dear ImGui. It abstracts away boilerplate code for window creation, input handling, and platform integration, allowing developers to focus on building user interfaces and interactive content quickly. Targeted at prototyping, tools, and GUI apps, hello_imgui works on Windows, macOS, Linux, iOS, and WebAssembly with minimal setup. Its plug-and-play nature makes it ideal for teaching,...
    Downloads: 8 This Week
    Last Update:
    See Project
  • 21
    templUI

    templUI

    A growing collection of beautifully designed UI components for Go

    templUI is a modern UI component library built specifically for Go applications using the templ templating language, offering a developer-first approach to building interfaces with strong type safety and performance. It provides a growing collection of beautifully designed, production-ready components styled with Tailwind CSS, allowing developers to construct modern web interfaces without relying on heavy frontend frameworks. One of its defining principles is code ownership, meaning...
    Downloads: 11 This Week
    Last Update:
    See Project
  • 22
    SurveyJS

    SurveyJS

    JavaScript Survey and Form Library

    SurveyJS Form Library is distributed as npm packages and as scripts and style sheets that you can reference on your page. You can use it in any React, Angular, Vue, Knockout, or jQuery application. React, Angular, Knockout, and Vue3 are supported natively. To communicate with the server, the libraries use JSON objects that represent form schemas (content and layout of a form) and form results (answers). You have the option to build dynamic JSON-driven forms using our free full-featured...
    Downloads: 13 This Week
    Last Update:
    See Project
  • 23
    p5.js

    p5.js

    Client-side JS platform for artists, designers and students to express

    p5.js is a JavaScript library for creative coding, with a focus on making coding accessible and inclusive for artists, designers, educators, beginners, and anyone else! p5.js is free and open-source because we believe software, and the tools to learn it, should be accessible to everyone. Using the metaphor of a sketch, p5.js has a full set of drawing functionality. However, you’re not limited to your drawing canvas. You can think of your whole browser page as your sketch, including HTML5...
    Downloads: 8 This Week
    Last Update:
    See Project
  • 24
    Plotly Graphing Library for MATLAB

    Plotly Graphing Library for MATLAB

    Plotly Graphing Library for MATLAB

    Create interactive charts in your web browser with MATLAB® and Plotly.
    Downloads: 3 This Week
    Last Update:
    See Project
  • 25
    AtlantaFX

    AtlantaFX

    Modern JavaFX CSS theme collection with additional controls

    Modern JavaFX CSS theme collection with additional controls.
    Downloads: 6 This Week
    Last Update:
    See Project
  • Previous
  • You're on page 1
  • 2
  • 3
  • 4
  • 5
  • Next
MongoDB Logo MongoDB