Search Results for "parallel compression"
Sort By:
Showing 26 open source projects for "parallel compression"
View related business solutions-
Fax.Cloud delivers encrypted, point-to-point faxing with guaranteed delivery and built-in audit trailsUnlike email or file-sharing tools, Fax.Cloud doesn’t bounce around the internet, exposed and vulnerable. It’s direct, encrypted, and verified. You get delivery confirmation, audit trails, and peace of mind, without the spam filters, metadata leaks, or cyber threats.
-
Feroot AI automates website security with 24/7 monitoringFeroot unifies JavaScript behavior analysis, web compliance scanning, third-party script monitoring, consent enforcement, and data privacy posture management to stop Magecart, formjacking, and unauthorized tracking.
-
1
compress
Optimized Go Compression Packages
This package provides various compression algorithms. zstandard compression and decompression in pure Go. S2 is a high-performance replacement for Snappy. Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib. snappy is a drop-in replacement for github.com/golang/snappy offering better compression and concurrent streams. huff0 and FSE implementations for raw entropy encoding. gzhttp Provides client and server wrappers for handling gzipped requests efficiently. pgzip is a separate package that provides a very fast parallel gzip implementation. ... -
2
Vortex
The LLVM of columnar file formats
Vortex is a high-performance toolkit designed for working with compressed Apache Arrow arrays, providing functionality for in-memory, on-disk, and over-the-wire data handling. It aims to be an advanced successor to Apache Parquet, offering dramatically faster random access reads and scans, while maintaining similar compression ratios. Vortex's modular design allows for extensibility, enabling developers to implement custom encodings for efficient data management, particularly for large-scale... -
3
HDF5
Official HDF5® Library Repository
HDF5 (Hierarchical Data Format v5) is a widely-used data management library and file format for storing large and complex scientific data sets efficiently. -
4
Koog
Koog is the official Kotlin framework for building AI agents
...It features pure Kotlin implementation, seamless Model Control Protocol (MCP) integration for enhanced model management, vector embeddings for semantic search, and a flexible system for creating and extending tools that access external systems and APIs. Ready‑to‑use components address common AI engineering challenges, while intelligent history compression optimizes token usage and preserves context. A powerful streaming API enables real‑time response processing and parallel tool calls. Persistent memory allows agents to retain knowledge across sessions and between agents, and comprehensive tracing facilities provide detailed debugging and monitoring. -
Cortex: Boost Developer Coding SkillsCortex is a simple portal that helps developers work smarter by linking all your tools, setting clear rules, and slashing repetitive tasks. It speeds up onboarding, updates old code, and fixes issues fast. Over 100 big companies use it to save time and get better results.
-
5
Reliable PostgreSQL Backup & Restore
Reliable PostgreSQL backup and restore
pgBackRest aims to be a reliable, easy-to-use backup and restore solution that can seamlessly scale up to the largest databases and workloads by utilizing algorithms that are optimized for database-specific requirements. Compression is usually the bottleneck during backup operations but, even with now-ubiquitous multi-core servers, most database backup solutions are still single-process. pgBackRest solves the compression bottleneck with parallel processing. Utilizing multiple cores for compression makes it possible to achieve 1TB/hr raw throughput even on a 1Gb/s link. ... -
6
Google Highway
Performance-portable, length-agnostic SIMD with runtime dispatch
Google Highway is a high-performance C++ library designed to provide portable SIMD (Single Instruction, Multiple Data) vectorization across multiple CPU architectures while maintaining predictable and efficient behavior. It abstracts low-level vector intrinsics into a consistent API that maps closely to hardware instructions, allowing developers to write high-performance code without relying heavily on compiler auto-vectorization. Highway enables the same source code to run across different... -
7
Barman
Backup and Recovery Manager for PostgreSQL
Barman (Backup and Recovery Manager) is an enterprise-grade tool for managing backups and disaster recovery of PostgreSQL databases. It supports both full and incremental backups, Point-In-Time Recovery (PITR), and remote backup via SSH. Barman is widely used by DBAs to ensure secure, reliable, and consistent backups in production environments. -
8
ClickHouse
A fast open-source OLAP database management system
ClickHouse® is a fast, open-source column-oriented database management system that can generate analytical data reports through SQL queries in real time. According to several independent benchmarks, it far exceeds other comparable column-oriented database management systems, working even up to 1000 times faster. It is able to process hundreds of millions to more than a billion rows and tens of gigabytes of data per single server per second. Apart from its blazing speed, ClickHouse is highly... -
9
LinDB
LinDB is a scalable, high performance, high availability database
LinDB is a scalable, high-performance, high-availability distributed time series database. A single server could easily support more than one million write TPS; With fundamental techniques like efficient compression storage and parallel computing, LinDB delivers highly optimized query performance. The multi-channel replication protocol supports any amount of nodes, and ensures the system's availability. Schema-free multi-dimensional data model with Metric, Tags, and Fields; The LinQL is flexible yet handy for real-time data analytics. Horizontal scalable is made simple by adding more new broker and storage nodes without too much thinking and manual operations. ... -
Deliver trusted data with dbtData teams use dbt to codify business logic and make it accessible to the entire organization—for use in reporting, ML modeling, and operational workflows.
-
10
LocustDB
Massively parallel, high performance analytics database
An experimental analytics database aiming to set a new standard for query performance and storage efficiency on commodity hardware. See How to Analyze Billions of Records per Second on a Single Desktop PC and How to Read 100s of Millions of Records per Second from a Single Disk for an overview of current capabilities. Download the latest binary release, which can be run from the command line on most x64 Linux systems, including Windows Subsystem for Linux. When loading .csv or .csv.gz files... -
11
ERNIE
The official repository for ERNIE 4.5 and ERNIEKit
...The project also emphasizes optimization techniques for large-scale training, including mixed-precision and hybrid-parallel strategies that are commonly needed for multi-node GPU clusters. In addition to training, it includes guidance and example materials intended to help developers adopt ERNIE models for real product scenarios rather than only research demonstrations. -
12
OpenAS2
AS2 1.1 server implementation in Java
OpenAS2 is a java-based implementation of the EDIINT AS2 standard. It is intended to be used as a server. It is extremely configurable and supports a wide variety of signing and encryption algorithms. Supports very high traffic volume allowing parallel processing of files per partner. SUPPORT: Please use the Github issues and Discussions channels here: https://github.com/OpenAS2/OpenAs2App/ Requirements: - Java 11 or newer (tested with the LTS versions of Java up to 21) ... -
13
ADAMS
ADAMS is a workflow engine for building complex knowledge workflows.
ADAMS is a flexible workflow engine aimed at quickly building and maintaining data-driven, reactive workflows, easily integrated into business processes. Instead of placing operators on a canvas and manually connecting them, a tree structure and flow control operators determine how data is processed (sequentially/parallel). This allows rapid development and easy maintenance of large workflows, with hundreds or thousands of operators. Operators include machine learning (WEKA, MOA, MEKA)... -
14
Sqlite Index Blaster
Create huge Sqlite indexes at breakneck speeds
SQLite Blaster is an advanced SQLite extension that enhances database performance by enabling multi-threading, data compression, and memory optimizations. It is designed for applications that require fast local storage with improved query efficiency. -
15
Crunch
Insane(ly slow but wicked good) PNG image optimization
Crunch is an image compression tool for lossy PNG image file optimization. Using a combination of selective bit depth, color palette reduction and color type, as well as zopfli DEFLATE compression algorithm encoding that employs the pngquant and zopflipng PNG optimization tools, Crunch is effectively able to optimize and compress images with minimal decrease in image quality. While it may produce file size gains larger than those produced by lossless approaches, the impact on image quality... -
16
XZ
Multi-core, cross-platform XZ compression tool written in Go
Parallel implementation of XZ for modern multi-core machines written in Go. This project is licensed under the ISC License. Copyright © 2020-2021 ALBANESE Research Lab Source code: https://github.com/pedroalbanese/xz-2 Visit: http://albanese.atwebpages.com -
17
Parallel MILC
Parallel decompression for the MILC algorithm
The MILC compression has been developed specifically for medical images and proven to be effective. This algorithm is a parallel version for the decompression phase, meant to exploit the parallel computing potential of the modern hardware. Developed with OpenCL framework (version 1.2), runs on CPU and GPU OpenCL devices. -
18
Icarus Scene Engine
3D .NET engine for Windows, Linux, Mac
Icarus Scene Engine is a cross-platform, CLS-compliant 3D framework, integrating open source APIs into a cohesive all-open-source, all .NET solution. For Windows, MacOSX & Linux. Uses OpenTK, OpenGL, OpenAL APIs for graphics and audio. Runs on Mono/.NET. Links to FFMpeg for streaming video, and FreeType as a temporary conversion mod to bring fonts into a native XML format. Binary builds of Icarus will be infrequent, roughly every 18 months or so, the GIT code updates regularly on a... -
19
COMRAD-MPI
Compression of Large Genomic Datasets using Parallel Computing
COMRAD-MPI is a parallel computing algorithm for reducing the computational time for compressing the large genomic data sets based on COMRAD algorithm. It captures the long range repeat redundancies in large genomes there by providing a way to compress the large DNA data set. Three stages- Substitution, Clean up and Huffman encoding have been parallelized using message passing library. -
20
Parallel DPCM
C# Parallel implementation of DPCM alogrithm
An implementation of the DPCM algorithm for compressing and decompressing an avi video file. It shows the compression size, as it is saved in a txt file near the avi original file. The decompression results in another avi file, saved near the original file. If the width and height of the frames aren't a multiple of 16, they are completed with black pixels. The pixel macroblocks of 16x16 are generated on separate threads for each frame and they are replaced on frames at decompression... -
21
pixz
Parallel compression utility
New releases of pixz live at the GitHub page: https://github.com/vasi/pixz/releases -
22
dspTransfer
complete file transfer command line client
dspFileTransfer can upload and download files using ftp, sftp (ftp over ssh), http and https. The command line utility offers many advanced features such as upload or download a file only if the file is newer, parallel downloads etc. In addition the program can perform compression, decompression, encryption and decryption of processed files. The program is a part of batch file processing suite called Data Stream Processor. For full download and support go to www.datastreamprocessor.com. To use the transfer service on its you need to specify '-nodbuse' option to the program so that it doesn't try use DSP db. -
23
LauraDCT
parallel and non-parallel implementation in Java of the Discreet Cosin
A parallel and non-parallel implementation in Java of the Discrete Cosine Transform, which allows for "lossy" compression of image data. The purpose for uploading both implementations was to provide a comparison between the time required to process the blocks of pixels both parallel and and non-parallel model. -
24
fpsc is a Fast Parallel Stream Compressor. It can use multiple CPUs to compress (gzip or bzip2) a data stream. (from stdin to stdout) It is built to achieve maximum throughput, everything else (especially memory usage) can be sacrified.
-
25
czip is a unix based utility written for the cell broadband engine. It allows its users to compress files in gzip format. It uses Jean-loup Gailly's deflate algorithm and is based in part on pigz, Mark Adlers parallel version of zlib.
