Search Results for "server benchmark"

mistral.rs is a fast and flexible LLM inference engine implemented in Rust, designed to run and serve modern language models with an emphasis on performance and practical deployment. It provides multiple entry points for developers, including a CLI for running models locally and an HTTP server that exposes an OpenAI-compatible API surface for easy integration with existing clients. The project includes hardware-aware tooling that can benchmark a system and choose sensible quantization and device-mapping strategies, helping users get strong performance without manual tuning. It also supports serving multiple models from the same server process, enabling routing or quick switching between models depending on workload needs. ...

A secure, stable and high-performance reverse proxy for NAT traversal, written in Rust. rathole, like frp and ngrok, can help to expose the service on the device behind the NAT to the Internet, via a server with a public IP. High Performance Much higher throughput can be achieved than frp, and more stable when handling a large volume of connections. Low Resource Consumption Consumes much fewer memory than similar tools. See Benchmark. The binary can be as small as ~500KiB to fit the constraints of devices, like embedded devices as routers. ...