Download
Perception Models is a state-of-the-art framework developed by Facebook Research for advanced image and video perception tasks. It introduces two primary components: the Perception Encoder (PE) for visual feature extraction and the Perception Language Model (PLM) for multimodal decoding and reasoning. The PE module is a family of vision encoders designed to excel in image and video understanding, surpassing models like SigLIP2, InternVideo2, and DINOv2 across multiple benchmarks. Meanwhile, PLM integrates with PE to power vision-language modeling, achieving results competitive with leading multimodal systems such as QwenVL2.5 and InternVL3, all while being fully reproducible with open data. The project supports a wide range of research applications, from visual recognition and dense prediction to fine-grained multimodal understanding. Additionally, it includes several large-scale open datasets for both image and video perception.
Features
- Combines Perception Encoder (PE) for vision encoding and Perception Language Model (PLM) for multimodal decoding
- State-of-the-art performance in image, video, and vision-language benchmarks
- Open, reproducible models using freely available datasets for transparency
- Multiple PE variants specialized for core, language-aligned, and spatial tasks
- PLM available in 1B, 3B, and 8B parameter sizes for flexible research needs
- Integrated with popular tools such as Hugging Face Transformers, timm, and lmms-eval
Project Samples
