Pretrains self-supervised vision transformer backbones for dense spatial perception using masked boundary modeling.

https://t.co/MjJJbzqZz8 https://t.co/ADsMqpObAo

Robbyant/lingbot-vision

Source: https://github.com/Robbyant/lingbot-vision

🔭 Meet LingBot-Vision! A family of self-supervised ViT backbones for dense spatial perception! 🖼️📐

LingBot-Vision is a family of self-supervised Vision Transformer backbones for dense spatial perception, from ViT-S/16 up to a 1.1B-parameter ViT-g/16. The flagship model is pretrained with masked boundary modeling — a boundary-centric objective that encourages spatially structured patch features while retaining strong semantic representations.

_{Boundary-centric masked modeling. Each row shows the input image, the PCA projection of frozen patch tokens, teacher-discovered boundary tokens, and cosine-similarity maps from selected boundary-token queries. The features capture semantic grouping and geometric structure at the same time.}

LingBot-Vision learns boundaries, shapes, and semantic regions all together, making it a drop-in visual encoder for dense downstream tasks:

• 🎨 Dense feature visualization — PCA maps of frozen patch tokens reveal coherent object regions and crisp boundaries
• 📏 Depth estimation — frozen patch tokens expose spatial structure to lightweight dense readouts
• 🧩 Semantic segmentation — boundary-faithful features align region transitions with object contours
• 🎬 Video object segmentation — training-free token matching and label propagation with frozen features
• 🤖 Depth completion — LingBot-Vision is the visual encoder initialization for LingBot-Depth 2.0 (see below)

🌊 Meet LingBot-Depth 2.0

By simply replacing the encoder with LingBot-Vision at the ViT-L/16 and ViT-g/16 scales, and scaling the curated RGB-D training corpus from 3M to 150M samples, LingBot-Depth 2.0 achieves substantial performance gains over the previous and other system, as detailed in the technical report.

_{LingBot-Depth 2.0 on mirror and glass scenes. Each group shows input RGB, raw sensor depth, refined depth, and refined point clouds. Raw depth is missing on difficult surfaces such as window panes, glass balustrades, and reflective floors. LingBot-Depth 2.0 completes these regions as stable, contiguous surfaces across frames.}

📦 Model Zoo

We train a ViT-g/16 teacher with roughly 1.1B parameters and distill ViT-L, ViT-B, and ViT-S backbones from it for inference and downstream use. Full training and evaluation details are covered in the technical report.

All released weights are backbone-only .pt checkpoints, stored as model.pt in each model repository (see the full Hugging Face collection):

Config files are packaged under lingbot_vision/configs/ and selected automatically by load_pretrained_backbone.

🔧 Installation

Requirements: Python ≥ 3.10 · PyTorch ≥ 2.0 · CUDA-capable GPU (recommended for large-model inference)

1. Clone the repository

git clone https://github.com/robbyant/lingbot-vision.git
cd lingbot-vision

2. Create a conda environment

conda create -n lingbot-vision python=3.10 -y
conda activate lingbot-vision

3. Install lingbot-vision

python -m pip install -r requirements.txt
python -m pip install -e .

🚀 Quick Start

Load LingBot-Vision Pretrains with PyTorch

The model is automatically downloaded from Hugging Face on first use. This example uses the small model for a lightweight smoke run; large is the default variant, and giant is available as the largest backbone.

import torch

from lingbot_vision import load_pretrained_backbone, extract_patch_tokens, load_image

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

# Downloads model.pt from robbyant/lingbot-vision-vit-small.
backbone, embed_dim = load_pretrained_backbone(
    variant="small",
    device=device,
    dtype=dtype,
)

img_norm, _, _ = load_image(
    "examples/example.png",
    size=512,
    patch_size=backbone.patch_size,
    mode="square",
)
patch_tokens, patch_grid = extract_patch_tokens(backbone, img_norm, device, dtype)

print(patch_tokens.shape, patch_grid, embed_dim)
# torch.Size([1, 1024, 384]) (32, 32) 384

patch_tokens has shape [B, H * W, C], where H and W are the patch-grid dimensions. variant can be giant, large, base, or small; if omitted, it defaults to large. You can also pass a local directory or an explicit Hugging Face model repo to load_pretrained_backbone.

Run the PCA demo

Download a backbone checkpoint from Hugging Face (or ModelScope, once available), then run:

./scripts/run_pca_demo.sh \
  --config-file lingbot_vision/configs/lbot_vision_vitl.yaml \
  --ckpt /path/to/model.pt \
  --input examples/example.png \
  --out outputs/pca_demo \
  --size 512 \
  --mode square \
  --dtype bf16

Images are loaded as RGB, resized according to --size and --mode, aligned to the model patch size, and normalized with ImageNet statistics. The demo maps the top three PCA components of the patch tokens to RGB and writes both PCA-only and input/PCA panel visualizations to the output directory. Use --dtype fp32 --device cpu for CPU-only inference.

Checkpoint format

Released checkpoints are .pt files containing backbone weights only — no optimizer states, projection heads, or training-time boundary heads. The loader accepts a raw state dict or a dictionary with a backbone entry:

state_dict
{"backbone": state_dict}

If checkpoint keys are prefixed with backbone., the loader strips the prefix automatically.

📖 Citation

@article{lingbot-vision2026,
  title={Vision Pretraining for Dense Spatial Perception},
  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},
  journal={arXiv preprint arXiv:2607.05247},
  year={2026}
}

📜 License

This project is released under the Apache License 2.0. See LICENSE for details.

🙏 Acknowledgments

LingBot-Vision is part of the LingBot spatial perception effort. We thank DINOv2 and DINOv3 for their contributions to self-supervised learning.

📮 Contact

For questions, discussions, or collaborations:

• Issues: Open an issue on GitHub
• Email: Contact Zelin Fu ([email protected]) or Nan Xue ([email protected])