Show HN: Run TRELLIS.2 Image-to-3D generation natively on Apple Silicon

Hacker News Top Tools
image-to-3d apple-silicon pytorch-mps open-source mesh-generation mac-native

Summary

A community port of Microsoft's TRELLIS.2 image-to-3D generation model to run natively on Apple Silicon Macs via PyTorch MPS, eliminating the need for NVIDIA GPUs and generating high-quality 3D meshes with 400K+ vertices in ~3.5 minutes on M4 Pro.

I ported Microsoft&#x27;s TRELLIS.2 (4B parameter image-to-3D model) to run on Apple Silicon via PyTorch MPS. The original requires CUDA with flash_attn, nvdiffrast, and custom sparse convolution kernels: none of which work on Mac.<p>I replaced the CUDA-specific ops with pure-PyTorch alternatives: a gather-scatter sparse 3D convolution, SDPA attention for sparse transformers, and a Python-based mesh extraction replacing CUDA hashmap operations. Total changes are a few hundred lines across 9 files.<p>Generates ~400K vertex meshes from single photos in about 3.5 minutes on M4 Pro (24GB). Not as fast as H100 (where it takes seconds), but it works offline with no cloud dependency.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;shivampkumar&#x2F;trellis-mac" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;shivampkumar&#x2F;trellis-mac</a>
Original Article
View Cached Full Text

Cached at: 04/20/26, 02:44 PM

shivampkumar/trellis-mac

Source: https://github.com/shivampkumar/trellis-mac

TRELLIS.2 for Apple Silicon

Run TRELLIS.2 image-to-3D generation natively on Mac.

This is a port of Microsoft’s TRELLIS.2 — a state-of-the-art image-to-3D model — from CUDA-only to Apple Silicon via PyTorch MPS. No NVIDIA GPU required.

Results

Generates 400K+ vertex meshes from single images in ~3.5 minutes on M4 Pro.

Output includes textured OBJ and GLB files with PBR materials, ready for use in 3D applications.

Example (nearest-neighbor sampling, trilinear interpolation WIP)

Input → Generated 3D mesh (424K vertices, 858K triangles)

Requirements

  • macOS on Apple Silicon (M1 or later)
  • Python 3.11+
  • 24GB+ unified memory recommended (the 4B model is large)
  • ~15GB disk space for model weights (downloaded on first run)

Quick Start

# Clone this repo
git clone https://github.com/shivampkumar/trellis-mac.git
cd trellis-mac

# Log into HuggingFace (needed for gated model weights)
hf auth login

# Request access to these gated models (usually instant approval):
#   https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m
#   https://huggingface.co/briaai/RMBG-2.0

# Run setup (creates venv, installs deps, clones & patches TRELLIS.2)
bash setup.sh

# Activate the environment
source .venv/bin/activate

# Generate a 3D model from an image
python generate.py path/to/image.png

Output files are saved to the current directory (or use --output to specify a path).

Usage

# Basic usage
python generate.py photo.png

# With options
python generate.py photo.png --seed 123 --output my_model --pipeline-type 512

# All options
python generate.py --help
OptionDefaultDescription
--seed42Random seed for generation
--outputoutput_3dOutput filename (without extension)
--pipeline-type512Pipeline resolution: 512, 1024, 1024_cascade

What Was Ported

TRELLIS.2 depends on several CUDA-only libraries. This port replaces them with pure-PyTorch and pure-Python alternatives:

Original (CUDA)ReplacementPurpose
flex_gemmbackends/conv_none.pySparse 3D convolution via gather-scatter
o_voxel._C hashmapbackends/mesh_extract.pyMesh extraction from dual voxel grid
flash_attnPyTorch SDPAScaled dot-product attention for sparse transformers
cumeshStub (graceful skip)Hole filling, mesh simplification
nvdiffrastStubDifferentiable rasterization (texture export)

Additionally, all hardcoded .cuda() calls throughout the codebase were patched to use the active device instead.

Technical Details

Sparse 3D Convolution (backends/conv_none.py): Implements submanifold sparse convolution by building a spatial hash of active voxels, gathering neighbor features for each kernel position, applying weights via matrix multiplication, and scatter-adding results back. Neighbor maps are cached per-tensor to avoid redundant computation.

Mesh Extraction (backends/mesh_extract.py): Reimplements flexible_dual_grid_to_mesh using Python dictionaries instead of CUDA hashmap operations. Builds a coordinate-to-index lookup table, finds connected voxels for each edge, and triangulates quads using normal alignment heuristics.

Attention (patched full_attn.py): Adds an SDPA backend to the sparse attention module. Pads variable-length sequences into batches, runs torch.nn.functional.scaled_dot_product_attention, then unpads results.

Performance

Benchmarks on M4 Pro (24GB), pipeline type 512:

StageTime
Model loading~45s
Image preprocessing~5s
Sparse structure sampling~15s
Shape SLat sampling~90s
Texture SLat sampling~50s
Mesh decoding~30s
Total~3.5 min

Memory usage peaks at around 18GB unified memory during generation.

Limitations

  • No texture export: Texture baking requires nvdiffrast (CUDA-only differentiable rasterizer). Meshes export with vertex colors only.
  • Hole filling disabled: Mesh hole filling requires cumesh (CUDA). Meshes may have small holes.
  • Slower than CUDA: The pure-PyTorch sparse convolution is ~10x slower than the CUDA flex_gemm kernel. This is the main bottleneck.
  • No training support: Inference only.

License

The porting code in this repository (backends, patches, scripts) is released under the MIT License.

Upstream model weights are subject to their own licenses:

Credits

  • TRELLIS.2 by Microsoft Research — the original model and codebase
  • DINOv3 by Meta — image feature extraction
  • RMBG-2.0 by BRIA AI — background removal

Similar Articles