Hi all,
I’ve open-sourced ros2-fast-stereo, a ROS2 (Jazzy) workspace for real-time stereo depth estimation using a TensorRT-accelerated port of NVIDIA’s Fast-FoundationStereo model.
Recently, foundational approaches in downstream perception tasks such as FastFoundationStereo have been closing the accuracy gap with offline, non-real-time stereo methods, but the surrounding integration work needed to actually deploy them on arbitrary camera rigs has lagged behind.
Moreover, most real-time stereo pipelines assume a forward-facing camera pair whose optical axes are already roughly parallel, and often bounce data between CPU and GPU across separate rectification and inference stages. This project relaxes both of those assumptions.
Camera pairs are described through a unified abstraction spanning seven projection models (Pinhole, Double Sphere, fisheye polynomial, Kannala-Brandt 8, equirectangular, Mei, and Omni), all behind one pixelToRay/rayToPixel interface. A “stereo pair” can be two wide-FOV, non-parallel cameras whose optical axes point in different directions, as long as their fields of view overlap; the rectifying rotation that aligns them onto a shared virtual pinhole view is derived automatically from each camera’s calibrated pose.
Once a synchronised frame pair arrives, every stage (GPU remap, normalisation, TensorRT stereo-matching inference, and disparity-to-depth conversion) runs on-device via CUDA, NPP, and TensorRT, with image data crossing the host/device boundary only once per frame in and once out. This keeps CPU-GPU synchronisation off the hot path end-to-end.
The same setup extends to multiple overlapping stereo pairs. I’ve included a quad-stereo configuration (4 pairs / 8 cameras) that runs through a single batch-4 TensorRT engine for omnidirectional depth coverage.
Three packages:
camera_models: the 7-model camera abstraction libraryperception_utils: CPU and CUDA/NPP-ready reprojection map precomputation between arbitrary camera pairsffs_depth: the ROS2 nodes (ffs_depth_node,quad_stereo_node) that rectify on the GPU and run the TensorRT stereo-matching engine, publishing disparity images and colored point clouds
Performance:
| Configuration | Weights | Compute | Mean Inference |
|---|---|---|---|
| Single Stereo | 23-36-37, 8 iters | RTX 5000 Blackwell | 35 ms |
| Quad Stereo | 20-30-48, 4 iters | RTX 5000 Blackwell | 140 ms |
| Single Stereo | 23-36-37, 8 iters | Jetson AGX Orin | 128 ms |
The repo ships a VS Code Dev Container (ROS2 Jazzy, CUDA 13.2, TensorRT 10.16, GPU passthrough) so it should be quick to get running if you have a compatible NVIDIA GPU.
Github link: ros2-fast-stereo
Fisheye Pair → Rectified Stereo Pair → Estimated Depth:
Happy to answer questions on the camera model abstraction, the batch-4 TensorRT setup, or the reprojection pipeline. I hope this may be of interest to the community.