目录
- figuresUse actual squares instead of the text "square" in the depth map comparison2年前
- inputinitial commit5年前
- midasFix usage of Next-ViT2年前
- mobileupdate links2年前
- outputinitial commit5年前
- rosUpdate README.md2年前
- tfupdate links2年前
- weightsVision transformers for dense prediction (#98)3年前
- .gitignoreUpdate to latest version of MiDaS5年前
- Dockerfileupdate link2年前
- LICENSEInitial commit5年前
- README.mdUpdate README.md with links to technical report, video, ZoeDepth and LDM3D1年前
- SECURITY.mdadd SECURITY.md6个月前
- environment.yamlMiDaS v3.12年前
- hubconf.pyUpdate hubconf.py2年前
- run.pyMiDaS v3.12年前
- utils.pyMiDaS v3.12年前
邀请码
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
This repository contains code to compute depth from a single image. It accompanies our paper:
and our preprint:
For the latest release MiDaS 3.1, a technical report and video are available.
MiDaS was trained on up to 12 datasets (ReDWeb, DIML, Movies, MegaDepth, WSVD, TartanAir, HRWSI, ApolloScape, BlendedMVS, IRS, KITTI, NYU Depth V2) with multi-objective optimization. The original model that was trained on 5 datasets (
MIX 5in the paper) can be found here. The figure below shows an overview of the different MiDaS models; the bubble size scales with number of parameters.Setup
weightsfolder:MiDaS 3.1
MiDaS 3.0: Legacy transformer models dpt_large_384 and dpt_hybrid_384
MiDaS 2.1: Legacy convolutional models midas_v21_384 and midas_v21_small_256
Set up dependencies:
optional
For the Next-ViT model, execute
For the OpenVINO model, install
Usage
Place one or more input images in the folder
input.Run the model with
where
<model_type>is chosen from dpt_beit_large_512, dpt_beit_large_384, dpt_beit_base_384, dpt_swin2_large_384, dpt_swin2_base_384, dpt_swin2_tiny_256, dpt_swin_large_384, dpt_next_vit_large_384, dpt_levit_224, dpt_large_384, dpt_hybrid_384, midas_v21_384, midas_v21_small_256, openvino_midas_v21_small_256.The resulting depth maps are written to the
outputfolder.optional
--height. Unsupported height values will throw an error. Note that using this argument may decrease the model accuracy.--square.via Camera
If you want the input images to be grabbed from the camera and shown in a window, leave the input and output paths away and choose a model type as shown above:
The argument
--sideis optional and causes both the input RGB image and the output depth map to be shown side-by-side for comparison.via Docker
Make sure you have installed Docker and the NVIDIA Docker runtime.
Build the Docker image:
Run inference:
This command passes through all of your NVIDIA GPUs to the container, mounts the
inputandoutputdirectories and then runs the inference.via PyTorch Hub
The pretrained model is also available on PyTorch Hub
via TensorFlow or ONNX
See README in the
tfsubdirectory.Currently only supports MiDaS v2.1.
via Mobile (iOS / Android)
See README in the
mobilesubdirectory.via ROS1 (Robot Operating System)
See README in the
rossubdirectory.Currently only supports MiDaS v2.1. DPT-based models to be added.
Accuracy
We provide a zero-shot error $\epsilon_d$ which is evaluated for 6 different datasets (see paper). Lower error values are better. $\color{green}{\textsf{Overall model quality is represented by the improvement}}$ (Imp.) with respect to MiDaS 3.0 DPTL-384. The models are grouped by the height used for inference, whereas the square training resolution is given by the numbers in the model names. The table also shows the number of parameters (in millions) and the frames per second for inference at the training resolution (for GPU RTX 3090):
* No zero-shot error, because models are also trained on KITTI and NYU Depth V2
$\square$ Validation performed at square resolution, either because the transformer encoder backbone of a model does not support non-square resolutions (Swin, Swin2, LeViT) or for comparison with these models. All other validations keep the aspect ratio. A difference in resolution limits the comparability of the zero-shot error and the improvement, because these quantities are averages over the pixels of an image and do not take into account the advantage of more details due to a higher resolution.
Best values per column and same validation height in bold
Improvement
The improvement in the above table is defined as the relative zero-shot error with respect to MiDaS v3.0 DPTL-384 and averaging over the datasets. So, if $\epsilon_d$ is the zero-shot error for dataset $d$, then the $\color{green}{\textsf{improvement}}$ is given by $100(1-(1/6)\sum_d\epsilon_d/\epsilon_{d,\rm{DPT_{L-384}}})$%.
Note that the improvements of 10% for MiDaS v2.0 → v2.1 and 21% for MiDaS v2.1 → v3.0 are not visible from the improvement column (Imp.) in the table but would require an evaluation with respect to MiDaS v2.1 Large384 and v2.0 Large384 respectively instead of v3.0 DPTL-384.
Depth map comparison
Zoom in for better visibility
Speed on Camera Feed
Test configuration
Speed: 22 FPS
Applications
MiDaS is used in the following other projects from Intel Labs:
Changelog
Citation
Please cite our paper if you use this code or any of the models:
@article{Ranftl2021, author = {Ren```
Please cite the technical report for MiDaS 3.1 models:
For ZoeDepth, please use
and for LDM3D
Acknowledgements
Our work builds on and uses code from timm and Next-ViT. We’d like to thank the authors for making these libraries available.
License
MIT License {e} Ranftl and Katrin Lasinger and David Hafner and Konrad Schindler and Vladlen Koltun”, title = “Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer”, journal = “IEEE Transactions on Pattern Analysis and Machine Intelligence”, year = “2022”, volume = “44”, number = “3” }
@article{Ranftl2021, author = {Ren'{e} Ranftl and Alexey Bochkovskiy and Vladlen Koltun}, title = {Vision Transformers for Dense Prediction}, journal = {ICCV}, year = {2021}, }
@article{birkl2023midas, title={MiDaS v3.1 – A Model Zoo for Robust Monocular Relative Depth Estimation}, author={Reiner Birkl and Diana Wofk and Matthias M{"u}ller}, journal={arXiv preprint arXiv:2307.14460}, year={2023} }
@article{bhat2023zoedepth, title={Zoedepth: Zero-shot transfer by combining relative and metric depth}, author={Bhat, Shariq Farooq and Birkl, Reiner and Wofk, Diana and Wonka, Peter and M{"u}ller, Matthias}, journal={arXiv preprint arXiv:2302.12288}, year={2023} }
@article{stan2023ldm3d, title={LDM3D: Latent Diffusion Model for 3D}, author={Stan, Gabriela Ben Melech and Wofk, Diana and Fox, Scottie and Redden, Alex and Saxton, Will and Yu, Jean and Aflalo, Estelle and Tseng, Shao-Yen and Nonato, Fabio and Muller, Matthias and others}, journal={arXiv preprint arXiv:2305.10853}, year={2023} }
Please cite the technical report for MiDaS 3.1 models:
For ZoeDepth, please use
and for LDM3D
Acknowledgements
Our work builds on and uses code from timm and Next-ViT. We’d like to thank the authors for making these libraries available.
License
MIT License {e} Ranftl and Katrin Lasinger and David Hafner and Konrad Schindler and Vladlen Koltun”, title = “Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer”, journal = “IEEE Transactions on Pattern Analysis and Machine Intelligence”, year = “2022”, volume = “44”, number = “3” }
@article{Ranftl2021, author = {Ren```
Please cite the technical report for MiDaS 3.1 models:
For ZoeDepth, please use
and for LDM3D
Acknowledgements
Our work builds on and uses code from timm and Next-ViT. We’d like to thank the authors for making these libraries available.
License
MIT License {e} Ranftl and Alexey Bochkovskiy and Vladlen Koltun}, title = {Vision Transformers for Dense Prediction}, journal = {ICCV}, year = {2021}, }
@article{birkl2023midas, title={MiDaS v3.1 – A Model Zoo for Robust Monocular Relative Depth Estimation}, author={Reiner Birkl and Diana Wofk and Matthias M{"u}ller}, journal={arXiv preprint arXiv:2307.14460}, year={2023} }
@article{bhat2023zoedepth, title={Zoedepth: Zero-shot transfer by combining relative and metric depth}, author={Bhat, Shariq Farooq and Birkl, Reiner and Wofk, Diana and Wonka, Peter and M{"u}ller, Matthias}, journal={arXiv preprint arXiv:2302.12288}, year={2023} }
@article{stan2023ldm3d, title={LDM3D: Latent Diffusion Model for 3D}, author={Stan, Gabriela Ben Melech and Wofk, Diana and Fox, Scottie and Redden, Alex and Saxton, Will and Yu, Jean and Aflalo, Estelle and Tseng, Shao-Yen and Nonato, Fabio and Muller, Matthias and others}, journal={arXiv preprint arXiv:2305.10853}, year={2023} }