目录
- cutlerFix demo's label names1年前
- datasetsInitial commit2年前
- docsAdd videocutler1年前
- maskcutvideocutler1年前
- third_partyInitial commit2年前
- videocutlervideocutler1年前
- .gitignorevideocutler1年前
- .gitmodulesInitial commit2年前
- CODE_OF_CONDUCT.mdInitial commit2年前
- CONTRIBUTING.mdInitial commit2年前
- INSTALL.mdInitial commit2年前
- LICENSEUpdate LICENSE1年前
- README.mdvideocutler1年前
- cog.yamlmove predict.py2年前
- requirements.txtInitial commit2年前
邀请码
Cut and Learn for Unsupervised Image & Video Object Detection and Instance Segmentation
Cut-and-LEaRn (CutLER) is a simple approach for training object detection and instance segmentation models without human annotations. It outperforms previous SOTA by 2.7 times for AP50 and 2.6 times for AR on 11 benchmarks.
[
project page] [arxiv] [colab] [bibtex]Unsupervised video instance segmentation (VideoCutLER) is also supported. We demonstrate that video instance segmentation models can be learned without using any human annotations, without relying on natural videos (ImageNet data alone is sufficient), and even without motion estimations! The code is available here.
[
code] [PDF] [arxiv] [bibtex]Features
Installation
See installation instructions.
Dataset Preparation
See Preparing Datasets for CutLER.
Method Overview
Cut-and-Learn has two stages: 1) generating pseudo-masks with MaskCut and 2) learning unsupervised detectors from pseudo-masks of unlabeled data.1. MaskCut
MaskCut can be used to provide segmentation masks for multiple instances of each image.
MaskCut Demo
Try out the MaskCut demo using Colab (no GPU needed):
Try out the web demo:
(thanks to @hysts!)
If you want to run MaskCut locally, we provide
demo.pythat is able to visualize the pseudo-masks produced by MaskCut. Run it with:We give a few demo images in maskcut/imgs/. If you want to run demo.py with cpu, simply add “–cpu” when running the demo script. For imgs/demo4.jpg, you need to use “–N 6” to segment all six instances in the image. Following, we give some visualizations of the pseudo-masks on the demo images.
Generating Annotations for ImageNet-1K with MaskCut
To generate pseudo-masks for ImageNet-1K using MaskCut, first set up the ImageNet-1K dataset according to the instructions in datasets/README.md, then execute the following command:
As the process of generating pseudo-masks for all 1.3 million images in 1,000 folders takes a significant amount of time, it is recommended to use multiple runs. Each run should process the pseudo-mask generation for a smaller number of image folders by setting “–num-folder-per-job” and “–job-index”. Once all runs are completed, you can merge all the resulting json files by using the following command:
The “–num-folder-per-job”, “–fixed-size”, “–tau” and “–N” of merge_jsons.py should match the ones used to run maskcut.py.
We also provide a submitit script to launch the pseudo-mask generation process with multiple nodes.
After that, you can use “merge_jsons.py” to merge all these json files as described above.
2. CutLER
Inference Demo for CutLER with Pre-trained Models
Try out the CutLER demo using Colab (no GPU needed):
Try out the web demo:
(thanks to @hysts!)
Try out Replicate demo and the API:
If you want to run CutLER demos locally,
model_zoo/configs/CutLER-ImageNet/cascade_mask_rcnn_R_50_FPN.yaml.demo.pythat is able to demo builtin configs. Run it with: ``` cd cutler python demo/demo.py –config-file model_zoo/configs/CutLER-ImageNet/cascade_mask_rcnn_R_50_FPN_demo.yaml \–input demo/imgs/*.jpg
[–other-options] –opts MODEL.WEIGHTS /path/to/cutler_w_cascade_checkpoint
cd cutler export DETECTRON2_DATASETS=/path/to/DETECTRON2_DATASETS/ python train_net.py –num-gpus 8
–config-file model_zoo/configs/CutLER-ImageNet/cascade_mask_rcnn_R_50_FPN.yaml
cd cutler sbatch tools/train-1node.sh
–config-file model_zoo/configs/CutLER-ImageNet/cascade_mask_rcnn_R_50_FPN.yaml
MODEL.WEIGHTS /path/to/dino/d2format/model
OUTPUT_DIR output/
python train_net.py –num-gpus 8
–config-file model_zoo/configs/CutLER-ImageNet/cascade_mask_rcnn_R_50_FPN.yaml
–test-dataset imagenet_train
–eval-only TEST.DETECTIONS_PER_IMAGE 30
MODEL.WEIGHTS output/model_final.pth \ # load previous stage/round checkpoints OUTPUT_DIR output/ # path to save model predictions
python tools/get_self_training_ann.py
–new-pred output/inference/coco_instances_results.json \ # load model predictions –prev-ann DETECTRON2_DATASETS/imagenet/annotations/imagenet_train_fixsize480_tau0.15_N3.json \ # path to the old annotation file. –save-path DETECTRON2_DATASETS/imagenet/annotations/cutler_imagenet1k_train_r1.json \ # path to save a new annotation file. –threshold 0.7
python train_net.py –num-gpus 8
–config-file model_zoo/configs/CutLER-ImageNet/cascade_mask_rcnn_R_50_FPN_self_train.yaml
–train-dataset imagenet_train_r1
MODEL.WEIGHTS output/model_final.pth \ # load previous stage/round checkpoints OUTPUT_DIR output/self-train-r1/ # path to save checkpoints
bash tools/eval.sh
python train_net.py –num-gpus 8
–config-file model_zoo/configs/COCO-Semisupervised/cascade_mask_rcnn_R_50_FPN_{K}perc.yaml
MODEL.WEIGHTS /path/to/cutler_pretrained_model
python train_net.py
–config-file model_zoo/configs/COCO-Semisupervised/cascade_mask_rcnn_R_50_FPN_{K}perc.yaml
–eval-only MODEL.WEIGHTS /path/to/checkpoint_file
@inproceedings{wang2023cut, title={Cut and learn for unsupervised object detection and instance segmentation}, author={Wang, Xudong and Girdhar, Rohit and Yu, Stella X and Misra, Ishan}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={3124–3134}, year={2023} }
@article{wang2023videocutler, title={VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation}, author={Wang, Xudong and Misra, Ishan and Zeng, Ziyun and Girdhar, Rohit and Darrell, Trevor}, journal={arXiv preprint arXiv:2308.14710}, year={2023} }