MODNet webcam portrait matting (safetensors)
The official pre-trained MODNet webcam portrait matting checkpoint as a single
safetensors file. MODNet is a small real-time network that predicts a person matte from an RGB image without a
trimap. The webcam variant uses a MobileNetV2 backbone and was trained for video.
This repository is a format conversion only. Every entry of the original checkpoint keeps its name and value; nothing was retrained, pruned or quantised.
Why this exists
The CorridorKey OBS plugin can use MODNet as a coarse person hint for CorridorKey 1.0. It runs weight-free ONNX graphs and binds the weights in C++. The original checkpoint is a PyTorch pickle on a Google Drive folder, which C++ cannot read and which has no stable download URL.
Source
| Original project | ZHKKKe/MODNet, commit 28165a451e4610c9d77cfdf925a94610bb2810fb |
| Original file | modnet_webcam_portrait_matting.ckpt from the project's official pre-trained models folder (Google Drive) |
| sha256 of the original file | 913b82b66558db39b6286c150f809017d7528c872b156eb14333c9c6cb52108b |
sha256 of model.safetensors |
ff7b42f33371d1ba7be124487ce3586e230708b3bef289742f13fac491d07102 |
model.safetensors holds all 751 entries of the original state_dict, with the module. prefix from training
unchanged. The checkpoint stores the backbone twice, as module.backbone.* and module.lr_branch.backbone.*
(one module under two names). safetensors cannot store shared tensors, so these are two identical copies here.
The conversion script is tools/make_modnet_safetensors.py in the CorridorKey real-time project. It checks the
sha256 of the source file first.
Usage
from safetensors.torch import load_file
from src.models.modnet import MODNet # from github.com/ZHKKKe/MODNet
net = MODNet(backbone_pretrained=False)
sd = {k.removeprefix("module."): v for k, v in load_file("model.safetensors").items()}
net.load_state_dict(sd)
net.eval()
# input: RGB in [0, 1], normalised with (x - 0.5) / 0.5, height and width multiples of 32
# _, _, matte = net(x, True)
License and attribution
The MODNet README states that its code, models and demos are released under the Apache License 2.0. The weights
are redistributed here under the same licence; the licence text is in LICENSE and the changes are
listed in NOTICE. This repository is not affiliated with the MODNet authors.
@InProceedings{MODNet,
author = {Zhanghan Ke and Jiayu Sun and Kaican Li and Qiong Yan and Rynson W.H. Lau},
title = {MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition},
booktitle = {AAAI},
year = {2022},
}