詳細記錄YOLACT實例分割ncnn實現(xiàn)
點擊上方“小白學(xué)視覺”，選擇加"星標"或“置頂”
重磅干貨，第一時間送達
端到端一階段完成實例分割 速度快，550x550圖片在TitanXP上號稱達到33FPS 開源代碼，pytorch大法好！
$ python eval.py --trained_model=weights/yolact_resnet50_54_800000.pth --score_threshold=0.15 --top_k=15 --image=test.jpg
0x3 去掉后處理導(dǎo)出onnx
def evalimage(net:Yolact, path:str, save_path:str=None):frame = torch.from_numpy(cv2.imread(path)).cuda().float()batch = FastBaseTransform()(frame.unsqueeze(0))preds = net(batch)torch.onnx._export(net, batch, "yolact.onnx", export_params=True, keep_initializers_as_inputs=True, opset_version=11)
# As of March 10, 2019, Pytorch DataParallel still doesn't support JIT Script Modulesuse_jit = False
后處理部分沒有標準化，每個項目作者的實現(xiàn)細節(jié)也各不相同，比如各種nms和bbox計算方式，ncnn很難用統(tǒng)一的op實現(xiàn)(caffe-ssd因為只有一種版本，所以有實現(xiàn)) 后處理在onnx中會轉(zhuǎn)換成一大坨膠水op，非常瑣碎，在框架中實現(xiàn)效率低下 onnx的大部分膠水op，ncnn不支持或有兼容問題，比如Gather等，無法直接使用
# return self.detect(pred_outs, self)return pred_outs;
$ python eval.py --trained_model=weights/yolact_resnet50_54_800000.pth --score_threshold=0.15 --top_k=15 --image=test.jpg
$ pip install -U onnx --user$ pip install -U onnxruntime --user$ pip install -U onnx-simplifier --user$ python -m onnxsim yolact.onnx yolact-sim.onnx
Graph must be in single static assignment (SSA) form, however '523' has been used as output names multiple times python -m onnxsim --skip-fuse-bn yolact.onnx yolact-sim.onnx
0x5 ncnn模型轉(zhuǎn)換和優(yōu)化
./onnx2ncnn yolact-sim.onnx yolact.param yolact.bin./ncnnoptimize yolact.param yolact.bin yolact-opt.param yolact-opt.bin 0
0x6 手工微調(diào)模型
Convolution Conv_263 1 1 617 619 0=32 1=1 5=1 6=8192 9=1Permute Transpose_265 1 1 619 620 0=3UnaryOp Tanh_400 1 1 814 815 0=16Concat Concat_401 5 1 634 673 712 751 790 816 0=-3Concat Concat_402 5 1 646 685 724 763 802 817 0=-3Concat Concat_403 5 1 659 698 737 776 815 818 0=-3Softmax Softmax_405 1 1 817 820 0=1 1=1
ncnn::Extractor ex = yolact.create_extractor();ncnn::Mat in(550, 550, 3);ex.input("input.1", in);ncnn::Mat b620;ncnn::Mat b816;ncnn::Mat b818;ncnn::Mat b820;ex.extract("620", b620);// 32 x 138x138ex.extract("816", b816);// 4 x 19248ex.extract("818", b818);// 32 x 19248ex.extract("820", b820);// 81 x 19248
Concat Concat_401 5 1 634 673 712 751 790 816 0=0Concat Concat_402 5 1 646 685 724 763 802 817 0=0Concat Concat_403 5 1 659 698 737 776 815 818 0=0
ncnn::Extractor ex = yolact.create_extractor();ncnn::Mat in(550, 550, 3);ex.input("input.1", in);ncnn::Mat maskmaps;ncnn::Mat location;ncnn::Mat mask;ncnn::Mat confidence;ex.extract("619", maskmaps);// 138x138 x 32ex.extract("816", location);// 4 x 19248ex.extract("818", mask);// maskdim 32 x 19248ex.extract("820", confidence);// 81 x 19248
0x7 生成prior
const int conv_ws[5] = {69, 35, 18, 9, 5};const int conv_hs[5] = {69, 35, 18, 9, 5};const float aspect_ratios[3] = {1.f, 0.5f, 2.f};const float scales[5] = {24.f, 48.f, 96.f, 192.f, 384.f};
// make priorboxncnn::Mat priorbox(4, 19248);{float* pb = priorbox;for (int p = 0; p < 5; p++){int conv_w = conv_ws[p];int conv_h = conv_hs[p];float scale = scales[p];for (int i = 0; i < conv_h; i++){for (int j = 0; j < conv_w; j++){// +0.5 because priors are in center-size notationfloat cx = (j + 0.5f) / conv_w;float cy = (i + 0.5f) / conv_h;for (int k = 0; k < 3; k++){float ar = aspect_ratios[k];ar = sqrt(ar);float w = scale * ar / 550;float h = scale / ar / 550;// This is for backward compatability with a bug where I made everything square by accident// cfg.backbone.use_square_anchors:h = w;pb[0] = cx;pb[1] = cy;pb[2] = w;pb[3] = h;pb += 4;}}}}}
0x8 YOLACT全流程實現(xiàn)
預(yù)處理部分
# These are in BGR and are for ImageNetMEANS = (103.94, 116.78, 123.68)STD = (57.38, 57.12, 58.40)
const int target_size = 550;int img_w = bgr.cols;int img_h = bgr.rows;ncnn::Mat in = ncnn::Mat::from_pixels_resize(bgr.data, ncnn::Mat::PIXEL_BGR2RGB, img_w, img_h, target_size, target_size);const float mean_vals[3] = {123.68f, 116.78f, 103.94f};const float norm_vals[3] = {1.0/58.40f, 1.0/57.12f, 1.0/57.38f};in.substract_mean_normalize(mean_vals, norm_vals);
后處理部分
// generate all candidates for each classfor (int i=0; i<num_priors; i++){// find class id with highest score// start from 1 to skip background// ignore background or low scoreif (label == 0 || score <= confidence_thresh)continue;// apply center_size to priorbox with locfloat var[4] = {0.1f, 0.1f, 0.2f, 0.2f};float pb_cx = pb[0];float pb_cy = pb[1];float pb_w = pb[2];float pb_h = pb[3];float bbox_cx = var[0] * loc[0] * pb_w + pb_cx;float bbox_cy = var[1] * loc[1] * pb_h + pb_cy;float bbox_w = (float)(exp(var[2] * loc[2]) * pb_w);float bbox_h = (float)(exp(var[3] * loc[3]) * pb_h);float obj_x1 = bbox_cx - bbox_w * 0.5f;float obj_y1 = bbox_cy - bbox_h * 0.5f;float obj_x2 = bbox_cx + bbox_w * 0.5f;float obj_y2 = bbox_cy + bbox_h * 0.5f;// clip inside image// append object candidate}// merge candidate box for each classfor (int i=0; i<(int)class_candidates.size(); i++){// sort + nms}// sort all result by score// keep_top_k
分割圖生成
0x9 補充學(xué)習(xí)資料
https://link.zhihu.com/?target=https%3A//github.com/Tencent/ncnn
好消息！
小白學(xué)視覺知識星球
開始面向外開放啦??????
下載1：OpenCV-Contrib擴展模塊中文版教程 在「小白學(xué)視覺」公眾號后臺回復(fù)：擴展模塊中文教程，即可下載全網(wǎng)第一份OpenCV擴展模塊教程中文版，涵蓋擴展模塊安裝、SFM算法、立體視覺、目標跟蹤、生物視覺、超分辨率處理等二十多章內(nèi)容。 下載2：Python視覺實戰(zhàn)項目52講 在「小白學(xué)視覺」公眾號后臺回復(fù)：Python視覺實戰(zhàn)項目，即可下載包括圖像分割、口罩檢測、車道線檢測、車輛計數(shù)、添加眼線、車牌識別、字符識別、情緒檢測、文本內(nèi)容提取、面部識別等31個視覺實戰(zhàn)項目，助力快速學(xué)校計算機視覺。 下載3：OpenCV實戰(zhàn)項目20講 在「小白學(xué)視覺」公眾號后臺回復(fù)：OpenCV實戰(zhàn)項目20講，即可下載含有20個基于OpenCV實現(xiàn)20個實戰(zhàn)項目，實現(xiàn)OpenCV學(xué)習(xí)進階。 交流群
歡迎加入公眾號讀者群一起和同行交流，目前有SLAM、三維視覺、傳感器、自動駕駛、計算攝影、檢測、分割、識別、醫(yī)學(xué)影像、GAN、算法競賽等微信群（以后會逐漸細分），請掃描下面微信號加群，備注：”昵稱+學(xué)校/公司+研究方向“，例如：”張三 + 上海交大 + 視覺SLAM“。請按照格式備注，否則不予通過。添加成功后會根據(jù)研究方向邀請進入相關(guān)微信群。請勿在群內(nèi)發(fā)送廣告，否則會請出群，謝謝理解~
評論
圖片
表情