?幾個(gè)攝像頭和雷達(dá)融合的目標(biāo)檢測(cè)方法

點(diǎn)擊左上方藍(lán)字關(guān)注我們

一個(gè)專注于目標(biāo)檢測(cè)與深度學(xué)習(xí)知識(shí)分享的公眾號(hào)

轉(zhuǎn)載自 | 3D視覺(jué)工坊

關(guān)于傳感器融合，特別是攝像頭、激光雷達(dá)和雷達(dá)的前融合和和特征融合，是一個(gè)引人注意的方向。

1 “YOdar: Uncertainty-based Sensor Fusion for Vehicle Detection with Camera and Radar Sensors“， 11，2020

基于不確定性的融合方法。后處理采用gradient boosting，視覺(jué)來(lái)自YOLOv3，雷達(dá)來(lái)自1D segmentation network。

FCN-8 inspired radar network

Image of a radar detection example with four predicted slice bundles

YOdar

2 “Warping of Radar Data into Camera Image for Cross-Modal Supervision in Automotive Applications”，12，2020

將雷達(dá)的range-Doppler (RD) spectrum投射到攝像頭平面。由于設(shè)計(jì)的warping函數(shù)可微分，所以在訓(xùn)練框架下做BP。該warping操作依賴于環(huán)境精確的scene flow，故提出一個(gè)來(lái)自激光雷達(dá)、攝像頭和雷達(dá)的scene flow估計(jì)方法，以提高warping操作精度。實(shí)驗(yàn)應(yīng)用涉及了direction-of-arrival (DoA) estimation, target detection, semantic segmentation 和 estimation of radar power from camera data等。

model pipeline

DRISFwR overview （deep rigid instance scene flow with radar）

Automatic scene flow alignment to Radar data via DRISFwR：

RGB image and RD-map with two vehicles

Scale-space of radar data used in DRISFwR with energy & partial derivative

Power projections

RD-map warping into camera image：

Loss in scale-space：

最后實(shí)驗(yàn)結(jié)果比較：

Qualitative results of target detection on test data examples

Qualitative results of semantic segmentation on test data examples

Overview of the model pipeline for camera based estimators for NN training：

Qualitative results of SNR prediction on test data：

3 "RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization", 2 2021

雷達(dá)目標(biāo)檢測(cè)網(wǎng)絡(luò)RODNet，但訓(xùn)練是通過(guò)一個(gè)攝像頭-雷達(dá)監(jiān)督算法，無(wú)需標(biāo)注，可實(shí)現(xiàn)射頻（RF）圖像的實(shí)時(shí)目標(biāo)檢測(cè)。原始毫米波雷達(dá)信號(hào)轉(zhuǎn)換為range-azimuth坐標(biāo)的RF圖像；RODNet預(yù)測(cè)雷達(dá)FoV的目標(biāo)似然性。兩個(gè)定制的模塊M-Net和temporal deformable convolution分別處理multi-chirp merging信息以及目標(biāo)相對(duì)運(yùn)動(dòng)。訓(xùn)練中采用camera-radar fusion (CRF) 策略，另外還建立一個(gè)新數(shù)據(jù)集CRUW1。

cross-modal supervision pipeline for radar object detection in a teacher-student platform

workflow of the RF image generation from the raw radar signals

The architecture and modules of RODNet

Three teacher’s pipelines for cross-model supervision

temporal inception convolution layer

4 “Radar Camera Fusion via Representation Learning in Autonomous Driving”，4，2021

重點(diǎn)討論data association問(wèn)題。而rule-based association methods問(wèn)題較多，故此討論radar-camera association via deep representation learning 以開發(fā)特征級(jí)的交互和全局推理。將檢測(cè)結(jié)果轉(zhuǎn)換成圖像通道，和原圖像一起送入一個(gè)深度CNN模型，即AssociationNet。另外，設(shè)計(jì)了一個(gè)loss sampling mechanism 和 ordinal loss 來(lái)克服不完美的標(biāo)注困難，確保一個(gè)類似人工的推理邏輯。

associations between radardetections (radar pins) and camera detections (2D bounding boxes).

AssociationNet

architecture of the neural network

process of obtaining final associationsfrom the learned representation vectors

illustration of radar pins, bounding boxes, and association relationships under BEV perspective

the red solid lines represent the true-positive associations; and the pink solid lines represent predicted positive associations but labeled as uncertain in the ground-truth

The added green lines represent the false-positive predictions; and the added black lines represent the false-negative predictions

本文僅做學(xué)術(shù)分享，如有侵權(quán)，請(qǐng)聯(lián)系刪文。

END

雙一流大學(xué)研究生團(tuán)隊(duì)創(chuàng)建，專注于目標(biāo)檢測(cè)與深度學(xué)習(xí)，希望可以將分享變成一種習(xí)慣！

整理不易，點(diǎn)贊三連↓