WebRTC 實(shí)戰(zhàn): QT for Windows 多人音視頻通話
簡(jiǎn)介
在經(jīng)過(guò)前面幾篇文章對(duì) WebRTC ?的描述，相信已經(jīng)不需再過(guò)多對(duì)它介紹了。前面幾篇文章我們實(shí)現(xiàn)了 Web 、Android 端的音視頻通話項(xiàng)目，該篇我們使用 QT UI 框架搭建 Windows 端的多 P2P 音視頻通話實(shí)戰(zhàn)項(xiàng)目。
項(xiàng)目地址:https://github.com/yangkun19921001/OpenRTCClient/tree/develop/examples/p2ps/p2ps
最終與 Android、Web 端運(yùn)行后的效果如下:
img_v2_2460f4ab-2447-4694-9771-45d5846f471g環(huán)境搭建
1. QT 環(huán)境準(zhǔn)備
首選我們?nèi)ハ螺d QT 6.6.0 最新版本，下載地址為: https://www.qt.io/download ,安裝好后選擇 cmake 進(jìn)行構(gòu)建項(xiàng)目
2. 信令服務(wù)器準(zhǔn)備
2.1 clone 信令服務(wù)器代碼
git?clone?https://github.com/yangkun19921001/OpenRTCProject.git
2.2 啟動(dòng)信令服務(wù)器
cd?p2ps/server
//修改你自己的證書(shū)
const?server?=?process.env.HTTPS?===?'true'???http.createServer({
????key?:?fs.readFileSync('./cert/rtcmedia.top.key'),
????cert:?fs.readFileSync('./cert/rtcmedia.top_bundle.pem')
??},?app):?http.createServer(app);
//執(zhí)行啟動(dòng)命令
./server_run.js
./proxy_server_run.js
//查看是否啟動(dòng)成功
lsof?-i:8880?和?443
3. webrtc 靜態(tài)庫(kù)準(zhǔn)備
- clone webrtc(m98) develop 代碼
git?clone?https://github.com/yangkun19921001/OpenRTCClient.git
- 按照 README 進(jìn)行編譯，編譯完成后拿到靜態(tài)庫(kù) build/win/debug/obj/webrtc.lib
4. socketio-client-cpp 2.0.0 靜態(tài)庫(kù)準(zhǔn)備
按照官方文檔進(jìn)行編譯，可參考:https://github.com/socketio/socket.io-client-cpp/blob/2.0.0/INSTALL.md#with-cmake
5. QT cmake 編寫
上序 4 步如果都準(zhǔn)備好，那么就可以通過(guò) cmake 進(jìn)行將其依賴進(jìn)來(lái)，如下所示:
cmake_minimum_required(VERSION 3.5)
project(p2ps VERSION 0.1 LANGUAGES CXX)
set(CMAKE_AUTOUIC ON)
set(CMAKE_AUTOMOC ON)
set(CMAKE_AUTORCC ON)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(BUILD_TYPE debug)
if(MSVC)
if(CMAKE_BUILD_TYPE MATCHES Debug)
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MTd")
set(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} /MTd")
elseif(CMAKE_BUILD_TYPE MATCHES Release)
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /MT")
set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} /MT")
#set(BUILD_TYPE release)
endif()
endif()
set(WEBRTC_THIRD_PARTY_PATH ${CMAKE_CURRENT_SOURCE_DIR}/../../../webrtc/third_party)
set(LIBWEBRTC_INCLUDE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/../../../webrtc)
set(LIBWEBRTC_BINARY_PATH ${CMAKE_CURRENT_SOURCE_DIR}/../../../build_system/build/win/x64/${BUILD_TYPE}/obj)
set(DEPS_ROOT_PATH ${CMAKE_CURRENT_SOURCE_DIR}/deps)
set(SOCKET_IO_BINARY_PATH ${DEPS_ROOT_PATH}/socketio/win/x64/${BUILD_TYPE})
set(SOCKET_IO_INCLUDE_PATH ${DEPS_ROOT_PATH}/socketio/include)
set(JSONCPP_SOURCE
"${WEBRTC_THIRD_PARTY_PATH}/jsoncpp/source/src/lib_json/json_reader.cpp"
"${WEBRTC_THIRD_PARTY_PATH}/jsoncpp/source/src/lib_json/json_tool.h"
"${WEBRTC_THIRD_PARTY_PATH}/jsoncpp/source/src/lib_json/json_value.cpp"
"${WEBRTC_THIRD_PARTY_PATH}/jsoncpp/source/src/lib_json/json_writer.cpp"
)
target_include_directories(${PROJECT_NAME} PUBLIC
"${LIBWEBRTC_INCLUDE_PATH}"
"${LIBWEBRTC_INCLUDE_PATH}/third_party/abseil-cpp"
"${LIBWEBRTC_INCLUDE_PATH}/third_party/jsoncpp/source/include"
"${LIBWEBRTC_INCLUDE_PATH}/third_party/jsoncpp/generated"
"${LIBWEBRTC_INCLUDE_PATH}/third_party/libyuv/include"
"${SOCKET_IO_INCLUDE_PATH}"
)
add_definitions(
#webrtc & qt 沖突
-DQT_DEPRECATED_WARNINGS
-DQT_NO_KEYWORDS
#jsoncpp
-DJSON_USE_EXCEPTION=0
-DJSON_USE_NULLREF=0
#socketio
#-DSIO_TLS
-DUSE_AURA=1
-D_HAS_EXCEPTIONS=0
-D__STD_C
-D_CRT_RAND_S
-D_CRT_SECURE_NO_DEPRECATE
-D_SCL_SECURE_NO_DEPRECATE
-D_ATL_NO_OPENGL
-D_WINDOWS
-DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS
-DPSAPI_VERSION=2
-DWIN32
-D_SECURE_ATL
-DWINUWP
-D__WRL_NO_DEFAULT_LIB__
# -DWINAPI_FAMILY=WINAPI_FAMILY_PC_APP
-DWIN10=_WIN32_WINNT_WIN10
-DWIN32_LEAN_AND_MEAN
-DNOMINMAX
-D_UNICODE
-DUNICODE
-DNTDDI_VERSION=NTDDI_WIN10_RS2
-D_WIN32_WINNT=0x0A00
-DWINVER=0x0A00
-DDEBUG
-DNVALGRIND
-DDYNAMIC_ANNOTATIONS_ENABLED=0
-DWEBRTC_ENABLE_PROTOBUF=0
-DWEBRTC_INCLUDE_INTERNAL_AUDIO_DEVICE
-DRTC_ENABLE_VP9
-DHAVE_SCTP
-DWEBRTC_LIBRARY_IMPL
-DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0
-DWEBRTC_WIN
-DABSL_ALLOCATOR_NOTHROW=1
-DHAVE_SCTP
-DWEBRTC_VIDEO_CAPTURE_WINRT)
target_link_libraries(p2ps PRIVATE
...
${SOCKET_IO_BINARY_PATH}/sioclient.lib ${SOCKET_IO_BINARY_PATH}/sioclient_tls.lib
${LIBWEBRTC_BINARY_PATH}/webrtc.lib
winmm.lib iphlpapi.lib wbemuuid.lib secur32.lib advapi32.lib Mmdevapi.lib
Mfuuid.lib msdmo.lib dmoguids.lib wmcodecdspuuid.lib comdlg32.lib dbghelp.lib
dnsapi.lib gdi32.lib msimg32.lib odbc32.lib odbccp32.lib oleaut32.lib shell32.lib
shlwapi.lib user32.lib usp10.lib uuid.lib version.lib wininet.lib winmm.lib
winspool.lib ws2_32.lib delayimp.lib kernel32.lib ole32.lib crypt32.lib
amstrmid.lib strmiids.lib
)
到此，在 QT 項(xiàng)目中的 WebRTC 和 SocketIO 環(huán)境已經(jīng)搭建完成了，這里為什么要搭建 SocketIO 呢？因?yàn)樵谥暗??web 、server、Android 都使用的是 socketio 開(kāi)源庫(kù)來(lái)做的信令通信，正好它又是跨平臺(tái)的，所以就不再更換了。
如何構(gòu)建多人音視頻通話?
要在 QT 中構(gòu)建一個(gè)多人的音視頻通話項(xiàng)目，必定要經(jīng)過(guò)如下幾個(gè)步驟
1. 信令交互協(xié)議
server 端的信令其實(shí)設(shè)計(jì)的很簡(jiǎn)單，就三組信令，如下所示:
join > joined ：加入房間和加入成功的通知 leave > leaved：離開(kāi)房間和離開(kāi)成功的通知 message：交換 offer ,candidate
更加詳細(xì)的流程可以參考下面的流程圖
2. 本地信令交互封裝
首先定義 ISinnalClient.h 抽象接口
namespace?PCS{
class?ISignalClient?{
public:
????enum?class?SignalEvent?{
????????JOINED?=?0,
????????JOIN,
????????LEAVED,
????????LEAVE,
????????MESSAGE
????};
????static?std::string?SignalEventToString(SignalEvent?event)?{
????????switch(event)?{
????????case?SignalEvent::JOINED:??return?"joined";
????????case?SignalEvent::LEAVED:??return?"leaved";
????????case?SignalEvent::JOIN:????return?"join";
????????case?SignalEvent::LEAVE:???return?"leave";
????????case?SignalEvent::MESSAGE:?return?"message";
????????default:?return?"unknown";
????????}
????}
????/*連接服務(wù)端*/
????virtual?bool?connect(const?std::string?url,OnSignalEventListener*?listener)?=?0;
????/*加入會(huì)話*/
????virtual?void?join(const?std::string?roomId)?=?0;
????/*離開(kāi)會(huì)話*/
????virtual?void?leave(const?std::string?roomId)?=?0;
????/*銷毀?client?*/
????virtual?void?release()?=?0;
????/*發(fā)送消息*/
????virtual?void?sendMessage(const?std::string?roomId,?const?std::string?remoteId,?const?std::string?message)?=?0;
????virtual?std::string?getSocketId()?=?0;
????virtual?~ISignalClient()?=?default;
};
}
然后 socketio 根據(jù)對(duì)應(yīng)的 api 去封裝實(shí)現(xiàn)即可，詳細(xì)的使用，可以參考 https://github.com/yangkun19921001/OpenRTCClient/blob/develop/examples/p2ps/p2ps/src/common/SocketIoSignalClientImpl.cpp
3. 多 PeerConnection 管理
多 PeerConnection 管理其實(shí)就是把每次加入房間的 Peer 添加到一個(gè)容器中，管理起來(lái)。這里我們可以定義一個(gè) map 結(jié)構(gòu)進(jìn)行管理，核心 api 如下
3.1 定義一個(gè) PeerConnection 管理結(jié)構(gòu)體
struct?Peer
{
????Peer()?{}
????rtc::scoped_refptr<webrtc::PeerConnectionInterface>?peer_conn_inter_;
????std::unique_ptr<PeerConnectionObserverImpl>?peer_conn_obser_impl_;
????std::unique_ptr<CreateSessionDescriptionObserImpl>?create_offer_sess_des_impl_;
????std::unique_ptr<CreateSessionDescriptionObserImpl>?create_answer_sess_des_impl_;
};
內(nèi)部主要包含每個(gè) PeerConnection 所需要的成員，然后通過(guò) std::map 進(jìn)行管理，如下所示
std::map<std::string?,std::unique_ptr<Peer>>?peers_;
map 中的key 就是連接到房間中的 id
3.2 創(chuàng)建 PeerConnectionFactory
通過(guò)如下核心代碼即可創(chuàng)建出 PeerConnectionFactory
???????peer_connection_factory_?=?webrtc::CreatePeerConnectionFactory(
???????????this->network_thread_.get()?/*?network_thread?*/,
???????????this->worker_thread_.get()?/*?worker_thread?*/,
???????????this->signaling_thread_.get(),?/*?signaling_thread?*/
???????????nullptr?/*?default_adm?*/,
???????????webrtc::CreateBuiltinAudioEncoderFactory(),
???????????webrtc::CreateBuiltinAudioDecoderFactory(),
???????????webrtc::CreateBuiltinVideoEncoderFactory(),
???????????webrtc::CreateBuiltinVideoDecoderFactory(),?nullptr?/*?audio_mixer?*/,
???????????nullptr?/*?audio_processing?*/);
當(dāng)創(chuàng)建成功后，再創(chuàng)建本地音視頻軌道，后續(xù)會(huì)將本地音視頻軌道添加到 PeerConnection 中
//音頻軌道
?audio_track_??=?peer_connection_factory_->CreateAudioTrack(
????????kAudioLabel,?peer_connection_factory_->CreateAudioSource(
????????????cricket::AudioOptions()));
?//視頻軌道
??rtc::scoped_refptr<CameraCapturerTrackSource>?video_device?=
??????CameraCapturerTrackSource::Create(1280,720,30);
??video_track_?=
??????peer_connection_factory_->CreateVideoTrack(kVideoLabel,?video_device);
3.3 創(chuàng)建 PeerConnection
bool?PeerManager::createPeerConnection(const?std::string?&peerId,?const?webrtc::PeerConnectionInterface::RTCConfiguration?&config
???????????????????????????????????????,??OnPeerManagerEvents*?ets)
{
??RTC_LOG(LS_INFO)?<<__FUNCTION__?<<"?peerId:"<<peerId;
??RTC_DCHECK(peer_connection_factory_);
??????std::unique_ptr<PeerConnectionObserverImpl>?peer_conn_obimpl?=
??????????std::make_unique<PeerConnectionObserverImpl>(peerId,ets);
??????auto?peerConnection?=?peer_connection_factory_->CreatePeerConnection(
??????????config,
??????????nullptr,
??????????nullptr,
??????????peer_conn_obimpl.get());
??????if?(peerConnection)?{
??????????auto?peer_ptr?=?std::make_unique<Peer>();
??????????peer_ptr->peer_conn_inter_?=?peerConnection;
??????????peer_ptr->peer_conn_obser_impl_?=std::move(peer_conn_obimpl);
??????????peers_[peerId]?=?std::move(peer_ptr);
??????????if(video_track_)
???????????peerConnection->AddTrack(video_track_,?{?kVideoLabel?});
???????????if(audio_track_)
??????????peerConnection->AddTrack(audio_track_,?{?kAudioLabel?});
??????????return?true;
??????}
??????return?false;
}
由上面的代碼得知，我們通過(guò) peer_connection_factory_->CreatePeerConnection 就可以構(gòu)建一個(gè) PeerConnection ，并把之前創(chuàng)建出來(lái)的本地音視頻軌道添加到 PeerConnection 中，最后我們將構(gòu)建出來(lái)的 PeConn 緩存到 map 中，便于后續(xù)的處理。
3.4 創(chuàng)建 offer
void?PeerManager::createOffer(const?std::string?&peerId,?OnPeerManagerEvents*?ets)
{
??RTC_LOG(LS_INFO)?<<__FUNCTION__?<<"?peerId:"<<peerId;
??auto?it?=?peers_.find(peerId);
??if?(it?!=?peers_.end()?&&?it->second->peer_conn_inter_?!=?nullptr)?{
??????auto?obs?=?std::make_unique<CreateSessionDescriptionObserImpl>(true,peerId,ets);
??????it->second->peer_conn_inter_->CreateOffer(obs.get(),?webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());
??????it->second->create_offer_sess_des_impl_?=?std::move(obs);
??}else?{
??????RTC_LOG(LS_ERROR)?<<?__FUNCTION__?<<??"?peers_?not?found?id:"<<peerId;
??}
}
上面的代碼，首先根據(jù) peerId 從緩存中拿到對(duì)應(yīng)的 PeerConnection ,然后再調(diào)用它內(nèi)部的 api 來(lái)進(jìn)行 CreateOffer
3.5 設(shè)置本地/遠(yuǎn)端 SDP
void?PeerManager::setLocalDescription(const?std::string?&peerId,webrtc::SessionDescriptionInterface?*desc_ptr)
{
??RTC_LOG(LS_INFO)?<<__FUNCTION__?<<"?peerId:"<<peerId;
??auto?pt?=?peers_.find(peerId);
??if?(pt?!=?peers_.end())?{
??????pt->second->peer_conn_inter_->SetLocalDescription(SetSessionDescriptionObserverImpl::Create(),desc_ptr);
??}else?{
??????RTC_LOG(LS_ERROR)?<<?__FUNCTION__?<<??"?peers_?not?found?id:"<<peerId;
??}
}
void?PeerManager::setRemoteDescription(const?std::string?&peerId,webrtc::SessionDescriptionInterface?*desc_ptr)
{
??RTC_LOG(LS_INFO)?<<__FUNCTION__?<<"?peerId:"<<peerId;
??auto?pt?=?peers_.find(peerId);
??if?(pt?!=?peers_.end())?{
??pt->second->peer_conn_inter_->SetRemoteDescription(SetSessionDescriptionObserverImpl::Create(),desc_ptr);
??}else?{
??RTC_LOG(LS_ERROR)?<<?__FUNCTION__?<<??"?peers_?not?found?id:"<<peerId;
??}
}
從緩存中拿到對(duì)應(yīng)的 PeerConnection ,然后設(shè)置本地或遠(yuǎn)端的 sdp 描述信息
3.6 創(chuàng)建 answer
void?PeerManager::createAnswer(const?std::string?&peerId,?OnPeerManagerEvents*?ets)
{
?RTC_LOG(LS_INFO)?<<__FUNCTION__?<<"?peerId:"<<peerId;
??auto?it?=?peers_.find(peerId);
??if?(it?!=?peers_.end()?&&?it->second->peer_conn_inter_?!=?nullptr)?{
??????auto?obs?=?std::make_unique<CreateSessionDescriptionObserImpl>(false,peerId,ets);
??????it->second->peer_conn_inter_->CreateAnswer(obs.get(),?webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());
??????it->second->create_answer_sess_des_impl_?=?std::move(obs);
??}else?{
??????RTC_LOG(LS_ERROR)?<<?__FUNCTION__?<<??"?peers_?not?found?id:"<<peerId;
??}
}
此處與上一步邏輯一樣
3.7 處理 ice
void?PeerManager::handleCandidate(std::string?peerId,?std::unique_ptr<webrtc::IceCandidateInterface>?candidate)
{
??????RTC_LOG(LS_INFO)?<<?__FUNCTION__?<<"?peerId:"<<peerId;
??????auto?pt?=?peers_.find(peerId);
??????if?(pt?!=?peers_.end())?{
??????????pt->second->peer_conn_inter_->AddIceCandidate(
??????????????std::move(candidate),
??????????????[peerId](webrtc::RTCError?error){
??????????????????if?(error.ok())?{
??????????????????????RTC_LOG(LS_INFO)?<<"?peerId:"<<?peerId?<<?"?AddIceCandidate?success.";
??????????????????}?else?{
??????????????????????RTC_LOG(LS_INFO)?<<"?peerId:"<<?peerId?<<?"AddIceCandidate?failed,?error:?"?<<?error.message();
??????????????????}
???????});
????}else?{
??????????RTC_LOG(LS_ERROR)?<<?__FUNCTION__?<<??"?peers_?not?found?id:"<<peerId;
????}
}
也是從緩存中拿到對(duì)應(yīng)的 PeerConnection ，然后將對(duì)方的候選者地址添加進(jìn)去。
4. 房間管理
定義 RTCRoomManager ，實(shí)現(xiàn)信令回調(diào)和PeerManager 回調(diào)，定義的核心 API 如下
namespace?PCS{
class?RTCRoomManager?:?public?OnSignalEventListener,public?OnPeerManagerEvents?{
public:
????RTCRoomManager();
????virtual?~RTCRoomManager();
????//連接服務(wù)器
????void?connect(const?std::string?url,OnRoomStateChangeCallback*?callback);
????//設(shè)置本地軌道的回調(diào)監(jiān)聽(tīng)
????void?setLocalTrackCallback(std::function<void(std::string,int,int,rtc::scoped_refptr<webrtc::VideoTrackInterface>)>?localVideoTrack);
????//加入房間
????void?join(const?std::string?roomId);
????//離開(kāi)房間
????void?leave(const?std::string?roomId);
????//銷毀
????void?release();
????//處理?ui?傳遞過(guò)來(lái)的?消息
????void?onUIMessage(Message?msg);
private:
????//實(shí)現(xiàn)連接成功的處理代碼
????void?onConnectSuccessful()?override;
????//實(shí)現(xiàn)正在連接的處理代碼
????void?onConnecting()?override?;
????//實(shí)現(xiàn)連接錯(cuò)誤的處理代碼
????void?onConnectError(const?std::string&?error)?override?;
????//實(shí)現(xiàn)已加入房間的處理代碼
????void?onJoined(const?std::string&?room,?const?std::string&?id,?const?std::vector<std::string>&?otherClientIds)?override;
????//實(shí)現(xiàn)離開(kāi)房間的處理代碼
????void?onLeaved(const?std::string&?room,?const?std::string&?id)?override?;
????//實(shí)現(xiàn)接收到消息的處理代碼
????void?onMessage(const?std::string&?from,?const?std::string&?to,?const?std::string&?message)?override?;
????//當(dāng)需要添加遠(yuǎn)端的軌道
?????void?OnAddTrack(std::string?peerid,
????????????????????????????rtc::scoped_refptr<webrtc::RtpReceiverInterface>?receiver,
????????????????????????????const?std::vector<rtc::scoped_refptr<webrtc::MediaStreamInterface>>&
????????????????????????????????streams)override;
????//當(dāng)刪除遠(yuǎn)端的軌道
?????void?OnRemoveTrack(std::string?peerid,
???????????????????????????????rtc::scoped_refptr<webrtc::RtpReceiverInterface>?receiver)?override;
?????//?datachannel?消息
?????void?OnDataChannel(std::string?peerid,
???????????????????????????????rtc::scoped_refptr<webrtc::DataChannelInterface>?channel)?override;
?????//ice?消息??????????????????????????
?????void?OnIceCandidate(std::string?peerid,const?webrtc::IceCandidateInterface*?candidate)?override;
?????//offer?or?answer?create?成功
?????void?OnCreateSuccess(bool?offer,std::string?peerid,webrtc::SessionDescriptionInterface*?desc)?override;
?????//offer?or?answer?create?失敗
?????void?OnCreateFailure(bool?offer,std::string?peerid,webrtc::RTCError?error)?override;
private:
????std::unique_ptr<ISignalClient>?socket_signal_client_imp_;
????std::unique_ptr<PeerManager>?peer_manager_;
????OnRoomStateChangeCallback?*?room_state_change_callback_;
????std::function<void(std::string,int,int,rtc::scoped_refptr<webrtc::VideoTrackInterface>)>?local_track_callback_;
????std::string?room_id_;
};
}?//?end?namespace?PCS
這就是核心 API, 它持有 peer_manager_、socket_signal_client_imp_ ，分別是對(duì) PeerConnection 和信令的交互。
比如現(xiàn)在 A 用戶先進(jìn)入房間，B 后進(jìn)入房間，然后對(duì)它們的管理流程是這樣的
- A 用戶連接服務(wù)器 -> 連接成功 ->發(fā)起 join 信令->收到 joined 信令->createPeerConnectionFactory->攝像頭開(kāi)始采集->等待預(yù)覽
- B 用戶連接服務(wù)器 -> 連接成功 ->發(fā)起 join 信令 ->收到j(luò)oined 信令(并帶上了 A 用戶在房間中的信息) ->createPeerConnectionFactory->等待本地預(yù)覽->createPeerConnection(A用戶)
- A 用戶收到 B 用戶 joined 加入房間的信令 -> createOffer -> 設(shè)置本地 SDP -> 發(fā)送 本地 SDP offer 信令到服務(wù)器
- B 用戶收到 服務(wù)器轉(zhuǎn)發(fā)過(guò)來(lái)的 A 用戶的 offer 信令 -> 設(shè)置遠(yuǎn)端的 SDP 信息 ->CreateAnswer(A用戶)->設(shè)置本地 SDP 信息 -> 發(fā)送 answer 消息給 A
- A 收到 B 發(fā)送的 answer 信令消息，調(diào)用 setRemoteDescription ?函數(shù)將 B 的 SDP 設(shè)置進(jìn)去
- A,B 交換 candidate 消息，并將對(duì)方的 candidate 調(diào)用 AddIceCandidate 添加進(jìn)去
- 到這一步后，就等待 onAddTrack 回調(diào)了，下一步就是如何將對(duì)方和本地的畫面進(jìn)行顯示了
5. 如何顯示
在 webrtc 架構(gòu)中，是通過(guò) rtc::VideoSinkInterface<webrtc::VideoFrame> 的 onFrame 虛函數(shù)進(jìn)行通知需要新視頻數(shù)據(jù)的渲染。
我們定義一個(gè) VideoRendererWidget 然后實(shí)現(xiàn) rtc::VideoSinkInterface<webrtc::VideoFrame> 的 onFrame 函數(shù)，如下所示:
namespace?PCS?{
class?VideoRendererWidget?:?public?QOpenGLWidget,?protected?QOpenGLFunctions,
????????????????????????????public?rtc::VideoSinkInterface<webrtc::VideoFrame>
{
????Q_OBJECT
public:
????VideoRendererWidget(std::string?peerId?="",QWidget*?parent?=?nullptr,webrtc::VideoTrackInterface?*track?=nullptr);
????~VideoRendererWidget();
public?Q_SLOTS:
????void?PlayOneFrame();
protected:
????void?initializeGL()?Q_DECL_OVERRIDE;
????void?resizeGL(int?w,?int?h)?Q_DECL_OVERRIDE;
????void?paintGL()?Q_DECL_OVERRIDE;
public:
????//?VideoSinkInterface?implementation
????void?OnFrame(const?webrtc::VideoFrame&?frame)?override;
private:
...
public:
???webrtc::VideoTrackInterface*??video_track_;
};
}
void?VideoRendererWidget::OnFrame(const?webrtc::VideoFrame?&video_frame)
{
????std::lock_guard<std::mutex>?guard(renderer_mutex_);
????rtc::scoped_refptr<webrtc::I420BufferInterface>?buffer(
????????video_frame.video_frame_buffer()->ToI420());
????if?(video_frame.rotation()?!=?webrtc::kVideoRotation_0)?{
????????buffer?=?webrtc::I420Buffer::Rotate(*buffer,?video_frame.rotation());
????}
????int?width?=?buffer->width();
????int?height?=?buffer->height();
????int?ySize?=?width?*?height;
????int?uvSize?=?ySize?/?4;??//?For?each?U?and?V?component
????if(m_nVideoW?==?0?&&?m_nVideoH?==0)
????{
???????if(!peer_id_.empty())??RTC_LOG(LS_INFO)?<<?__FUNCTION__?<<?"?init?w:"<<width<<"?height:"<<height?<<?"?peerId:"?<<peer_id_;
????}
????if(width?!=?m_nVideoW?&&?height?!=?m_nVideoH?&&?video_data_?!=?nullptr)
????{
?????????video_data_?.reset();
????????if(!peer_id_.empty())?RTC_LOG(LS_INFO)?<<?__FUNCTION__?<<?"?change?w:"<<width<<"?height:"<<height?<<?"?peerId:"?<<peer_id_;
????}
????if(video_data_?==?nullptr){
??????video_data_?=?std::make_unique<uint8_t[]>(width?*?height?*?1.5);?//?Use?make_unique?to?allocate?array
??????if(!peer_id_.empty())
??????RTC_LOG(LS_INFO)?<<?__FUNCTION__?<<?"?malloc?Id:"<<peer_id_<<"?width:"?<<?width?<<"?height:"<<height;
????}
????memcpy(video_data_.get(),?buffer->DataY(),?ySize);
????memcpy(video_data_.get()?+?ySize,?buffer->DataU(),?uvSize);
????memcpy(video_data_.get()?+?ySize?+?uvSize,?buffer->DataV(),?uvSize);
????m_nVideoW?=?width;
????m_nVideoH?=?height;
????//?刷新界面,觸發(fā)paintGL接口
????Q_EMIT?PlayOneFrame();
}
當(dāng)我們調(diào)用 PlayOneFrame 時(shí)，就可以通過(guò) QOpenGL 來(lái)進(jìn)行渲染 I420 YUV 數(shù)據(jù)了。
6. 如何管理多個(gè)窗口的創(chuàng)建和銷毀的
可以通過(guò)管理 PeerConnection 那樣管理 VideoRendererWidget ,還是定義一個(gè) map
std::map<std::string,?std::unique_ptr<PCS::VideoRendererWidget>>?video_renderer_widgets_;
根據(jù)對(duì)方的 peerid 來(lái)進(jìn)行緩存窗口，
當(dāng)需要添加窗口時(shí):
void?MainWindow::addVideoRendererWidgetToMainWindow(std::string?id,?std::unique_ptr<PCS::VideoRendererWidget>?renderer)
{
?????const?int?itemsPerRow?=?3;
?????std::unique_ptr<PCS::VideoRendererWidget>?videoRenderer;
?????if(renderer?==?nullptr)
????????videoRenderer?=??std::make_unique<PCS::VideoRendererWidget>();
?????else?{
????????videoRenderer?=?std::move(renderer);
?????}
?????videoRenderer->setSizePolicy(QSizePolicy::Expanding,?QSizePolicy::Expanding);
?????//?計(jì)算新的位置
?????int?count?=?ui->gridLayout->count();
?????int?row?=?count?/?itemsPerRow;
?????int?column?=?count?%?itemsPerRow;
?????//?添加到?gridLayout?中
?????ui->gridLayout->addWidget(videoRenderer.get(),row,column);
?????//videoRenderer->setFixedSize(1280/3,720/3);
?????//?將?widget?添加到?map?中
?????video_renderer_widgets_.insert({id,?std::move(videoRenderer)});
}
當(dāng)需要?jiǎng)h除窗口時(shí):
void?MainWindow::removeVideoRendererWidgetFromMainWindow(std::string?id)
????{
?????//?從?layout?中移除?widget，并刪除它
?????//?查找?widget
?????auto?it?=?video_renderer_widgets_.find(id);
?????if?(it?!=?video_renderer_widgets_.end())
?????{
????????//?從?layout?中移除?widget
????????ui->gridLayout->removeWidget(it->second.get());
????????//?從?map?中移除并刪除?widget
????????video_renderer_widgets_.erase(it);
?????}
????}
總結(jié)
我們通過(guò)簡(jiǎn)短的描述和一些基礎(chǔ)的 api 來(lái)介紹了如何通過(guò) QT webrtc 來(lái)構(gòu)建一個(gè)多人的音視頻通話的項(xiàng)目。由于本人對(duì) QT 不是太熟悉，所以 UI 上還有少許 Bug 。但不影響核心 API 調(diào)用。
到此，通過(guò)本篇文章和之前的幾篇文章我們已經(jīng)實(shí)現(xiàn)了 Web 、Android 、Windows 之前的互通，后續(xù)會(huì)繼續(xù)介紹 WebRTC 源碼分析和實(shí)戰(zhàn)項(xiàng)目的開(kāi)發(fā)。
評(píng)論
圖片
表情