亚洲日韩精品视频在线,综合一和综合二图片小说,欧美性爱-熊猫成人网,综合免费视频,国产性在线电影,在线黄网,一级黄色电影免费观看,久久久久三级视频

深入理解Android音視頻同步機制（一）概述
深入理解Android音視頻同步機制（二）ExoPlayer的avsync邏輯
深入理解Android音視頻同步機制（三）NuPlayer的avsync邏輯
深入理解Android音視頻同步機制（四）MediaSync的使用與原理
深入理解Android音視頻同步機制（五）如何從零開始寫一個音視頻同步的播放器

前面我們分析了三個播放器的av sync邏輯，可以看到他們都各有不同，那么究竟哪種方法可以達到最好的avsync結(jié)果？哪些邏輯是必要的？如果我們想自己從零開始寫一個av同步的播放器，都需要做哪些工作？

首先我們測試了幾個播放器的音視頻同步表現(xiàn)，使用的是syncOne官網(wǎng)的1080p 24fps H264 AAC測試片源，只測試speaker下的結(jié)果，測試結(jié)果如下

下面我們參考cts中的mediacodec使用示例，嘗試著寫出一個avsync結(jié)果與上面結(jié)果接近的播放器

MediaCodecPlayer Demo

demo的地址關(guān)注文末公眾號查看。

首先來看一下整體流程，如下圖所示

總體流程和ExoPlayer基本相似。圖中的doSomeWork是核心大loop，同步地調(diào)用MediaCodc接口，關(guān)鍵的avsync邏輯在drainOutputBuffer方法中實現(xiàn)。

最樸素簡單的同步邏輯

先來總體說一下同步邏輯，然后再詳細看看代碼

Video部分

1.對實際送顯時間的計算

private boolean drainOutputBuffer()

//傳入?yún)?shù)為pts，返回的realTimeUs其實就是pts，并沒有做什么調(diào)整
   long realTimeUs =
       mMediaTimeProvider.getRealTimeUsForMediaTime(info.presentationTimeUs);

   //這里的nowUs代表audio播放的時間
   long nowUs = mMediaTimeProvider.getNowUs(); //audio play time
   //nowUs - realTimeUs代表還有多久該播放這一幀
   long lateUs = nowUs - realTimeUs;
…
    //這里調(diào)用的是沒有timestamp參數(shù)的releaseOutputBuffer方法
    mCodec.releaseOutputBuffer(index, render);

Audio部分

1、current play time的計算

public long getAudioTimeUs()

//這里返回audio的播放時間，就是利用的getPlaybackHeadPosition方法
   int numFramesPlayed = mAudioTrack.getPlaybackHeadPosition();
   return (numFramesPlayed * 1000000L) / mSampleRate;

小結(jié)

就是拿video的pts與audioTrack.getPlaybackHeadPosition方法返回的播放時間進行對比，然后直接調(diào)用沒有timestamp參數(shù)的releaseOutputBuffer的方法進行顯示

詳細的代碼如下

private boolean drainOutputBuffer() {
...
   int index = mAvailableOutputBufferIndices.peekFirst().intValue();
   MediaCodec.BufferInfo info = mAvailableOutputBufferInfos.peekFirst();
…
   //傳入?yún)?shù)為pts，返回的realTimeUs其實就是pts，并沒有做什么調(diào)整，參見1.1
   long realTimeUs =
       mMediaTimeProvider.getRealTimeUsForMediaTime(info.presentationTimeUs);

   //這里的nowUs代表audio播放的時間，參見1.1
   long nowUs = mMediaTimeProvider.getNowUs(); //audio play time
   //nowUs - realTimeUs代表還有多久該播放這一幀
   long lateUs = nowUs - realTimeUs;

   if (mAudioTrack != null) {
       ...
   } else {
       // video
       boolean render;

       if (lateUs < -45000) {
           // too early;如果video來早了45ms，則等待下一次loop
           return false;
       } else if (lateUs > 30000) {
           //默認的丟幀門限是30ms
           Log.d(TAG, "video late by " + lateUs + " us.");
           render = false;
       } else {
           render = true;
           mPresentationTimeUs = info.presentationTimeUs;
       }

       //這里調(diào)用的是沒有timestamp參數(shù)的releaseOutputBuffer方法
       mCodec.releaseOutputBuffer(index, render);

       mAvailableOutputBufferIndices.removeFirst();
       mAvailableOutputBufferInfos.removeFirst();
       return true;
   }
}

public long getRealTimeUsForMediaTime(long mediaTimeUs) {
   if (mDeltaTimeUs == -1) {
      //第一次調(diào)用的時候會走進這個分支，mDeltaTimeUs代表初始pts的偏差，正常情況下它是0
       long nowUs = getNowUs();
       mDeltaTimeUs = nowUs - mediaTimeUs;
   }
   //所以getRealTimeUsForMediaTime返回的就是pts
   return mDeltaTimeUs + mediaTimeUs;
}

public long getNowUs() {
   //如果是video only的流，則返回系統(tǒng)時間，否則返回audio播放的時間
   if (mAudioTrackState == null) {
       return System.currentTimeMillis() * 1000;
   }

   return mAudioTrackState.getAudioTimeUs();
}

public long getAudioTimeUs() {
   //這里返回audio的播放時間，就是利用的getPlaybackHeadPosition方法
   int numFramesPlayed = mAudioTrack.getPlaybackHeadPosition();

   return (numFramesPlayed * 1000000L) / mSampleRate;
}

測試一下這個最簡單avsync邏輯的結(jié)果：-173ms，果然慘不忍睹

第一步改造

改造audio time的獲取，加上latency

public long getAudioTimeUs() {
  int numFramesPlayed = mAudioTrack.getPlaybackHeadPosition();
  if (getLatencyMethod != null) {
      try {
          latencyUs = (Integer) getLatencyMethod.invoke(mAudioTrack, (Object[]) null) * 1000L /2;
          latencyUs = Math.max(latencyUs, 0);
      } catch (Exception e){
          getLatencyMethod = null;
      }
  }
  return (numFramesPlayed * 1000000L) / mSampleRate - latencyUs;
}

此時的測試結(jié)果是：-128ms，好了一些，但還不夠

第二步改造

考慮到getLatency方法是一個被google自己吐槽的接口，我們再來下一步改造，用上getTimeStamp方法，如下

public long getAudioTimeUs() {
   long systemClockUs = System.nanoTime() / 1000;
   int numFramesPlayed = mAudioTrack.getPlaybackHeadPosition();
   if (systemClockUs - lastTimestampSampleTimeUs >= MIN_TIMESTAMP_SAMPLE_INTERVAL_US) {
       audioTimestampSet = mAudioTrack.getTimestamp(audioTimestamp);
       if (getLatencyMethod != null) {
           try {
               latencyUs = (Integer) getLatencyMethod.invoke(mAudioTrack, (Object[]) null) * 1000L / 2;
               latencyUs = Math.max(latencyUs, 0);
           } catch (Exception e) {
               getLatencyMethod = null;
           }
       }
       lastTimestampSampleTimeUs = systemClockUs;
   }

   if (audioTimestampSet) {
       // Calculate the speed-adjusted position using the timestamp (which may be in the future).
       long elapsedSinceTimestampUs = System.nanoTime() / 1000 - (audioTimestamp.nanoTime / 1000);
       long elapsedSinceTimestampFrames = elapsedSinceTimestampUs * mSampleRate / 1000000L;
       long elapsedFrames = audioTimestamp.framePosition + elapsedSinceTimestampFrames;
       long durationUs = (elapsedFrames * 1000000L) / mSampleRate;
       return durationUs;
   } else {
       long durationUs = (numFramesPlayed * 1000000L) / mSampleRate - latencyUs;
       //durationUs = Math.max(durationUs, 0);
       return durationUs;
   }
}

當(dāng)我們對比getTimeStamp和getPosition返回的結(jié)果，就能發(fā)現(xiàn)兩者的精度差距

1.getPosition, 最后的nowUs代表audio播放的duration
12-06 16:11:47.695 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 166667,realTimeUs is 46667,nowUs is 40000
12-06 16:11:47.696 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 208333,realTimeUs is 88333,nowUs is 40000
12-06 16:11:47.700 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 362666,realTimeUs is 242666,nowUs is 40000
12-06 16:11:47.706 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 208333,realTimeUs is 88333,nowUs is 40000
12-06 16:11:47.707 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 384000,realTimeUs is 264000,nowUs is 40000
12-06 16:11:47.714 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 208333,realTimeUs is 88333,nowUs is 80000
12-06 16:11:47.716 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 250000,realTimeUs is 130000,nowUs is 80000
12-06 16:11:47.720 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 405333,realTimeUs is 285333,nowUs is 80000
12-06 16:11:47.726 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 250000,realTimeUs is 130000,nowUs is 80000
12-06 16:11:47.728 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 426666,realTimeUs is 306666,nowUs is 80000
12-06 16:11:47.734 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 250000,realTimeUs is 130000,nowUs is 80000
12-06 16:11:47.736 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 448000,realTimeUs is 328000,nowUs is 120000
12-06 16:11:47.742 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 250000,realTimeUs is 130000,nowUs is 120000
12-06 16:11:47.743 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 291667,realTimeUs is 171667,nowUs is 120000
12-06 16:11:47.746 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 469333,realTimeUs is 349333,nowUs is 120000
12-06 16:11:47.753 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 291667,realTimeUs is 171667,nowUs is 120000
12-06 16:11:47.756 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 490666,realTimeUs is 370666,nowUs is 120000
12-06 16:11:47.764 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 291667,realTimeUs is 171667,nowUs is 160000
12-06 16:11:47.764 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 160000
12-06 16:11:47.767 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 512000,realTimeUs is 392000,nowUs is 160000
12-06 16:11:47.774 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 160000
12-06 16:11:47.776 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 533333,realTimeUs is 413333,nowUs is 160000
12-06 16:11:47.782 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 160000
12-06 16:11:47.783 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 554666,realTimeUs is 434666,nowUs is 160000
12-06 16:11:47.790 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 160000
12-06 16:11:47.791 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 576000,realTimeUs is 456000,nowUs is 160000
12-06 16:11:47.798 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 160000
12-06 16:11:47.806 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 597333,realTimeUs is 477333,nowUs is 200000
12-06 16:11:47.813 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 200000
12-06 16:11:47.814 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 200000
12-06 16:11:47.817 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 618666,realTimeUs is 498666,nowUs is 200000
12-06 16:11:47.825 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 200000
12-06 16:11:47.827 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 640000,realTimeUs is 520000,nowUs is 200000
12-06 16:11:47.836 20194-20657/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 200000
12-06 16:11:47.840 20194-20657/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 661333,realTimeUs is 541333,nowUs is 200000

2.getTimeStamp，同樣的，最后的nowUs代表audio播放的duration，其實這其中的一大部分功勞也是因為系統(tǒng)時間參與了計算
12-06 16:17:22.122 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 161312
12-06 16:17:22.124 30072-30257/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 554666,realTimeUs is 434666,nowUs is 163250
12-06 16:17:22.131 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 333333,realTimeUs is 213333,nowUs is 170125
12-06 16:17:22.131 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 170562
12-06 16:17:22.133 30072-30257/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 576000,realTimeUs is 456000,nowUs is 172666
12-06 16:17:22.141 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 180604
12-06 16:17:22.142 30072-30257/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 597333,realTimeUs is 477333,nowUs is 181666
12-06 16:17:22.150 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 189937
12-06 16:17:22.153 30072-30257/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 618666,realTimeUs is 498666,nowUs is 192145
12-06 16:17:22.159 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 198458
12-06 16:17:22.160 30072-30257/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 640000,realTimeUs is 520000,nowUs is 199687
12-06 16:17:22.166 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 205520
12-06 16:17:22.167 30072-30257/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 661333,realTimeUs is 541333,nowUs is 206812
12-06 16:17:22.173 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 375000,realTimeUs is 255000,nowUs is 212895
12-06 16:17:22.174 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 416667,realTimeUs is 296667,nowUs is 213104
12-06 16:17:22.176 30072-30257/com.example.zhanghui.avplayer D/avsync: audio: presentationUs is 682666,realTimeUs is 562666,nowUs is 215125
12-06 16:17:22.182 30072-30257/com.example.zhanghui.avplayer D/avsync: video: presentationUs is 416667,realTimeUs is 296667,nowUs is 221479

此時的測試結(jié)果是：-87.5ms，好了一些，已經(jīng)達到和ijkPlayer差不多的水平了，是否還能更好呢？

第三步改造

基于vsync調(diào)整送顯時間

audio部分似乎已經(jīng)沒什么可以改的了，下面我們來看看video部分加上基于vsync的調(diào)整后會怎么樣，前面我們看到在最樸素的邏輯中，使用的是不帶timestamp參數(shù)的releaseOutputBuffer方法，實際上，這時在framework中會把pts作為timestamp傳進去，如下

status_t MediaCodec::onReleaseOutputBuffer(const sp<AMessage> &msg) {
...
    BufferInfo *info = &mPortBuffers[kPortIndexOutput].editItemAt(index);
...
    if (render && info->mData != NULL && info->mData->size() != 0) {
        info->mNotify->setInt32("render", true);

        int64_t mediaTimeUs = -1;
        info->mData->meta()->findInt64("timeUs", &mediaTimeUs);

        int64_t renderTimeNs = 0;
        if (!msg->findInt64("timestampNs", &renderTimeNs)) {
            // use media timestamp if client did not request a specific render timestamp
            ALOGV("using buffer PTS of %lld", (long long)mediaTimeUs);
            renderTimeNs = mediaTimeUs * 1000;
        }
...

簡單起見，我們直接復(fù)用exoplayer中基于vsync調(diào)整的邏輯，做如下修改

這個方法以前是直接返回pts，現(xiàn)在我們讓他返回經(jīng)過vsync調(diào)整后的release時間
public long getRealTimeUsForMediaTime(long mediaTimeUs) {
   if (mDeltaTimeUs == -1) {
       long nowUs = getNowUs();
       mDeltaTimeUs = nowUs - mediaTimeUs; //-32000
   }
   long earlyUs = mDeltaTimeUs + mediaTimeUs - getNowUs();
   long unadjustedFrameReleaseTimeNs = System.nanoTime() + (earlyUs * 1000);
   long adjustedReleaseTimeNs = frameReleaseTimeHelper.adjustReleaseTime(
           mediaTimeUs, unadjustedFrameReleaseTimeNs);
   return adjustedReleaseTimeNs / 1000;
}

相應(yīng)的，在drainOutputBuffer中也要改一下
private boolean drainOutputBuffer() {
   ...

   long realTimeUs =
       mMediaTimeProvider.getRealTimeUsForMediaTime(info.presentationTimeUs); //返回調(diào)整后的releaseTime

   long nowUs = mMediaTimeProvider.getNowUs(); //audio play time

   long lateUs = System.nanoTime()/1000 - realTimeUs;

   if (mAudioTrack != null) {
       ...
       return true;
   } else {
       // video
       ….

       //mCodec.releaseOutputBuffer(index, render);
       mCodec.releaseOutputBuffer(index, realTimeUs*1000);
       mAvailableOutputBufferIndices.removeFirst();
       mAvailableOutputBufferInfos.removeFirst();
       return true;
   }
}

此時的測試結(jié)果是-76ms，又好了一些，但是已經(jīng)不甚明顯了，能不能再做進一步的優(yōu)化呢？

第四步改造

提前兩倍vsync時間調(diào)用releaseOutputBuffer

我們想到之前在NuPlayer和MediaSync中都有提前兩倍vsync duration時間調(diào)用releaseOutputBuffer方法的邏輯，加上試試看，如下

private boolean drainOutputBuffer() {
  ...
   long lateUs = System.nanoTime()/1000 - realTimeUs;

   if (mAudioTrack != null) {
      …
   } else {
       // video
       boolean render;
       //這里改為如果video比預(yù)期release時間來的早了2*vsyncduration時間以上，則跳過并且進入下次循環(huán)，否則予以顯示
       long twiceVsyncDurationUs = 2 * mMediaTimeProvider.getVsyncDurationNs()/1000;
       if (lateUs < -twiceVsyncDurationUs) {
           // too early;
           return false;
       } else if (lateUs > 30000) {
           Log.d(TAG, "video late by " + lateUs + " us.");
           render = false;
       } else {
           render = true;
           mPresentationTimeUs = info.presentationTimeUs;
       }

       //mCodec.releaseOutputBuffer(index, render);
       mCodec.releaseOutputBuffer(index, realTimeUs*1000);
...
   }
}

此時的測試結(jié)果是-66ms，又好了一點點，但是同樣已經(jīng)不甚明顯了

第五步改造

再做一些微調(diào)
比如在解碼audio時也做了一輪vsync調(diào)整，這顯然是多余的，去掉它

private boolean drainOutputBuffer() {
   
   if (mAudioTrack != null) {
   } else {
       // video
       boolean render;
       long twiceVsyncDurationUs = 2 * mMediaTimeProvider.getVsyncDurationNs()/1000;

       long realTimeUs =
               mMediaTimeProvider.getRealTimeUsForMediaTime(info.presentationTimeUs); 
       long nowUs = mMediaTimeProvider.getNowUs(); //audio play time
       long lateUs = System.nanoTime()/1000 - realTimeUs;
     ...
       mCodec.releaseOutputBuffer(index, realTimeUs*1000);
       mAvailableOutputBufferIndices.removeFirst();
       mAvailableOutputBufferInfos.removeFirst();
       return true;
   }
}

此時測試的結(jié)果是-56ms，看似是和avsync無關(guān)的調(diào)整，卻也產(chǎn)生了一定的效果，也從側(cè)面證明了根據(jù)vsync調(diào)整releaseTime是一個很耗時的操作

總結(jié)一下我們每一步所做的事情和avsync優(yōu)化的結(jié)果，如此，我們可以更好地理解avsync邏輯中各個環(huán)節(jié)的必要性和所產(chǎn)生的影響了

作者：張暉

來源：https://blog.csdn.net/nonmarking/article/details/78747369

深入理解Android音視頻同步機制（五）如何從零開始寫一個音視頻同步的播放器