從jitterbuffer取出frame，解碼

在ViEChannel類(lèi)中創(chuàng)建解碼線(xiàn)程，在VCMReceiver類(lèi)中調(diào)用jitterbuffer取出frame。

bool ViEChannel::ChannelDecodeProcess()
->int32_t Decode(uint16_t maxWaitTimeMs)
->int32_t VideoReceiver::Decode(uint16_t maxWaitTimeMs)
->VCMEncodedFrame* VCMReceiver::FrameForDecoding
```
先檢查是否存在完整的幀，存在則取出，若不存在，則檢查不完整的幀，滿(mǎn)足條件則取出，不滿(mǎn)足則不取出。
```
  bool found_frame = jitter_buffer_.NextCompleteTimestamp(
      max_wait_time_ms, &frame_timestamp);
  if (!found_frame)
    found_frame = jitter_buffer_.NextMaybeIncompleteTimestamp(&frame_timestamp);
```
在上一步取出frame_timestamp，再用frame_timestamp從jitterbuffer中取出幀數(shù)據(jù)
```
VCMEncodedFrame* frame = jitter_buffer_.ExtractAndSetDecode(frame_timestamp);
```
##接收到包，將包插入jitterbuffer
接收到包，將包插入jitterbuffer代碼流程
```
void UdpTransportImpl::IncomingRTPCallback
->void UdpTransportImpl::IncomingRTPFunction
->void VideoChannelTransport::IncomingRTPPacket
->int ViENetworkImpl::ReceivedRTPPacket
->int32_t ViEChannel::ReceivedRTPPacket
->int ViEReceiver::ReceivedRTPPacket
->int ViEReceiver::InsertRTPPacket
->bool ViEReceiver::ReceivePacket
->bool RtpReceiverImpl::IncomingRtpPacket
->int32_t RTPReceiverVideo::ParseRtpPacket
->int32_t ViEReceiver::OnReceivedPayloadData
->int32_t IncomingPacket
->int32_t VideoReceiver::IncomingPacket
->int32_t VCMReceiver::InsertPacket
->VCMFrameBufferEnum VCMJitterBuffer::InsertPacket
```

##decodable_frames_，incomplete_frames_，free_frames_
decodable_frames_，incomplete_frames_，free_frames_的處理，主要在VCMFrameBuffer類(lèi)中，其中包含的VCMSessionInfo _sessionInfo;為主要處理成員。

對(duì)于每一次收到的包，根據(jù)時(shí)間戳找到，當(dāng)前幀在哪個(gè)隊(duì)列中，在decodable_frames_或incomplete_frames_隊(duì)列中，若不存在，則從free_frames_隊(duì)列中給出一個(gè)空幀。
```  
VCMFrameBuffer* frame;
  FrameList* frame_list;
  const VCMFrameBufferEnum error = GetFrame(packet, &frame, &frame_list);```
```
// Gets frame to use for this timestamp. If no match, get empty frame.
VCMFrameBufferEnum VCMJitterBuffer::GetFrame(const VCMPacket& packet,
                                             VCMFrameBuffer** frame,
                                             FrameList** frame_list) {
  *frame = incomplete_frames_.PopFrame(packet.timestamp);
  if (*frame != NULL) {
    *frame_list = &incomplete_frames_;
    return kNoError;
  }
  *frame = decodable_frames_.PopFrame(packet.timestamp);
  if (*frame != NULL) {
    *frame_list = &decodable_frames_;
    return kNoError;
  }

  *frame_list = NULL;
  // No match, return empty frame.
  *frame = GetEmptyFrame();
  if (*frame == NULL) {
    // No free frame! Try to reclaim some...
    LOG(LS_WARNING) << "Unable to get empty frame; Recycling.";
    bool found_key_frame = RecycleFramesUntilKeyFrame();
    *frame = GetEmptyFrame();
    assert(*frame);
    if (!found_key_frame) {
      free_frames_.push_back(*frame);
      return kFlushIndicator;
    }
  }
  (*frame)->Reset();
  return kNoError;
}```

對(duì)于VCMSessionInfo中  complete_和decodable_的判定，每一次插入一個(gè)包，都要進(jìn)行UpdateCompleteSession();對(duì)于幀的完整性進(jìn)行檢查。
```
  size_t returnLength = InsertBuffer(frame_buffer, packet_list_it);
  UpdateCompleteSession();
  if (decode_error_mode == kWithErrors)
    decodable_ = true;
  else if (decode_error_mode == kSelectiveErrors)
    UpdateDecodableSession(frame_data);
  return static_cast<int>(returnLength);
```
UpdateCompleteSession();檢查是否有第一個(gè)包和最后一個(gè)包，并且都是按序的，即沒(méi)有丟包，若滿(mǎn)足這些條件，則判定為complete_ = true;
```
void VCMSessionInfo::UpdateCompleteSession() {
  if (HaveFirstPacket() && HaveLastPacket()) {
    // Do we have all the packets in this session?
    bool complete_session = true;
    PacketIterator it = packets_.begin();
    PacketIterator prev_it = it;
    ++it;
    for (; it != packets_.end(); ++it) {
      if (!InSequence(it, prev_it)) {
        complete_session = false;
        break;
      }
      prev_it = it;
    }
    complete_ = complete_session;
  }
}
```
下面則是對(duì)于decodable_ 的判定，如果decode_error_mode 為kWithErrors模式，則有一個(gè)包decodable_ 即可以為ture。如果decode_error_mode 為kSelectiveErrors，則根據(jù)rtt,幀類(lèi)型，已經(jīng)有的包數(shù)等條件來(lái)綜合得出decodable_ 。
具體條件則為：rtt<100，或者為關(guān)鍵幀，或者已經(jīng)收到的包數(shù)在0.2\*rolling_average_packets_per_frame和0.8\*rolling_average_packets_per_frame之間時(shí)，則decodable_ 不去改變，否則為true。rolling_average_packets_per_frame為平均每幀所含的包數(shù)。
```
void VCMSessionInfo::UpdateDecodableSession(const FrameData& frame_data) {
  // Irrelevant if session is already complete or decodable
  if (complete_ || decodable_)
    return;
  // TODO(agalusza): Account for bursty loss.
  // TODO(agalusza): Refine these values to better approximate optimal ones.
  // Do not decode frames if the RTT is lower than this.
  const int64_t kRttThreshold = 100;
  // Do not decode frames if the number of packets is between these two
  // thresholds.
  const float kLowPacketPercentageThreshold = 0.2f;
  const float kHighPacketPercentageThreshold = 0.8f;
  if (frame_data.rtt_ms < kRttThreshold
      || frame_type_ == kVideoFrameKey
      || !HaveFirstPacket()
      || (NumPackets() <= kHighPacketPercentageThreshold
                          * frame_data.rolling_average_packets_per_frame
          && NumPackets() > kLowPacketPercentageThreshold
                            * frame_data.rolling_average_packets_per_frame))
    return;

  decodable_ = true;
}
```
```
PS：
1、如果rolling_average_packets_per_frame>5,
kLowPacketPercentageThreshold * frame_data.rolling_average_packets_per_frame>1
那么來(lái)一個(gè)包就判定  decodable_ = true;這個(gè)是否合理？
2、 rtt很小的時(shí)候，不判定為true，是希望能夠complete_ ，至于kRttThreshold = 100是否合理，有待驗(yàn)證。
3、可見(jiàn)，關(guān)鍵幀都是完整的，不完整，不會(huì) 設(shè)置decodable_ = true;則一直為kIncomplete。
```

對(duì)于VCMFrameBuffer中的kCompleteSession和kDecodableSession分別對(duì)應(yīng)VCMSessionInfo中的complete_ 和decodable_ 。若既不是complete_ ，也不是decodable_，則對(duì)應(yīng)kIncomplete。
```
    if (_sessionInfo.complete()) {
      SetState(kStateComplete);
      return kCompleteSession;
    } else if (_sessionInfo.decodable()) {
      SetState(kStateDecodable);
      return kDecodableSession;
    }
    return kIncomplete;
```

對(duì)于上面提到的decode_error_mode，通過(guò)如下流程進(jìn)行設(shè)置。
```
int Conductor::VideoCreateStream
->int ViEBaseImpl::CreateChannel
->int ViEBaseImpl::CreateChannel
->int ViEChannelManager::CreateChannel
->bool ChannelGroup::CreateSendChannel
->bool ChannelGroup::CreateChannel
->int32_t ViEChannel::Init
->int32_t SetVideoProtection
->int32_t VideoReceiver::SetVideoProtection
->void VCMReceiver::SetDecodeErrorMode
->void VCMJitterBuffer::SetDecodeErrorMode
```
這里根據(jù)NACK和FEC的使用情況，來(lái)設(shè)置decode_error_mode。
```
// Enable or disable a video protection method.
// Note: This API should be deprecated, as it does not offer a distinction
// between the protection method and decoding with or without errors. If such a
// behavior is desired, use the following API: SetReceiverRobustnessMode.
int32_t VideoReceiver::SetVideoProtection(VCMVideoProtection videoProtection,
                                          bool enable) {
  // By default, do not decode with errors.
  _receiver.SetDecodeErrorMode(kNoErrors);
  switch (videoProtection) {
    case kProtectionNack:
    case kProtectionNackReceiver: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        // Enable NACK and always wait for retransmits.
        _receiver.SetNackMode(kNack, -1, -1);
      } else {
        _receiver.SetNackMode(kNoNack, -1, -1);
      }
      break;
    }

    case kProtectionKeyOnLoss: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        _keyRequestMode = kKeyOnLoss;
        _receiver.SetDecodeErrorMode(kWithErrors);
      } else if (_keyRequestMode == kKeyOnLoss) {
        _keyRequestMode = kKeyOnError;  // default mode
      } else {
        return VCM_PARAMETER_ERROR;
      }
      break;
    }

    case kProtectionKeyOnKeyLoss: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        _keyRequestMode = kKeyOnKeyLoss;
      } else if (_keyRequestMode == kKeyOnKeyLoss) {
        _keyRequestMode = kKeyOnError;  // default mode
      } else {
        return VCM_PARAMETER_ERROR;
      }
      break;
    }

    case kProtectionNackFEC: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        // Enable hybrid NACK/FEC. Always wait for retransmissions
        // and don't add extra delay when RTT is above
        // kLowRttNackMs.
        _receiver.SetNackMode(kNack, media_optimization::kLowRttNackMs, -1);
        _receiver.SetDecodeErrorMode(kNoErrors);
        _receiver.SetDecodeErrorMode(kNoErrors);
      } else {
        _receiver.SetNackMode(kNoNack, -1, -1);
      }
      break;
    }
    case kProtectionNackSender:
    case kProtectionFEC:
    case kProtectionPeriodicKeyFrames:
      // Ignore encoder modes.
      return VCM_OK;
  }
  return VCM_OK;
}
```


下面則介紹本文的核心，decodable_frames_，incomplete_frames_，free_frames_的處理。
```
  // Is the frame already in the decodable list?
  bool continuous = IsContinuous(*frame);
  switch (buffer_state) {
    case kGeneralError:
    case kTimeStampError:
    case kSizeError: {
      free_frames_.push_back(frame);
      break;
    }
    case kCompleteSession: {
      if (previous_state != kStateDecodable &&
          previous_state != kStateComplete) {
        CountFrame(*frame);
        if (continuous) {
          // Signal that we have a complete session.
          frame_event_->Set();
        }
      }
      FALLTHROUGH();
    }
    // Note: There is no break here - continuing to kDecodableSession.
    case kDecodableSession: {
      *retransmitted = (frame->GetNackCount() > 0);
      if (continuous) {
        decodable_frames_.InsertFrame(frame);
        FindAndInsertContinuousFrames(*frame);
      } else {
        incomplete_frames_.InsertFrame(frame);
      }
      break;
    }
    case kIncomplete: {
      if (frame->GetState() == kStateEmpty &&
          last_decoded_state_.UpdateEmptyFrame(frame)) {
        free_frames_.push_back(frame);
        return kNoError;
      } else {
        incomplete_frames_.InsertFrame(frame);
      }
      break;
    }
    case kNoError:
    case kOutOfBoundsPacket:
    case kDuplicatePacket: {
      // Put back the frame where it came from.
      if (frame_list != NULL) {
        frame_list->InsertFrame(frame);
      } else {
        free_frames_.push_back(frame);
      }
      ++num_duplicated_packets_;
      break;
    }
    case kFlushIndicator:
      free_frames_.push_back(frame);
      return kFlushIndicator;
    default: assert(false);
  }
```
其中第一句 bool continuous = IsContinuous(*frame);主要是判斷當(dāng)前收到幀和上一個(gè)解碼幀是不是連續(xù)的。其中還有一些特殊情況，如decode_error_mode_ == kWithErrors，或者frame->FrameType() == kVideoFrameKey等均判斷為連續(xù)的。由于不加FEC和NACK時(shí)，decode_error_mode_ = kWithErrors，所以，一直continuous 為true。
只有kCompleteSession時(shí)，才觸發(fā)事件frame_event_->Set();等待事件在
bool VCMJitterBuffer::NextCompleteTimestamp中，即取幀的函數(shù)中。
注意：kCompleteSession情況，后面沒(méi)有break,則將完成的幀也插入decodable_frames_隊(duì)列。
所以，對(duì)于VCMSessionInfo中的complete_ 和decodable_ ，都將插入decodable_frames_隊(duì)列。

##再回頭看從jitterbuffer取出frame
```
// Returns immediately or a |max_wait_time_ms| ms event hang waiting for a
// complete frame, |max_wait_time_ms| decided by caller.
bool VCMJitterBuffer::NextCompleteTimestamp(
    uint32_t max_wait_time_ms, uint32_t* timestamp) {
  crit_sect_->Enter();
  if (!running_) {
    crit_sect_->Leave();
    return false;
  }
  CleanUpOldOrEmptyFrames();

  if (decodable_frames_.empty() ||
      decodable_frames_.Front()->GetState() != kStateComplete) 
  {
    const int64_t end_wait_time_ms = clock_->TimeInMilliseconds() +
        max_wait_time_ms;
    int64_t wait_time_ms = max_wait_time_ms;
    while (wait_time_ms > 0) {
      crit_sect_->Leave();
      const EventTypeWrapper ret =
        frame_event_->Wait(static_cast<uint32_t>(wait_time_ms));
      crit_sect_->Enter();
      if (ret == kEventSignaled) {
        // Are we shutting down the jitter buffer?
        if (!running_) {
          crit_sect_->Leave();
          return false;
        }
        // Finding oldest frame ready for decoder.
        CleanUpOldOrEmptyFrames();
        if (decodable_frames_.empty() ||
            decodable_frames_.Front()->GetState() != kStateComplete) {
          wait_time_ms = end_wait_time_ms - clock_->TimeInMilliseconds();
        } else {
          break;
        }
      } else {
        break;
      }
    }
  }
  if (decodable_frames_.empty() ||
      decodable_frames_.Front()->GetState() != kStateComplete) {
    crit_sect_->Leave();
    return false;
  }
  *timestamp = decodable_frames_.Front()->TimeStamp();
  crit_sect_->Leave();
  return true;
}
```
1、其中decodable_frames_.Front()->GetState() 取得的_state，有下列情況賦值：
```
        if (packet.frameType != kFrameEmpty) {
            // first media packet
            SetState(kStateIncomplete);
        }
```
```
    if (_sessionInfo.complete()) {
      SetState(kStateComplete);
      return kCompleteSession;
    } else if (_sessionInfo.decodable()) {
      SetState(kStateDecodable);
      return kDecodableSession;
    }
```
可見(jiàn)，_state還是可以標(biāo)記這一幀數(shù)據(jù)的完成情況的，即完成時(shí)，為kStateComplete；未完成時(shí)，為kStateDecodable或kStateIncomplete，只有一個(gè)包時(shí)，為kStateIncomplete。凡是沒(méi)有從kStateIncomplete升為kStateDecodable，則依然為kStateIncomplete。

2、其中條件decodable_frames_.Front()->GetState() != kStateComplete
可見(jiàn)，必須是完整的幀才能夠從NextCompleteTimestamp函數(shù)中取出來(lái)。

再看取出不完整的幀：

```
bool VCMJitterBuffer::NextMaybeIncompleteTimestamp(uint32_t* timestamp) {
  CriticalSectionScoped cs(crit_sect_);
  if (!running_) {
    return false;
  }
  if (decode_error_mode_ == kNoErrors) {
    // No point to continue, as we are not decoding with errors.
    return false;
  }

  CleanUpOldOrEmptyFrames();

  VCMFrameBuffer* oldest_frame;
  if (decodable_frames_.empty()) {
    if (nack_mode_ != kNoNack || incomplete_frames_.size() <= 1) {
      return false;
    }
    oldest_frame = incomplete_frames_.Front();
    // Frame will only be removed from buffer if it is complete (or decodable).
    if (oldest_frame->GetState() < kStateComplete) {
      return false;
    }
  } else {
    oldest_frame = decodable_frames_.Front();
    // If we have exactly one frame in the buffer, release it only if it is
    // complete. We know decodable_frames_ is  not empty due to the previous
    // check.
    if (decodable_frames_.size() == 1 && incomplete_frames_.empty() &&
        oldest_frame->GetState() != kStateComplete) {
      return false;
    }
  }

  *timestamp = oldest_frame->TimeStamp();
  return true;
}
```
從條件中可以看出：
1、decodable_frames_為空時(shí)
incomplete_frames_.Front()->GetState()為  kStateEmpty或者kStateIncomplete，則取不出幀，否則可以取出。
2、decodable_frames_不空時(shí)
decodable_frames_.size() == 1 && incomplete_frames_.empty() &&
        oldest_frame->GetState() != kStateComplete時(shí)，取不出幀。否則取出decodable_frames_中不完整的數(shù)據(jù)幀。

##總結(jié)：
至此，jitterbuffer對(duì)于包、幀的處理，已經(jīng)比較清晰。
所有組包的幀都存在于decodable_frames_，incomplete_frames_隊(duì)列中，而decodable_frames_又根據(jù)狀態(tài)分為完整的幀和不完整的幀，incomplete_frames_主要保存狀態(tài)為kIncomplete的幀，也是不完整的，但有包數(shù)據(jù)。
而在取幀數(shù)據(jù)的時(shí)候先取完整的幀，取不到，則取一定條件下不完整的幀數(shù)據(jù)，不是有不完整的幀數(shù)據(jù)就去取。
所以，如果你不想取出丟包的幀，則只調(diào)用NextCompleteTimestamp去取完整的幀即可。因?yàn)槊看蝸?lái)的包，放入幀中之后，都是插入到decodable_frames_或者incomplete_frames_的最前面，所以不存在一直取不出來(lái)幀數(shù)據(jù)的情況。另外一個(gè)方法，就是設(shè)置decode_error_mode_ 為kNoErrors。
但是，不取出丟包的幀 ，不等于不存在馬賽克。因?yàn)閬G幀，所以解碼的時(shí)候，可能會(huì)存在參考幀的問(wèn)題。

##追加--如何杜絕馬賽克
由上述對(duì)于馬賽克存在的討論中可知，丟包是主要原因，如果只取出完整的幀，也會(huì)因?yàn)閰⒖紟瑔?wèn)題，導(dǎo)致馬賽克。
如果，丟包以后，就把整個(gè)GOP都刪除，直到下一個(gè)關(guān)鍵幀，同時(shí)，發(fā)現(xiàn)丟包就去請(qǐng)求關(guān)鍵幀，這樣就完全可以杜絕馬賽克了。但是這樣會(huì)導(dǎo)致在丟包時(shí)，流暢度更加不好。