在webrtc源碼分析之視頻編碼之一和webrtc源碼分析之視頻編碼之二分別分析了視頻編碼模塊的初始化流程和編碼流程，接下來(lái)說(shuō)一下視頻編碼模塊關(guān)鍵的幾個(gè)點(diǎn)。

硬編碼與軟編碼

視頻編碼有硬編碼和軟編碼，在android平臺(tái)上，MediaCodec封裝了硬編碼和軟編碼，對(duì)于軟編碼，也可以直接使用其他主流開(kāi)源編碼庫(kù)，比如H264編碼標(biāo)準(zhǔn)有l(wèi)ibx264、libopenh264，vp8/vp9編碼標(biāo)準(zhǔn)有l(wèi)ibvpx。

webrtc同時(shí)支持硬編碼和軟編碼，在android平臺(tái)上，硬編碼使用的是MediaCodec，是在java層封裝調(diào)用的，軟編碼H264使用的是libopenh264，vp8/vp9使用的是libvpx，都是在native層封裝調(diào)用的。webrtc定義了如下編碼相關(guān)的主要類：

java層主要類如下所示：

native層主要類如下所示：

在java層和native層都定義了VideoEncoderFactory接口和VideoEncoder和接口，VideoEncoderFactory創(chuàng)建VideoEncoder，VideoEncoder實(shí)現(xiàn)視頻編碼功能。從實(shí)現(xiàn)的角度來(lái)說(shuō)，可以分為以下幾類：

• 基于android系統(tǒng)提供的MediaCodec實(shí)現(xiàn)的HardwareVideoEncoder和MediaCodecVideoEncoder，支持H264、VP8、VP9編碼。
• 基于libopenh264實(shí)現(xiàn)的H264Encoder，支持H264編碼。
• 基于libvpx實(shí)現(xiàn)的VP8Encoder和VP9Encoder，分別支持VP8、VP9編碼。

其他類只是包裝類，比如native層VideoEncoderWrapper可以用于包裝java層HardwareVideoEncoder，因?yàn)榫幋a操作都是從native層調(diào)用的，借助VideoEncoderWrapper就可以在native層統(tǒng)一接口了。同理native層的VideoEncoderFactoryWrapper可以用于包裝java層VideoEncoderFactory對(duì)象。

以java層為例，VideoEncoderFactory定義如下：

/** Factory for creating VideoEncoders. */
public interface VideoEncoderFactory {
  /** Creates an encoder for the given video codec. */
  @CalledByNative VideoEncoder createEncoder(VideoCodecInfo info);

  /**
   * Enumerates the list of supported video codecs. This method will only be called once and the
   * result will be cached.
   */
  @CalledByNative VideoCodecInfo[] getSupportedCodecs();
}

createEncoder根據(jù)VideoCodecInfo創(chuàng)建對(duì)應(yīng)類型的編碼器，getSupportedCodecs獲取支持的編碼器類型，這里的類型指的是編碼標(biāo)準(zhǔn)，比如H264、VP8、VP9等。獲取的信息用于生成sdp信息用于協(xié)商會(huì)話使用的編碼器類型。實(shí)際定義了HardwareVideoEncoderFactory和SoftwareVideoEncoderFactory兩種，DefaultVideoEncoderFactory只是對(duì)它們的封裝。特定的VideoEncoderFactory創(chuàng)建特定的VideoEncoder，比如HardwareVideoEncoderFactory創(chuàng)建的是HardwareVideoEncoder，SoftwareVideoEncoderFactory創(chuàng)建的是VP8Encoder或者VP9Encoder，由info參數(shù)決定。

VideoEncoder主要定義如下：

  /**
   * Initializes the encoding process. Call before any calls to encode.
   */
  @CalledByNative VideoCodecStatus initEncode(Settings settings, Callback encodeCallback);

  /**
   * Releases the encoder. No more calls to encode will be made after this call.
   */
  @CalledByNative VideoCodecStatus release();

  /**
   * Requests the encoder to encode a frame.
   */
  @CalledByNative VideoCodecStatus encode(VideoFrame frame, EncodeInfo info);

  /**
   * Informs the encoder of the packet loss and the round-trip time of the network.
   *
   * @param packetLoss How many packets are lost on average per 255 packets.
   * @param roundTripTimeMs Round-trip time of the network in milliseconds.
   */
  @CalledByNative VideoCodecStatus setChannelParameters(short packetLoss, long roundTripTimeMs);

  /** Sets the bitrate allocation and the target framerate for the encoder. */
  @CalledByNative VideoCodecStatus setRateAllocation(BitrateAllocation allocation, int framerate);

  /** Any encoder that wants to use WebRTC provided quality scaler must implement this method. */
  @CalledByNative ScalingSettings getScalingSettings();

  /**
   * Should return a descriptive name for the implementation. Gets called once and cached. May be
   * called from arbitrary thread.
   */
  @CalledByNative String getImplementationName();

native層的EncoderAdapter的internal_encoder_factory_成員是一個(gè)InternalEncoderFactory對(duì)象，external_encoder_factory_成員是一個(gè)CricketToWebRtcEncoderFactory對(duì)象，CricketToWebRtcEncoderFactory的external_encoder_factory_成員是一個(gè)MediaCodecVideoEncoderFactory對(duì)象，VideoEncoderFactoryWrapper的encoder_factory_成員是一個(gè)java層的VideoEncoderFactory對(duì)象，VideoEncoderWrapper的encoder_成員是一個(gè)java層的VideoEncoder對(duì)象。

上面定義了這么多種VideoEncoderFactory和VideoEncoder，實(shí)際使用的是哪一種呢？實(shí)際使用哪一種跟調(diào)用webrtc api傳遞的參數(shù)、硬件平臺(tái)以及android系統(tǒng)版本相關(guān)。

參數(shù)主要是PeerConnectionFactory相關(guān)的，比如：

  public static void initializeFieldTrials(String fieldTrialsInitString) {
    nativeInitializeFieldTrials(fieldTrialsInitString);
  }

fieldTrialsInitString的值會(huì)影響VideoEncoderSoftwareFallbackWrapper的行為。
還有就是給PeerConnectionFactory構(gòu)造函數(shù)傳的encoderFactory的值。

相關(guān)的代碼如下所示：

  public PeerConnectionFactory(
      Options options, VideoEncoderFactory encoderFactory, VideoDecoderFactory decoderFactory) {
    checkInitializeHasBeenCalled();
    nativeFactory = nativeCreatePeerConnectionFactory(options, encoderFactory, decoderFactory);
    if (nativeFactory == 0) {
      throw new RuntimeException("Failed to initialize PeerConnectionFactory!");
    }
  }

jlong CreatePeerConnectionFactoryForJava(
    JNIEnv* jni,
    const JavaParamRef<jobject>& joptions,
    const JavaParamRef<jobject>& jencoder_factory,
    const JavaParamRef<jobject>& jdecoder_factory,
    rtc::scoped_refptr<AudioProcessing> audio_processor) {

  cricket::WebRtcVideoEncoderFactory* legacy_video_encoder_factory = nullptr;
  cricket::WebRtcVideoDecoderFactory* legacy_video_decoder_factory = nullptr;
  std::unique_ptr<cricket::MediaEngineInterface> media_engine;
  if (jencoder_factory.is_null() && jdecoder_factory.is_null()) {
    // This uses the legacy API, which automatically uses the internal SW
    // codecs in WebRTC.
    if (video_hw_acceleration_enabled) {
      legacy_video_encoder_factory = CreateLegacyVideoEncoderFactory();
      legacy_video_decoder_factory = CreateLegacyVideoDecoderFactory();
    }
    media_engine.reset(CreateMediaEngine(
        adm, audio_encoder_factory, audio_decoder_factory,
        legacy_video_encoder_factory, legacy_video_decoder_factory, audio_mixer,
        audio_processor));
  } else {
    // This uses the new API, does not automatically include software codecs.
    std::unique_ptr<VideoEncoderFactory> video_encoder_factory = nullptr;
    if (jencoder_factory.is_null()) {
      legacy_video_encoder_factory = CreateLegacyVideoEncoderFactory();
      video_encoder_factory = std::unique_ptr<VideoEncoderFactory>(
          WrapLegacyVideoEncoderFactory(legacy_video_encoder_factory));
    } else {
      video_encoder_factory = std::unique_ptr<VideoEncoderFactory>(
          CreateVideoEncoderFactory(jni, jencoder_factory));
    }

    std::unique_ptr<VideoDecoderFactory> video_decoder_factory = nullptr;
    if (jdecoder_factory.is_null()) {
      legacy_video_decoder_factory = CreateLegacyVideoDecoderFactory();
      video_decoder_factory = std::unique_ptr<VideoDecoderFactory>(
          WrapLegacyVideoDecoderFactory(legacy_video_decoder_factory));
    } else {
      video_decoder_factory = std::unique_ptr<VideoDecoderFactory>(
          CreateVideoDecoderFactory(jni, jdecoder_factory));
    }

    rtc::scoped_refptr<AudioDeviceModule> adm_scoped = nullptr;
    media_engine.reset(CreateMediaEngine(
        adm_scoped, audio_encoder_factory, audio_decoder_factory,
        std::move(video_encoder_factory), std::move(video_decoder_factory),
        audio_mixer, audio_processor));
  }

  rtc::scoped_refptr<PeerConnectionFactoryInterface> factory(
      CreateModularPeerConnectionFactory(
          network_thread.get(), worker_thread.get(), signaling_thread.get(),
          std::move(media_engine), std::move(call_factory),
          std::move(rtc_event_log_factory)));
  RTC_CHECK(factory) << "Failed to create the peer connection factory; "
                     << "WebRTC/libjingle init likely failed on this device";
  // TODO(honghaiz): Maybe put the options as the argument of
  // CreatePeerConnectionFactory.
  if (has_options) {
    factory->SetOptions(options);
  }
  OwnedFactoryAndThreads* owned_factory = new OwnedFactoryAndThreads(
      std::move(network_thread), std::move(worker_thread),
      std::move(signaling_thread), legacy_video_encoder_factory,
      legacy_video_decoder_factory, network_monitor_factory, factory.release());
  owned_factory->InvokeJavaCallbacksOnFactoryThreads();
  return jlongFromPointer(owned_factory);
}

可見(jiàn)傳給PeerConnectionFactory構(gòu)造函數(shù)encoderFactory參數(shù)的值直接影響使用哪一個(gè)VideoEncoderFactory，也就直接影響使用哪一個(gè)VideoEncoder。demo中調(diào)用如下：

    if (peerConnectionParameters.videoCodecHwAcceleration) {
      encoderFactory = new DefaultVideoEncoderFactory(
          rootEglBase.getEglBaseContext(), true /* enableIntelVp8Encoder */, enableH264HighProfile);
      decoderFactory = new DefaultVideoDecoderFactory(rootEglBase.getEglBaseContext());
    } else {
      encoderFactory = new SoftwareVideoEncoderFactory();
      decoderFactory = new SoftwareVideoDecoderFactory();
    }

    factory = new PeerConnectionFactory(options, encoderFactory, decoderFactory);

可見(jiàn)使能hardware acceleration時(shí)，使用的是DefaultVideoEncoderFactory，而DefaultVideoEncoderFactory createEncoder時(shí)會(huì)先后嘗試HardwareVideoEncoderFactory和SoftwareVideoEncoderFactory兩種，如下所示：

  public VideoEncoder createEncoder(VideoCodecInfo info) {
    final VideoEncoder videoEncoder = hardwareVideoEncoderFactory.createEncoder(info);
    if (videoEncoder != null) {
      return videoEncoder;
    }
    return softwareVideoEncoderFactory.createEncoder(info);
  }

也就是無(wú)法使用硬編碼的情況下使用軟編碼，而是否能使用硬編碼取決于硬件平臺(tái)以及android系統(tǒng)版本是否支持對(duì)應(yīng)的編碼標(biāo)準(zhǔn)，如下所示：

public VideoEncoder createEncoder(VideoCodecInfo input) {
    VideoCodecType type = VideoCodecType.valueOf(input.name);
    MediaCodecInfo info = findCodecForType(type);

    if (info == null) {
      // No hardware support for this type.
      // TODO(andersc): This is for backwards compatibility. Remove when clients have migrated to
      // new DefaultVideoEncoderFactory.
      if (fallbackToSoftware) {
        SoftwareVideoEncoderFactory softwareVideoEncoderFactory = new SoftwareVideoEncoderFactory();
        return softwareVideoEncoderFactory.createEncoder(input);
      } else {
        return null;
      }
    }

    String codecName = info.getName();
    String mime = type.mimeType();
    Integer surfaceColorFormat = MediaCodecUtils.selectColorFormat(
        MediaCodecUtils.TEXTURE_COLOR_FORMATS, info.getCapabilitiesForType(mime));
    Integer yuvColorFormat = MediaCodecUtils.selectColorFormat(
        MediaCodecUtils.ENCODER_COLOR_FORMATS, info.getCapabilitiesForType(mime));

    if (type == VideoCodecType.H264) {
      boolean isHighProfile = nativeIsSameH264Profile(input.params, getCodecProperties(type, true))
          && isH264HighProfileSupported(info);
      boolean isBaselineProfile =
          nativeIsSameH264Profile(input.params, getCodecProperties(type, false));

      if (!isHighProfile && !isBaselineProfile) {
        return null;
      }
    }

    return new HardwareVideoEncoder(codecName, type, surfaceColorFormat, yuvColorFormat,
        input.params, getKeyFrameIntervalSec(type), getForcedKeyFrameIntervalMs(type, codecName),
        createBitrateAdjuster(type, codecName), sharedContext);
  }

VideoCodecInfo的name包含要使用的編碼類型信息，比如H264、VP8、VP9，可以對(duì)應(yīng)一個(gè)VideoCodecType，如下所示：

enum VideoCodecType {
  VP8("video/x-vnd.on2.vp8"),
  VP9("video/x-vnd.on2.vp9"),
  H264("video/avc");

  private final String mimeType;

  private VideoCodecType(String mimeType) {
    this.mimeType = mimeType;
  }

  String mimeType() {
    return mimeType;
  }
}

調(diào)用findCodecForType判斷使用的硬件平臺(tái)以及android系統(tǒng)版本是否支持該編碼類型，支持的話就使用HardwareVideoEncoder，否則就會(huì)調(diào)用SoftwareVideoEncoderFactory創(chuàng)建對(duì)應(yīng)的軟解碼器，findCodecForType定義如下所示：

  private MediaCodecInfo findCodecForType(VideoCodecType type) {
    for (int i = 0; i < MediaCodecList.getCodecCount(); ++i) {
      MediaCodecInfo info = null;
      try {
        info = MediaCodecList.getCodecInfoAt(i);
      } catch (IllegalArgumentException e) {
        Logging.e(TAG, "Cannot retrieve encoder codec info", e);
      }

      if (info == null || !info.isEncoder()) {
        continue;
      }

      if (isSupportedCodec(info, type)) {
        return info;
      }
    }
    return null; // No support for this type.
  }

isSupportedCodec定義如下所示：

  private boolean isSupportedCodec(MediaCodecInfo info, VideoCodecType type) {
    if (!MediaCodecUtils.codecSupportsType(info, type)) {
      return false;
    }
    // Check for a supported color format.
    if (MediaCodecUtils.selectColorFormat(
            MediaCodecUtils.ENCODER_COLOR_FORMATS, info.getCapabilitiesForType(type.mimeType()))
        == null) {
      return false;
    }
    return isHardwareSupportedInCurrentSdk(info, type);
  }

先調(diào)用codecSupportsType比對(duì)mimeType，如下所示：

  static boolean codecSupportsType(MediaCodecInfo info, VideoCodecType type) {
    for (String mimeType : info.getSupportedTypes()) {
      if (type.mimeType().equals(mimeType)) {
        return true;
      }
    }
    return false;
  }

接著調(diào)用isHardwareSupportedInCurrentSdk比對(duì)硬件平臺(tái)以及android系統(tǒng)版本，如下所示：

  private boolean isHardwareSupportedInCurrentSdk(MediaCodecInfo info, VideoCodecType type) {
    switch (type) {
      case VP8:
        return isHardwareSupportedInCurrentSdkVp8(info);
      case VP9:
        return isHardwareSupportedInCurrentSdkVp9(info);
      case H264:
        return isHardwareSupportedInCurrentSdkH264(info);
    }
    return false;
  }

如果是H264，調(diào)用的是isHardwareSupportedInCurrentSdkH264，如下所示：

  private boolean isHardwareSupportedInCurrentSdkH264(MediaCodecInfo info) {
    // First, H264 hardware might perform poorly on this model.
    if (H264_HW_EXCEPTION_MODELS.contains(Build.MODEL)) {
      return false;
    }
    String name = info.getName();
    // QCOM H264 encoder is supported in KITKAT or later.
    return (name.startsWith(QCOM_PREFIX) && Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT)
        // Exynos H264 encoder is supported in LOLLIPOP or later.
        || (name.startsWith(EXYNOS_PREFIX)
               && Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP);
  }

可見(jiàn)，在這里完成了硬件平臺(tái)以及android系統(tǒng)版本的判斷。

總的來(lái)說(shuō)，實(shí)際使用哪一種編碼器跟調(diào)用webrtc api傳遞的參數(shù)、硬件平臺(tái)以及android系統(tǒng)版本相關(guān)。

碼率控制

Todo

Todo