目錄
- MediaExtractor MediaMuxer 能做什么
- 視頻解封裝和合成的API以及流程介紹
- 三個(gè)實(shí)踐(視頻解封裝提取純音軌和視頻軌文件、再合成新視頻、給視頻換個(gè)背景音)
- 遇到的問(wèn)題
- 收獲
一、有什么實(shí)際應(yīng)用
在我們?nèi)粘J褂枚桃曨l軟件的時(shí)候,對(duì)視頻的裁剪,拼湊,加入背景是很常用的操作,這些功能是如何實(shí)現(xiàn)的吶?其實(shí)是將視頻多信道的分離出來(lái),比如音軌和視頻軌道分隔出來(lái),可以做到二次合成。
今天我們通過(guò)對(duì)來(lái)MediaExtractor和MediaMuxer的學(xué)習(xí)分析和實(shí)踐來(lái)實(shí)現(xiàn) “把視頻分離(提取&解封裝)出純音頻和純視頻文件”、“替換背景音樂(lè),合成新的視頻文件”。
二、視頻解封裝和合成的API以及流程介紹
2.1 MediaExtractor:視頻軌道提取器(解封裝)
主要API介紹
setDataSource(path):path本地或者網(wǎng)絡(luò)文件
getTrackCount:獲取軌道數(shù)
getTrackFormat(i):對(duì)應(yīng)軌道的格式 MediaFormat
selectTrack(I):切換到(選定)某個(gè)軌道
readSampleData(ByteBuffer byteBuff, int offset): 把指定軌道中的樣本數(shù)據(jù)按偏移量讀取到ByteBuffer字節(jié)緩沖區(qū)
advance(): 提取到下一幀數(shù)據(jù) 作用有點(diǎn)類似于cursor
unselectTrack(i)
release()
getSampleFlags: 獲取數(shù)據(jù)的flag,數(shù)據(jù)為什么要用Sample來(lái)表示,因?yàn)橐粢曨l的數(shù)據(jù)是采樣數(shù)據(jù)。
getSampleTime:返回當(dāng)前的時(shí)間戳
數(shù)據(jù)提取(解封裝)流程如下:
//1. 構(gòu)造MediaExtractor
MediaExtractor mediaExtractor = new MediaExtractor();
try {
//2.設(shè)置數(shù)據(jù)源
mediaExtractor.setDataSource(inputFile.getAbsolutePath());
//3. 獲取軌道數(shù)
int trackCount = mediaExtractor.getTrackCount();
Log.i(TAG, "demuxerMP4: trackCount=" + trackCount);
//遍歷軌道,查看音頻軌或者視頻軌道信息
for (int i = 0; i < trackCount; i++) {
//4. 獲取某一軌道的媒體格式
MediaFormat trackFormat = mediaExtractor.getTrackFormat(i);
String keyMime = trackFormat.getString(MediaFormat.KEY_MIME);
Log.i(TAG, "demuxerMp4: keyMime=" + keyMime);
if (TextUtils.isEmpty(keyMime)) {
continue;
}
//5.通過(guò)mime信息識(shí)別音軌或視頻軌道,打印相關(guān)信息
if (keyMime.startsWith("video/")) {
//打印視頻的寬高
Log.i(TAG, "extractorAndMuxerMP4: videoWidth="+trackFormat.getInteger(MediaFormat.KEY_WIDTH)+" videoHeight="+trackFormat.getInteger(MediaFormat.KEY_HEIGHT));
} else if (keyMime.startsWith("audio/")) {
//打印音軌的通道數(shù)以及比特率
Log.i(TAG, "extractorAndMuxerMP4: channelCount="+trackFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT)+" bitRate="+trackFormat.getInteger(MediaFormat.KEY_BIT_RATE));
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
mediaExtractor.release();
}
2.2 MediaMuxer:合成(封裝)
把音軌和視頻軌道合成封裝為新的視頻
主要API介紹
MediaMuxer(path,format):path 輸出文件的名稱;foramt輸出文件的格式,當(dāng)前只支持mp4
addTrack(trackFormat):添加軌道,通常是使用MediaCodec.getOutputForma()或MediaExtractor.getTrackFormat(int index)來(lái)獲取MediaFormat
start():開(kāi)始封裝合成
writeSampleData (int trackIndex, ByteBuffer byteBuf, MediaCodec.BufferInfo bufferInfo): 把數(shù)據(jù)寫(xiě)入到
stop()
release()
封裝(合成)流程如下:
{
MediaFormat trackFormat = mediaExtractor.getTrackFormat(i);
MediaMuxer mediaMuxer;
mediaExtractor.selectTrack(i);
//1. 構(gòu)造MediaMuxer
mediaMuxer = new MediaMuxer(outputFile.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
//2. 添加軌道信息 參數(shù)為MediaFormat
mediaMuxer.addTrack(trackFormat);
//3. 開(kāi)始合成
mediaMuxer.start();
//4. 設(shè)置buffer
ByteBuffer buffer = ByteBuffer.allocate(500 * 1024);
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
//5.通過(guò)mediaExtractor.readSampleData讀取數(shù)據(jù)流
int sampleSize = 0;
while ((sampleSize = mediaExtractor.readSampleData(buffer, 0)) > 0) {
bufferInfo.flags = mediaExtractor.getSampleFlags();
bufferInfo.offset = 0;
bufferInfo.size = sampleSize;
bufferInfo.presentationTimeUs = mediaExtractor.getSampleTime();
int isEOS = bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM;
Log.i(TAG, "demuxerMp4: flags=" + bufferInfo.flags + " size=" + sampleSize + " time=" + bufferInfo.presentationTimeUs + " outputName" + outputName+" isEOS="+isEOS);
//6. 把通過(guò)mediaExtractor解封裝的數(shù)據(jù)通過(guò)writeSampleData寫(xiě)入到對(duì)應(yīng)的軌道
mediaMuxer.writeSampleData(0, buffer, bufferInfo);
mediaExtractor.advance();
}
Log.i(TAG, "extractorAndMuxer: " + outputName + "提取封裝完成");
mediaExtractor.unselectTrack(i);
//6.關(guān)閉
mediaMuxer.stop();
mediaMuxer.release();
}
三、 實(shí)踐(以及ffmpeg的實(shí)現(xiàn))
1. 提取視頻分離出純音頻和純視頻文件
private void extractorAndMuxerMP4() {
tvOut.setText("");
File inputFile = new File(getExternalFilesDir(Environment.DIRECTORY_MUSIC), "forme.mp4");
if (!inputFile.exists()) {
Toast.makeText(this, "文件不存在", Toast.LENGTH_SHORT).show();
return;
}
//數(shù)據(jù)提取(解封裝)
//1. 構(gòu)造MediaExtractor
MediaExtractor mediaExtractor = new MediaExtractor();
try {
//2.設(shè)置數(shù)據(jù)源
mediaExtractor.setDataSource(inputFile.getAbsolutePath());
//3. 獲取軌道數(shù)
int trackCount = mediaExtractor.getTrackCount();
Log.i(TAG, "demuxerMP4: trackCount=" + trackCount);
//遍歷軌道,查看音頻軌或者視頻軌道信息
for (int i = 0; i < trackCount; i++) {
//4. 獲取某一軌道的媒體格式
MediaFormat trackFormat = mediaExtractor.getTrackFormat(i);
String keyMime = trackFormat.getString(MediaFormat.KEY_MIME);
Log.i(TAG, "demuxerMp4: keyMime=" + keyMime);
if (TextUtils.isEmpty(keyMime)) {
continue;
}
//5.通過(guò)mime信息識(shí)別音軌或視頻軌道,打印相關(guān)信息
if (keyMime.startsWith("video/")) {
File outputFile = extractorAndMuxer(mediaExtractor, i, "/video.mp4");
tvOut.setText("純視頻文件路徑:" + outputFile.getAbsolutePath());
Log.i(TAG, "extractorAndMuxerMP4: videoWidth="+trackFormat.getInteger(MediaFormat.KEY_WIDTH)+" videoHeight="+trackFormat.getInteger(MediaFormat.KEY_HEIGHT));
} else if (keyMime.startsWith("audio/")) {
File outputFile = extractorAndMuxer(mediaExtractor, i, "/audio.aac");
Log.i(TAG, "extractorAndMuxerMP4: channelCount="+trackFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT)+" bitRate="+trackFormat.getInteger(MediaFormat.KEY_BIT_RATE));
tvOut.setText(tvOut.getText().toString() + "\n純音頻路徑:" + outputFile.getAbsolutePath());
tvOut.setVisibility(View.VISIBLE);
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
mediaExtractor.release();
}
}
private File extractorAndMuxer(MediaExtractor mediaExtractor, int i, String outputName) throws IOException {
MediaFormat trackFormat = mediaExtractor.getTrackFormat(i);
MediaMuxer mediaMuxer;
mediaExtractor.selectTrack(i);
File outputFile = new File(getExternalFilesDir(Environment.DIRECTORY_MUSIC).getAbsolutePath() + outputName);
if (outputFile.exists()) {
outputFile.delete();
}
//1. 構(gòu)造MediaMuxer
mediaMuxer = new MediaMuxer(outputFile.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
//2. 添加軌道信息 參數(shù)為MediaFormat
mediaMuxer.addTrack(trackFormat);
//3. 開(kāi)始合成
mediaMuxer.start();
//4. 設(shè)置buffer
ByteBuffer buffer = ByteBuffer.allocate(500 * 1024);
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
//5.通過(guò)mediaExtractor.readSampleData讀取數(shù)據(jù)流
int sampleSize = 0;
while ((sampleSize = mediaExtractor.readSampleData(buffer, 0)) > 0) {
bufferInfo.flags = mediaExtractor.getSampleFlags();
bufferInfo.offset = 0;
bufferInfo.size = sampleSize;
bufferInfo.presentationTimeUs = mediaExtractor.getSampleTime();
int isEOS = bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM;
Log.i(TAG, "demuxerMp4: flags=" + bufferInfo.flags + " size=" + sampleSize + " time=" + bufferInfo.presentationTimeUs + " outputName" + outputName+" isEOS="+isEOS);
//6. 把通過(guò)mediaExtractor解封裝的數(shù)據(jù)通過(guò)writeSampleData寫(xiě)入到對(duì)應(yīng)的軌道
mediaMuxer.writeSampleData(0, buffer, bufferInfo);
mediaExtractor.advance();
}
Log.i(TAG, "extractorAndMuxer: " + outputName + "提取封裝完成");
mediaExtractor.unselectTrack(i);
//6.關(guān)閉
mediaMuxer.stop();
mediaMuxer.release();
return outputFile;
}
2. 把純音頻文件和純視頻文件(封裝)合成為視頻文件
/**
* 把音軌和視頻軌再合成新的視頻
*/
private String muxerMp4(String inputAudio , String outPutVideo) {
File videoFile = new File(getExternalFilesDir(Environment.DIRECTORY_MUSIC), "video.mp4");
File audioFile = new File(getExternalFilesDir(Environment.DIRECTORY_MUSIC), inputAudio);
File outputFile = new File(getExternalFilesDir(Environment.DIRECTORY_MUSIC), outPutVideo);
if (outputFile.exists()) {
outputFile.delete();
}
if (!videoFile.exists()) {
Toast.makeText(this, "視頻源文件不存在", Toast.LENGTH_SHORT).show();
return "";
}
if (!audioFile.exists()) {
Toast.makeText(this, "音頻源文件不存在", Toast.LENGTH_SHORT).show();
return "";
}
MediaExtractor videoExtractor = new MediaExtractor();
MediaExtractor audioExtractor = new MediaExtractor();
try {
MediaMuxer mediaMuxer = new MediaMuxer(outputFile.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
int videoTrackIndex = 0;
int audioTrackIndex = 0;
//先添加視頻軌道
videoExtractor.setDataSource(videoFile.getAbsolutePath());
int trackCount = videoExtractor.getTrackCount();
Log.i(TAG, "muxerToMp4: trackVideoCount=" + trackCount);
for (int i = 0; i < trackCount; i++) {
MediaFormat trackFormat = videoExtractor.getTrackFormat(i);
String mimeType = trackFormat.getString(MediaFormat.KEY_MIME);
if (TextUtils.isEmpty(mimeType)) {
continue;
}
if (mimeType.startsWith("video/")) {
videoExtractor.selectTrack(i);
videoTrackIndex = mediaMuxer.addTrack(trackFormat);
Log.i(TAG, "muxerToMp4: videoTrackIndex=" + videoTrackIndex);
break;
}
}
//再添加音頻軌道
audioExtractor.setDataSource(audioFile.getAbsolutePath());
int trackCountAduio = audioExtractor.getTrackCount();
Log.i(TAG, "muxerToMp4: trackCountAduio=" + trackCountAduio);
for (int i = 0; i < trackCountAduio; i++) {
MediaFormat trackFormat = audioExtractor.getTrackFormat(i);
String mimeType = trackFormat.getString(MediaFormat.KEY_MIME);
if (TextUtils.isEmpty(mimeType)) {
continue;
}
if (mimeType.startsWith("audio/")) {
audioExtractor.selectTrack(i);
audioTrackIndex = mediaMuxer.addTrack(trackFormat);
Log.i(TAG, "muxerToMp4: audioTrackIndex=" + audioTrackIndex);
break;
}
}
//再進(jìn)行合成
mediaMuxer.start();
ByteBuffer byteBuffer = ByteBuffer.allocate(500 * 1024);
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int sampleSize = 0;
while ((sampleSize = videoExtractor.readSampleData(byteBuffer, 0)) > 0) {
bufferInfo.flags = videoExtractor.getSampleFlags();
bufferInfo.offset = 0;
bufferInfo.size = sampleSize;
bufferInfo.presentationTimeUs = videoExtractor.getSampleTime();
mediaMuxer.writeSampleData(videoTrackIndex, byteBuffer, bufferInfo);
videoExtractor.advance();
}
int audioSampleSize = 0;
MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
while ((audioSampleSize = audioExtractor.readSampleData(byteBuffer, 0)) > 0) {
audioBufferInfo.flags = audioExtractor.getSampleFlags();
audioBufferInfo.offset = 0;
audioBufferInfo.size = audioSampleSize;
audioBufferInfo.presentationTimeUs = audioExtractor.getSampleTime();
mediaMuxer.writeSampleData(audioTrackIndex, byteBuffer, audioBufferInfo);
audioExtractor.advance();
}
//最后釋放資源
videoExtractor.release();
audioExtractor.release();
mediaMuxer.stop();
mediaMuxer.release();
} catch (IOException e) {
e.printStackTrace();
return "";
}
return outputFile.getAbsolutePath();
}
3. 替換背景音樂(lè),合成新的視頻文件
其實(shí)和第二步一樣了,通過(guò)傳入不同的aac音頻源即可,這里需要注意一點(diǎn),mediamuxer 只支持 aac 格式的,不支持mp3,否則會(huì)報(bào)如下異常,所以需要先把mp3轉(zhuǎn)為aac??梢圆捎胒fmpeg如下命令截取和轉(zhuǎn)換
java.lang.IllegalStateException: Failed to add the track to the muxer
at android.media.MediaMuxer.nativeAddTrack(Native Method)
at android.media.MediaMuxer.addTrack(MediaMuxer.java:638)
—> 添加音軌不是aac格式,而是mp3格式時(shí),在medimuter.addTrack(audioFromat)時(shí)會(huì)報(bào)上述錯(cuò)誤,解決方案:把mp3轉(zhuǎn)成aac
ffmpeg -i 輸入.mp3 -acodec aac 輸出.aac -y
四、遇到的問(wèn)題
4.1 在合成寫(xiě)入數(shù)據(jù)時(shí)報(bào) IllegalArgumentException: trackIndex is invalid
java.lang.IllegalArgumentException: trackIndex is invalid
at android.media.MediaMuxer.writeSampleData(MediaMuxer.java:669)
—> mediaMuxer.writeSampleData(0,buffer,bufferInfo);
原因和方案: 解封裝時(shí)候輸出的trackIndex不對(duì)導(dǎo)致,因?yàn)椴还苁羌円糗夁€是純視頻軌道文件只有一個(gè)軌道。
4.2 解碼出來(lái)的存視頻文件的長(zhǎng)度比原視頻少了,而音頻的長(zhǎng)度一致。
和視頻源有關(guān)系,有的原視頻最后幾秒只有音頻播放畫(huà)面不動(dòng),就是這種情況,剛開(kāi)時(shí)不知道,還以為是什么bug,最后通過(guò)ffmpeg直接對(duì)原視頻進(jìn)行提取,得到的結(jié)果一樣。
用ffmpeg命令提取純視頻 對(duì)比看下
ffmpeg -i 輸入.mp4 -vcodec copy -an 輸出.mp4 -y 查看生成的視頻也是一樣。
說(shuō)明這個(gè)視頻中視頻流就是比音頻流要短。
ffmpeg -i 輸入.mp4 -acodec copy -vn 輸出.aac -y 查看生成的音頻流。和通過(guò)medieExtractor和mediamuxter提取的一致。
4.3 mediaExtractor.advance()時(shí)報(bào)IllegalArgumentException: bufferInfo must specify a valid buffer
通過(guò)查看bufferinfo的信息此時(shí)flags和presntationTimeUs都為-1,是advance調(diào)用時(shí)間不對(duì)引起
java.lang.IllegalArgumentException: bufferInfo must specify a valid buffer offset, size and presentation time
at android.media.MediaMuxer.writeSampleData(MediaMuxer.java:682)
解決方案: 先調(diào)mediaMuxer.writeSampleData 后再mediaExtractor.advance();
4.4 合成時(shí)報(bào)如下錯(cuò)誤,這個(gè)mediaMuxer.start之前沒(méi)有添加軌道導(dǎo)致(流程不熟導(dǎo)致)
java.lang.IllegalStateException: Failed to start the muxer
at android.media.MediaMuxer.nativeStart(Native Method)
at android.media.MediaMuxer.start(MediaMuxer.java:452)
start之前只是構(gòu)造了mediaMuxer但沒(méi)有mediaMuxer.addTrack(trackFormat);
4.5 在把音軌和視頻軌道合成新視頻時(shí),復(fù)用了MediaTractor導(dǎo)致異常
java.io.IOException: Failed to instantiate extractor.
com.av.mediajourney W/System.err: at android.media.MediaExtractor.nativeSetDataSource(Native Method)
com.av.mediajourney W/System.err: at android.media.MediaExtractor.setDataSource(MediaExtractor.java:203)
解決:在把視頻軌道的源文件路徑通過(guò)setDataSource設(shè)置到mediaextractor后,再把音軌的源文件setdataSource就報(bào)了這個(gè)錯(cuò)誤
正確的做法是針對(duì)每一個(gè)源設(shè)置一個(gè)MediaExtractor,不同共用
4.6 把純音軌和純視頻軌道合成新視頻后,播放視頻沒(méi)有聲音 時(shí)間是對(duì)的,但是沒(méi)有聲音
猜測(cè) 會(huì)不會(huì)是因?yàn)檐壍?是視頻,軌道1是音頻的原因? 用ffmpeg對(duì)比查看了下原視頻和合成的視頻這點(diǎn)有些差異,嘗試調(diào)下順序看下。
—>調(diào)整后沒(méi)有效果。。。繼續(xù)通過(guò)兩證ffmpeg -i輸出信息定位,發(fā)現(xiàn)metaChange不同
—>折騰了半天也沒(méi)結(jié)果,最后通過(guò)ffplay來(lái)播放合成的視頻,一切正常,只能說(shuō)播放器的原因吧,也可能在合成時(shí)設(shè)置不全導(dǎo)致部分播放器無(wú)法播放,暫時(shí)不得結(jié)果
4.7 還出現(xiàn)一種情況是合成后的時(shí)長(zhǎng)變成了音頻加視頻的時(shí)長(zhǎng)總和了
原因是bufferInfo.presentationTimeUs的值不對(duì)導(dǎo)致。
異常實(shí)現(xiàn)如下:
` ByteBuffer byteBuffer = ByteBuffer.allocate(500 * 1024);
//先計(jì)算出視頻幀間隔時(shí)間
long smapleTime = 0;
videoExtractor.readSampleData(byteBuffer, 0);
if (videoExtractor.getSampleFlags() == MediaExtractor.SAMPLE_FLAG_SYNC) {
videoExtractor.advance();
}
videoExtractor.readSampleData(byteBuffer, 0);
long secondTime = videoExtractor.getSampleTime();
videoExtractor.advance();
long thirdtime = videoExtractor.getSampleTime();
smapleTime = Math.abs(thirdtime - secondTime);
Log.i(TAG, "muxerStart: smapleTime=" + smapleTime);
videoExtractor.unselectTrack(videoTrackIndex);
videoExtractor.selectTrack(videoTrackIndex);
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int sampleSize = 0;
while ((sampleSize = videoExtractor.readSampleData(byteBuffer, 0)) > 0) {
bufferInfo.flags = videoExtractor.getSampleFlags();
bufferInfo.offset = 0;
bufferInfo.size = sampleSize;
bufferInfo.presentationTimeUs += smapleTime;
//bufferInfo.presentationTimeUs = videoExtractor.getSampleTime();
mediaMuxer.writeSampleData(videoTrackIndex, byteBuffer, bufferInfo);
videoExtractor.advance();
}
int audioSampleSize = 0;
ByteBuffer audioByteBuffer = ByteBuffer.allocate(500 * 1024);
MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
while ((audioSampleSize = audioExtractor.readSampleData(byteBuffer, 0)) > 0) {
audioBufferInfo.flags = audioExtractor.getSampleFlags();
audioBufferInfo.offset = 0;
audioBufferInfo.size = audioSampleSize;
audioBufferInfo.presentationTimeUs += smapleTime;
// audioBufferInfo.presentationTimeUs = videoExtractor.getSampleTime();
mediaMuxer.writeSampleData(audioTrackIndex, audioByteBuffer, audioBufferInfo);
audioExtractor.advance();
}
這些遇到的問(wèn)題一部分是對(duì)mediaExtractor和mediaMuxer的流程不熟悉導(dǎo)致。而有些需要借助ffpmpeg和ffplay進(jìn)行協(xié)助分析排查。
五、參考
[Android 音視頻學(xué)習(xí):使用 MediaExtractor 和 MediaMuxer 解析和封裝 mp4 文件](https://mp.weixin.qq.com/s/KsOdAnTCQ7B_V7agZ0OwXQ)
[Android 視頻分離和合成(MediaMuxer和MediaExtractor)](https://blog.csdn.net/zhi184816/article/details/52514138)
六、收獲
- 了解MediaExtractor和Mediamuxer的作用
- MediaExtractor和Mediamuxer熟悉API和使用流程
- 提取音軌和視頻軌然后進(jìn)行再合成或者替換音軌實(shí)現(xiàn)換背景音樂(lè)
- 遇到問(wèn)題的分析解決以及復(fù)盤(pán)。
感謝你的閱讀。
下一篇我們來(lái)一起學(xué)習(xí)實(shí)踐MediaCodec硬編硬解,歡迎關(guān)注“音視頻開(kāi)發(fā)之旅”交流。
歡迎討論