快手在2020年中旬開源了一個線上OOM監(jiān)控上報的框架:KOOM,這里簡單研究下。
項目地址:https://github.com/KwaiAppTeam/KOOM/tree/v1.0.5
一、官方項目介紹
1.1 描述:
KOOM是快手性能優(yōu)化團隊在處理移動端OOM問題的過程中沉淀出的一套完整解決方案。其中Android Java內(nèi)存部分在LeakCanary的基礎(chǔ)上進行了大量優(yōu)化,解決了線上內(nèi)存監(jiān)控的性能問題,在不影響用戶體驗的前提下線上采集內(nèi)存鏡像并解析。從 2020 年春節(jié)后在快手主APP上線至今解決了大量OOM問題,其性能和穩(wěn)定性經(jīng)受住了海量用戶與設(shè)備的考驗,因此決定開源以回饋社區(qū)。
1.2 特點:
- 比leakCanary更豐富的泄漏場景檢測;
- 比leakCanary更好的檢測性能;
- 功能全面的支持線上大規(guī)模部署的閉環(huán)監(jiān)控系統(tǒng);
1.3 KOOM框架

1.4 快手KOOM核心流程包括:
- 配置下發(fā)決策;
- 監(jiān)控內(nèi)存狀態(tài);
- 采集內(nèi)存鏡像;
- 解析鏡像文件(以下簡稱hprof)生成報告并上傳;
- 問題聚合報警與分配跟進。
1.5 泄漏檢測觸發(fā)機制優(yōu)化:
泄漏檢測觸發(fā)機制leakCanary做法是GC過后對象WeakReference一直不被加入 ReferenceQueue,它可能存在內(nèi)存泄漏。這個過程會主動觸發(fā)GC做確認(rèn),可能會造成用戶可感知的卡頓,而KOOM采用內(nèi)存閾值監(jiān)控來觸發(fā)鏡像采集,將對象是否泄漏的判斷延遲到了解析時,閾值監(jiān)控只要在子線程定期獲取關(guān)注的幾個內(nèi)存指標(biāo)即可,性能損耗很低。

1.6 heap dump優(yōu)化:
傳統(tǒng)方案會凍結(jié)應(yīng)用幾秒,KOOM會fork新進程來執(zhí)行dump操作,對父進程的正常執(zhí)行沒有影響。暫停虛擬機需要調(diào)用虛擬機的art::Dbg::SuspendVM函數(shù),谷歌從Android 7.0開始對調(diào)用系統(tǒng)庫做了限制,快手自研了kwai-linker組件,通過caller address替換和dl_iterate_phdr解析繞過了這一限制。

隨機采集線上真實用戶的內(nèi)存鏡像,普通dump和fork子進程dump阻塞用戶使用的耗時如下:

而從官方給出的測試數(shù)據(jù)來看,效果似乎是非常好的。
二、官方demo演示
這里就直接跑下官方提供的koom-demo
點擊按鈕,經(jīng)過dump heap -> heap analysis -> report cache/koom/report/三個流程(heap analysis時間會比較長,但是完全不影響應(yīng)用的正常操作),最終在應(yīng)用的cache/koom/report里生成json報告:
cepheus:/data/data/com.kwai.koom.demo/cache/koom/report # ls
2020-12-08_15-23-32.json
模擬一個最簡單的單例CommonUtils持有LeakActivity實例的內(nèi)存泄漏,看下json最終上報的內(nèi)容是個啥:
{
"analysisDone":true,
"classInfos":[
{
"className":"android.app.Activity",
"instanceCount":4,
"leakInstanceCount":3
},
{
"className":"android.app.Fragment",
"instanceCount":4,
"leakInstanceCount":3
},
{
"className":"android.graphics.Bitmap",
"instanceCount":115,
"leakInstanceCount":0
},
{
"className":"libcore.util.NativeAllocationRegistry",
"instanceCount":1513,
"leakInstanceCount":0
},
{
"className":"android.view.Window",
"instanceCount":4,
"leakInstanceCount":0
}
],
"gcPaths":[
{
"gcRoot":"Local variable in native code",
"instanceCount":1,
"leakReason":"Activity Leak",
"path":[
{
"declaredClass":"java.lang.Thread",
"reference":"android.os.HandlerThread.contextClassLoader",
"referenceType":"INSTANCE_FIELD"
},
{
"declaredClass":"java.lang.ClassLoader",
"reference":"dalvik.system.PathClassLoader.runtimeInternalObjects",
"referenceType":"INSTANCE_FIELD"
},
{
"declaredClass":"",
"reference":"java.lang.Object[]",
"referenceType":"ARRAY_ENTRY"
},
{
"declaredClass":"com.kwai.koom.demo.CommonUtils",
"reference":"com.kwai.koom.demo.CommonUtils.context",
"referenceType":"STATIC_FIELD"
},
{
"reference":"com.kwai.koom.demo.LeakActivity",
"referenceType":"instance"
}
],
"signature":"378fc01daea06b6bb679bd61725affd163d026a8"
}
],
"runningInfo":{
"analysisReason":"RIGHT_NOW",
"appVersion":"1.0",
"buildModel":"MI 9 Transparent Edition",
"currentPage":"LeakActivity",
"dumpReason":"MANUAL_TRIGGER",
"jvmMax":512,
"jvmUsed":2,
"koomVersion":1,
"manufacture":"Xiaomi",
"nowTime":"2020-12-08_16-07-34",
"pss":32,
"rss":123,
"sdkInt":29,
"threadCount":17,
"usageSeconds":40,
"vss":5674
}
}
這里主要分三個部分:類信息、gc引用路徑、運行基本信息。這里從gcPaths中能看出LeakActivity被CommonUtils持有了引用。
框架使用這里參考官方接入文檔即可,這里不贅述:
https://github.com/KwaiAppTeam/KOOM/blob/master/README.zh-CN.md
三、框架解析
3.1 類圖

3.2 時序圖
KOOM初始化流程

KOOM執(zhí)行初始化方法,10秒延遲之后會在threadHandler子線程中先通過check狀態(tài)判斷是否開始工作,工作內(nèi)容是先檢查是不是有未完成分析的文件,如果有就就觸發(fā)解析,沒有則監(jiān)控內(nèi)存。
heap dump流程

HeapDumpTrigger
- startTrack:監(jiān)控自動觸發(fā)dump hprof操作。開啟內(nèi)存監(jiān)控,子線程5s觸發(fā)一次檢測,看當(dāng)前是否滿足觸發(fā)heap dump的條件。條件是由一系列閥值組織,這部分后面詳細(xì)分析。滿足閥值后會通過監(jiān)聽回調(diào)給HeapDumpTrigger去執(zhí)行trigger。
- trigger:主動觸發(fā)dump hprof操作。這里是fork子進程來處理的,這部分也到后面詳細(xì)分析。dump完成之后通過監(jiān)聽回調(diào)觸發(fā)HeapAnalysisTrigger.startTrack觸發(fā)heap分析流程。
heap analysis流程

HeapAnalysisTrigger
- startTrack 根據(jù)策略觸發(fā)hprof文件分析。
- trigger 直接觸發(fā)hprof文件分析。由單獨起進程的service來處理,工作內(nèi)容主要分內(nèi)存泄漏檢測(activity/fragment/bitmap/window)和泄漏數(shù)據(jù)整理緩存為json文件以供上報。
四、核心源碼解析
經(jīng)過前面的分析,基本上對框架的使用和結(jié)構(gòu)有了一個宏觀了解,這部分就打算對一些實現(xiàn)細(xì)節(jié)進行簡單分析。
4.1 內(nèi)存監(jiān)控觸發(fā)dump規(guī)則
這里主要是研究HeapMonitor中isTrigger規(guī)則,每隔5S都會循環(huán)判斷該觸發(fā)條件。
com/kwai/koom/javaoom/monitor/HeapMonitor.java
@Override
public boolean isTrigger() {
if (!started) {
return false;
}
HeapStatus heapStatus = currentHeapStatus();
if (heapStatus.isOverThreshold) {
if (heapThreshold.ascending()) {
if (lastHeapStatus == null || heapStatus.used >= lastHeapStatus.used) {
currentTimes++;
} else {
currentTimes = 0;
}
} else {
currentTimes++;
}
} else {
currentTimes = 0;
}
lastHeapStatus = heapStatus;
return currentTimes >= heapThreshold.overTimes();
}
private HeapStatus lastHeapStatus;
private HeapStatus currentHeapStatus() {
HeapStatus heapStatus = new HeapStatus();
heapStatus.max = Runtime.getRuntime().maxMemory();
heapStatus.used = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
heapStatus.isOverThreshold = 100.0f * heapStatus.used / heapStatus.max > heapThreshold.value();
return heapStatus;
}
com/kwai/koom/javaoom/common/KConstants.java
public static class HeapThreshold {
public static int VM_512_DEVICE = 510;
public static int VM_256_DEVICE = 250;
public static int VM_128_DEVICE = 128;
public static float PERCENT_RATIO_IN_512_DEVICE = 80;
public static float PERCENT_RATIO_IN_256_DEVICE = 85;
public static float PERCENT_RATIO_IN_128_DEVICE = 90;
public static float getDefaultPercentRation() {
int maxMem = (int) (Runtime.getRuntime().maxMemory() / MB);
if (maxMem >= VM_512_DEVICE) {
return KConstants.HeapThreshold.PERCENT_RATIO_IN_512_DEVICE;
} else if (maxMem >= VM_256_DEVICE) {
return KConstants.HeapThreshold.PERCENT_RATIO_IN_256_DEVICE;
} else if (maxMem >= VM_128_DEVICE) {
return KConstants.HeapThreshold.PERCENT_RATIO_IN_128_DEVICE;
}
return KConstants.HeapThreshold.PERCENT_RATIO_IN_512_DEVICE;
}
public static int OVER_TIMES = 3;
public static int POLL_INTERVAL = 5000;
}
這里就是針對不同內(nèi)存大小做了不同的閥值比例:
- 應(yīng)用內(nèi)存>512M 80%
- 應(yīng)用內(nèi)存>256M 85%
- 應(yīng)用內(nèi)存>128M 90%
- 低于128M的默認(rèn)按80%
應(yīng)用已使用內(nèi)存/最大內(nèi)存超過該比例則會觸發(fā)heapStatus.isOverThreshold。連續(xù)滿足3次觸發(fā)heap dump,但是這個過程會考慮內(nèi)存增長性,3次范圍內(nèi)出現(xiàn)了使用內(nèi)存下降或者使用內(nèi)存/最大內(nèi)存低于對應(yīng)閥值了則清零。
因此規(guī)則總結(jié)為:3次滿足>閥值條件且內(nèi)存一直處于上升期才觸發(fā)。這樣能減少無效的dump。
4.2 fork進程執(zhí)行dump操作實現(xiàn)
目前項目中默認(rèn)使用ForkJvmHeapDumper來執(zhí)行dump。
com/kwai/koom/javaoom/dump/ForkJvmHeapDumper.java
@Override
public boolean dump(String path) {
boolean dumpRes = false;
try {
int pid = trySuspendVMThenFork();//暫停虛擬機,copy-on-write fork子進程
if (pid == 0) {//子進程中
Debug.dumpHprofData(path);//dump hprof
exitProcess();//_exit(0) 退出進程
} else {//父進程中
resumeVM();//resume當(dāng)前虛擬機
dumpRes = waitDumping(pid);//waitpid異步等待pid進程結(jié)束
}
} catch (Exception e) {
e.printStackTrace();
}
return dumpRes;
}
谷歌從Android 7.0開始對調(diào)用系統(tǒng)庫做了限制,快手自研了kwai-linker組件,通過caller address替換和dl_iterate_phdr解析繞過了這一限制,詳細(xì)分析在后續(xù)的這篇文章有展開:KOOM V1.0.5 fork dump方案解析。
官方文檔對限制的說明:
https://developer.android.google.cn/about/versions/nougat/android-7.0-changes
4.3 內(nèi)存泄漏檢測實現(xiàn)
內(nèi)存泄漏檢測核心代碼在于SuspicionLeaksFinder.find
public Pair<List<ApplicationLeak>, List<LibraryLeak>> find() {
boolean indexed = buildIndex();
if (!indexed) {
return null;
}
initLeakDetectors();
findLeaks();
return findPath();
}
4.3.1 buildIndex()
private boolean buildIndex() {
Hprof hprof = Hprof.Companion.open(hprofFile.file());
//選擇可以作為gcroot的類類型
KClass<GcRoot>[] gcRoots = new KClass[]{
Reflection.getOrCreateKotlinClass(GcRoot.JniGlobal.class),
//Reflection.getOrCreateKotlinClass(GcRoot.JavaFrame.class),
Reflection.getOrCreateKotlinClass(GcRoot.JniLocal.class),
//Reflection.getOrCreateKotlinClass(GcRoot.MonitorUsed.class),
Reflection.getOrCreateKotlinClass(GcRoot.NativeStack.class),
Reflection.getOrCreateKotlinClass(GcRoot.StickyClass.class),
Reflection.getOrCreateKotlinClass(GcRoot.ThreadBlock.class),
Reflection.getOrCreateKotlinClass(GcRoot.ThreadObject.class),
Reflection.getOrCreateKotlinClass(GcRoot.JniMonitor.class)};
//解析hprof文件為HeapGraph對象
heapGraph = HprofHeapGraph.Companion.indexHprof(hprof, null,
kotlin.collections.SetsKt.setOf(gcRoots));
return true;
}
fun indexHprof(
hprof: Hprof,
proguardMapping: ProguardMapping? = null,
indexedGcRootTypes: Set<KClass<out GcRoot>> = setOf(
JniGlobal::class,
JavaFrame::class,
JniLocal::class,
MonitorUsed::class,
NativeStack::class,
StickyClass::class,
ThreadBlock::class,
ThreadObject::class,
JniMonitor::class
)
): HeapGraph {
//確認(rèn)對應(yīng)的record的index
val index = HprofInMemoryIndex.createReadingHprof(hprof, proguardMapping, indexedGcRootTypes)
//HprofHeapGraph是HeapGraph的實現(xiàn)類
return HprofHeapGraph(hprof, index)
}
HprofInMemoryIndex.createReadingHprof核心邏輯:讀取hprof文件,將不同內(nèi)容封裝為不同的record,然后將record轉(zhuǎn)為索引化的index封裝,之后查找內(nèi)容可以通過index去索引到。
HprofReader.readHprofRecords() 封裝record
- LoadClassRecord
- InstanceSkipContentRecord
- ObjectArraySkipContentRecord
- PrimitiveArraySkipContentRecord
HprofInMemoryIndex.onHprofRecord() 封裝index:
- classIndex
- instanceIndex
- objectArrayIndex
- primitiveArrayIndex
class HprofHeapGraph internal constructor(
private val hprof: Hprof,
private val index: HprofInMemoryIndex
) : HeapGraph {
...
override val gcRoots: List<GcRoot>
get() = index.gcRoots()
override val objects: Sequence<HeapObject>
get() {
return index.indexedObjectSequence().map {wrapIndexedObject(it.second, it.first)}
}
override val classes: Sequence<HeapClass>
get() {
return index.indexedClassSequence().map {val objectId = it.first
val indexedObject = it.second
HeapClass(this, indexedObject, objectId)
}
}
override val instances: Sequence<HeapInstance>
get() {
return index.indexedInstanceSequence().map {val objectId = it.first
val indexedObject = it.second
val isPrimitiveWrapper = index.primitiveWrapperTypes.contains(indexedObject.classId)
HeapInstance(this, indexedObject, objectId, isPrimitiveWrapper)
}
}
override val objectArrays: Sequence<HeapObjectArray>
get() = index.indexedObjectArraySequence().map {val objectId = it.first
val indexedObject = it.second
val isPrimitiveWrapper = index.primitiveWrapperTypes.contains(indexedObject.arrayClassId)
HeapObjectArray(this, indexedObject, objectId, isPrimitiveWrapper)
}
override val primitiveArrays: Sequence<HeapPrimitiveArray>
get() = index.indexedPrimitiveArraySequence().map {val objectId = it.first
val indexedObject = it.second
HeapPrimitiveArray(this, indexedObject, objectId)
}
Hprof 經(jīng)過層層轉(zhuǎn)換最終封裝為HprofHeapGraph。
簡而言之,這部分功能主要是將Hrpof文件按照掃描的格式解析為結(jié)構(gòu)化的索引關(guān)系圖,索引化后的內(nèi)容封裝為HprofHeapGraph,由它去通過對應(yīng)的起始索引去定位每類數(shù)據(jù)。沒有細(xì)摳這部分的實現(xiàn)細(xì)節(jié),實現(xiàn)這個功能的庫之前是squere的HAHA,現(xiàn)在改為shark,但是提供的功能大同小異。
4.3.2 initLeakDetectors() 與findLeaks()
初始化泄漏檢測者:
private void initLeakDetectors() {
addDetector(new ActivityLeakDetector(heapGraph));
addDetector(new FragmentLeakDetector(heapGraph));
addDetector(new BitmapLeakDetector(heapGraph));
addDetector(new NativeAllocationRegistryLeakDetector(heapGraph));
addDetector(new WindowLeakDetector(heapGraph));
ClassHierarchyFetcher.initComputeGenerations(computeGenerations);
leakReasonTable = new HashMap<>();
}
初始化各類型泄漏的檢測者,主要包含Activity、Fragment、Bitmap+NativeAllocationRegistry、window的泄漏檢測。
其次是梳理以上幾類對象類繼承關(guān)系串,檢測覆蓋到他們的子類。
public void findLeaks() {
KLog.i(TAG, "start find leaks");
//從HprofHeapGraph中獲取所有instance
Sequence<HeapObject.HeapInstance> instances = heapGraph.getInstances();
Iterator<HeapObject.HeapInstance> instanceIterator = instances.iterator();
while (instanceIterator.hasNext()) {
HeapObject.HeapInstance instance = instanceIterator.next();
if (instance.isPrimitiveWrapper()) {
continue;
}
ClassHierarchyFetcher.process(instance.getInstanceClassId(),
instance.getInstanceClass().getClassHierarchy());
for (LeakDetector leakDetector : leakDetectors) {
//是檢測對象的子類&滿足對應(yīng)泄漏條件
if (leakDetector.isSubClass(instance.getInstanceClassId())
&& leakDetector.isLeak(instance)) {
ClassCounter classCounter = leakDetector.instanceCount();
if (classCounter.leakInstancesCount <=
SAME_CLASS_LEAK_OBJECT_GC_PATH_THRESHOLD) {
leakingObjects.add(instance.getObjectId());
leakReasonTable.put(instance.getObjectId(), leakDetector.leakReason());
}
}
}
}
//關(guān)注class和對應(yīng)instance數(shù)量,加入json
HeapAnalyzeReporter.addClassInfo(leakDetectors);
findPrimitiveArrayLeaks();
findObjectArrayLeaks();
}
這里重點看看各類型對象是如何判斷泄漏的:
ActivityLeakDetector:
private static final String ACTIVITY_CLASS_NAME = "android.app.Activity";
private static final String FINISHED_FIELD_NAME = "mFinished";
private static final String DESTROYED_FIELD_NAME = "mDestroyed";
public boolean isLeak(HeapObject.HeapInstance instance) {
activityCounter.instancesCount++;
HeapField destroyField = instance.get(ACTIVITY_CLASS_NAME, DESTROYED_FIELD_NAME);
HeapField finishedField = instance.get(ACTIVITY_CLASS_NAME, FINISHED_FIELD_NAME);
assert destroyField != null;
assert finishedField != null;
boolean abnormal = destroyField.getValue().getAsBoolean() == null
|| finishedField.getValue().getAsBoolean() == null;
if (abnormal) {
return false;
}
boolean leak = destroyField.getValue().getAsBoolean()
|| finishedField.getValue().getAsBoolean();
if (leak) {
activityCounter.leakInstancesCount++;
}
return leak;
}
mDestroyed和mFinish字段為true,但是實例還存在的Activity是疑似泄漏對象。
FragmentLeakDetector:
private static final String NATIVE_FRAGMENT_CLASS_NAME = "android.app.Fragment";
// native android Fragment, deprecated as of API 28.
private static final String SUPPORT_FRAGMENT_CLASS_NAME = "android.support.v4.app.Fragment";
// pre-androidx, support library version of the Fragment implementation.
private static final String ANDROIDX_FRAGMENT_CLASS_NAME = "androidx.fragment.app.Fragment";
// androidx version of the Fragment implementation
private static final String FRAGMENT_MANAGER_FIELD_NAME = "mFragmentManager”;
private static final String FRAGMENT_MCALLED_FIELD_NAME = "mCalled”;//Used to verify that subclasses call through to super class.
public FragmentLeakDetector(HeapGraph heapGraph) {
HeapObject.HeapClass fragmentHeapClass =
heapGraph.findClassByName(ANDROIDX_FRAGMENT_CLASS_NAME);
fragmentClassName = ANDROIDX_FRAGMENT_CLASS_NAME;
if (fragmentHeapClass == null) {
fragmentHeapClass = heapGraph.findClassByName(NATIVE_FRAGMENT_CLASS_NAME);
fragmentClassName = NATIVE_FRAGMENT_CLASS_NAME;
}
if (fragmentHeapClass == null) {
fragmentHeapClass = heapGraph.findClassByName(SUPPORT_FRAGMENT_CLASS_NAME);
fragmentClassName = SUPPORT_FRAGMENT_CLASS_NAME;
}
assert fragmentHeapClass != null;
fragmentClassId = fragmentHeapClass.getObjectId();
fragmentCounter = new ClassCounter();
}
public boolean isLeak(HeapObject.HeapInstance instance) {
if (VERBOSE_LOG) {
KLog.i(TAG, "run isLeak");
}
fragmentCounter.instancesCount++;
boolean leak = false;
HeapField fragmentManager = instance.get(fragmentClassName, FRAGMENT_MANAGER_FIELD_NAME);
if (fragmentManager != null && fragmentManager.getValue().getAsObject() == null) {
HeapField mCalledField = instance.get(fragmentClassName, FRAGMENT_MCALLED_FIELD_NAME);
boolean abnormal = mCalledField == null || mCalledField.getValue().getAsBoolean() == null;
if (abnormal) {
KLog.e(TAG, "ABNORMAL mCalledField is null");
return false;
}
leak = mCalledField.getValue().getAsBoolean();
if (leak) {
if (VERBOSE_LOG) {
KLog.e(TAG, "fragment leak : " + instance.getInstanceClassName());
}
fragmentCounter.leakInstancesCount++;
}
}
return leak;
}
這里分了三種fragment:
- android.app.Fragment
- android.support.v4.app.Fragment
- androidx.fragment.app.Fragment
對應(yīng)的FragmentManager實例為null(這表示fragment被remove了)且滿足對應(yīng)的mCalled為true,即非perform狀態(tài),而是對應(yīng)生命周期被回調(diào)狀態(tài)(onDestroy),但是實例還存在的Fragment是疑似泄漏對象。
BitmapLeakDetector
private static final String BITMAP_CLASS_NAME = "android.graphics.Bitmap”;
public boolean isLeak(HeapObject.HeapInstance instance) {
if (VERBOSE_LOG) {
KLog.i(TAG, "run isLeak");
}
bitmapCounter.instancesCount++;
HeapField fieldWidth = instance.get(BITMAP_CLASS_NAME, "mWidth");
HeapField fieldHeight = instance.get(BITMAP_CLASS_NAME, "mHeight");
assert fieldHeight != null;
assert fieldWidth != null;
boolean abnormal = fieldHeight.getValue().getAsInt() == null
|| fieldWidth.getValue().getAsInt() == null;
if (abnormal) {
KLog.e(TAG, "ABNORMAL fieldWidth or fieldHeight is null");
return false;
}
int width = fieldWidth.getValue().getAsInt();
int height = fieldHeight.getValue().getAsInt();
boolean suspicionLeak = width * height >= KConstants.BitmapThreshold.DEFAULT_BIG_BITMAP;
if (suspicionLeak) {
KLog.e(TAG, "bitmap leak : " + instance.getInstanceClassName() + " " +
"width:" + width + " height:" + height);
bitmapCounter.leakInstancesCount++;
}
return suspicionLeak;
}
這里是針對Bitmap size做判斷,超過768*1366這個size的認(rèn)為泄漏。
另外,NativeAllocationRegistryLeakDetector和WindowLeakDetector兩類還沒做具體泄漏判斷規(guī)則,不參與對象泄漏檢測,只是做了統(tǒng)計。
總結(jié):
整體看下來,KOOM有兩個值得借鑒的點:
1.觸發(fā)內(nèi)存泄漏檢測,常規(guī)是watcher activity/fragment的onDestroy,而KOOM是定期輪詢查看當(dāng)前內(nèi)存是否到達閥值;
2.dump hprof,常規(guī)是對應(yīng)進程dump,而KOOM是fork進程dump。