[Node] 內(nèi)存溢出與 old-space 大小調(diào)整

1. 內(nèi)存溢出

V8 為 Node.js 應(yīng)用，默認(rèn)只會(huì)分配了大概 1400 MB（僅本地測(cè)試的結(jié)果） 的內(nèi)存空間。
超出了這個(gè)限額，就會(huì)內(nèi)存溢出。

我們來寫一段程序，制造一個(gè)內(nèi)存溢出錯(cuò)誤，

// index.js

const main = n => {
  const is = [];
  while (true) {
    const js = [];
    for (let j = 0; j < n; j++) {
      const ks = [];
      for (let k = 0; k < n; k++) {
        const ls = [];
        for (let l = 0; l < n; l++) {
          ls.push([]);
        }
        ks.push(ls);
      }
      js.push(ks);
    }
    is.push(js);
    showMemoryUsage();
  }
};

const showMemoryUsage = () => {
  const { rss, heapTotal, heapUsed } = process.memoryUsage();
  log(`rss = ${byteToMB(rss)}, heapTotal = ${byteToMB(heapTotal)}, heapUsed = ${byteToMB(heapUsed)}`);
};

const byteToMB = byte => `${(byte / 1024 / 1024).toFixed(2)} MB`;
const log = message => console.log(message);

main(10);

執(zhí)行一下，

$ node index.js
rss = 22.30 MB, heapTotal = 6.23 MB, heapUsed = 3.88 MB
rss = 22.74 MB, heapTotal = 9.23 MB, heapUsed = 3.83 MB
...
rss = 1450.73 MB, heapTotal = 1426.73 MB, heapUsed = 1400.11 MB
rss = 1450.77 MB, heapTotal = 1426.73 MB, heapUsed = 1400.17 MB

<--- Last few GCs --->

[40615:0x103800000]    19849 ms: Mark-sweep 1398.1 (1424.7) -> 1398.0 (1425.7) MB, 2473.5 / 0.0 ms  (+ 0.0 ms in 20 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2592 ms) (average mu = 0.124, current mu = 0.084) alloca

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x234ab8cdbe3d]
Security context: 0x016d8749e6c1 <JSObject>
    1: main [0x16d0208b7c1] [/Users/.../index.js:~1] [pc=0x234ab8ce5238](this=0x016d5928d461 <JSGlobal Object>,n=10)
    2: /* anonymous */ [0x16d0208b7f9] [/Users/.../index.js:29] [bytecode=0x16df5ad7991 offset=43](this=0x016d0208b929 <Object map = 0x16d3ce02571>,exports=0x016d0208b929 <Object map = 0x16d3ce02571>,require=0x0...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100d68631 node::Abort() (.cold.1) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 2: 0x10003b124 node_module_register [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 3: 0x10003b2e5 node::OnFatalError(char const*, char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 4: 0x1001a9097 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 5: 0x1001a9031 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 6: 0x100593592 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 7: 0x100595d33 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 8: 0x100591bda v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 9: 0x10058fac5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
10: 0x10059cc55 v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
11: 0x10059ccbf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
12: 0x10056cdb3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
13: 0x1007fddaf v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
14: 0x234ab8cdbe3d
15: 0x234ab8ce5238
16: 0x234ab8c918d5
17: 0x234ab8c918d5
18: 0x234ab8c918d5
19: 0x234ab8c918d5
20: 0x234ab8c918d5
[1]    40615 abort      node index.js

可以看到當(dāng)內(nèi)存占用達(dá)到 1400 MB 時(shí)，就報(bào)錯(cuò)了，

rss = 1450.77 MB, heapTotal = 1426.73 MB, heapUsed = 1400.17 MB

2. new-space & old-space

V8 使用了分代垃圾回收機(jī)制，將堆（heap）劃分成了幾個(gè)不同的空間。

參考一篇比較老的文章，A tour of V8: Garbage Collection #Heap organization

  • New-space
  • Old-pointer-space
  • Old-data-space
  • Large-object-space
  • Code-space
  • Cell-space, property-cell-space and map-space

文中提到了 new-space 和 old-space，

In the vast majority of programs, objects tend to die young: most objects have a very short lifetime, while a small minority of objects tend to live much longer. To take advantage of this behavior, V8 divides the heap into two generations. Objects are allocated in new-space, which is fairly small (between 1 and 8 MB, depending on behavior heuristics). Allocation in new space is very cheap: we just have an allocation pointer which we increment whenever we want to reserve space for a new object. When the allocation pointer reaches the end of new space, a scavenge (minor garbage collection cycle) is triggered, which quickly removes the dead objects from new space. Objects which have survived two minor garbage collections are promoted to old-space. Old-space is garbage collected during a mark-sweep or mark-compact (major garbage collection cycle), which is much less frequent. A major garbage collection cycle is triggered when we have promoted a certain amount of memory to old space. This threshold shifts over time depending on the size of old space and the behavior of the program.

大致含義是，新對(duì)象一般會(huì)先分配到 new-space 中（只有 1 - 8 M 大?。?br> new-space 滿了以后會(huì)觸發(fā)一次小型的（minor）垃圾回收，
如果對(duì)象經(jīng)歷了兩次小型（minor）垃圾回收還活著，就會(huì)被移到 old-space 中（這個(gè)空間比較大）。
old-space 滿了會(huì)經(jīng)歷一次大型（major）垃圾回收（采用標(biāo)記-清除回收算法）。

所以，如果我們一直分配新對(duì)象且不釋放它，就會(huì)最終被放到 old-space 中。
一旦 old-space 占用率超出限額，就會(huì)造成內(nèi)存溢出。

注：
V8 的垃圾回收機(jī)制也是與時(shí)俱進(jìn)的，
最近的一些進(jìn)展，可參考 Trash talk: the Orinoco garbage collector，
上面介紹的內(nèi)容大同小異。

3. 調(diào)整 old-space 大小

有以下兩種方式，可以調(diào)整 old-space 大小，
在某些情況下，或許能暫時(shí)避免 Node.js 內(nèi)存溢出。

$ node --max-old-space-size=2048 index.js

$ NODE_OPTIONS='--max-old-space-size=2048' node index.js

我們來看看效果，

$ node --max-old-space-size=2048 index.js
rss = 22.30 MB, heapTotal = 6.23 MB, heapUsed = 3.88 MB
rss = 22.74 MB, heapTotal = 9.23 MB, heapUsed = 3.83 MB
...

rss = 2110.81 MB, heapTotal = 2084.73 MB, heapUsed = 2047.35 MB
rss = 2110.86 MB, heapTotal = 2084.73 MB, heapUsed = 2047.40 MB

<--- Last few GCs --->

[40880:0x103800000]    36415 ms: Mark-sweep 2045.2 (2082.2) -> 2045.1 (2082.7) MB, 3743.6 / 0.0 ms  (+ 0.0 ms in 13 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3907 ms) (average mu = 0.108, current mu = 0.061) alloca[40880:0x103800000]    36455 ms: Scavenge 2046.4 (2083.7) -> 2046.4 (2084.7) MB, 13.2 / 0.0 ms  (average mu = 0.108, current mu = 0.061) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x37e08555be3d]
Security context: 0x10607f39e6c1 <JSObject>
    1: main [0x1060cbe8c051] [/Users/.../index.js:~1] [pc=0x37e0855f0ab8](this=0x10607f10d461 <JSGlobal Object>,n=10)
    2: /* anonymous */ [0x1060cbe8c089] [/Users/.../index.js:29] [bytecode=0x10602e4d79e1 offset=43](this=0x1060cbe8c1b9 <Object map = 0x1060e9202571>,exports=0x1060cbe8c1b9 <Object map = 0x1060e9202571>,requir...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100d68631 node::Abort() (.cold.1) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 2: 0x10003b124 node_module_register [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 3: 0x10003b2e5 node::OnFatalError(char const*, char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 4: 0x1001a9097 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 5: 0x1001a9031 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 6: 0x100593592 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 7: 0x100595d33 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 8: 0x100591bda v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 9: 0x10058fac5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
10: 0x10059cc55 v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
11: 0x10059ccbf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
12: 0x10056cdb3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
13: 0x1007fddaf v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
14: 0x37e08555be3d
15: 0x37e0855f0ab8
16: 0x37e0855118d5
17: 0x37e0855118d5
18: 0x37e0855118d5
19: 0x37e0855118d5
20: 0x37e0855118d5
[1]    40880 abort      node --max-old-space-size=2048 index.js

我們看到產(chǎn)生內(nèi)存溢出時(shí)，內(nèi)存占用情況不同了，

# 默認(rèn) old-space 大小
rss = 1450.77 MB, heapTotal = 1426.73 MB, heapUsed = 1400.17 MB

# 調(diào)整 old-space 大小為 2048 MB
rss = 2110.86 MB, heapTotal = 2084.73 MB, heapUsed = 2047.40 MB

參考

A tour of V8: Garbage Collection
Trash talk: the Orinoco garbage collector
Node.js - Command line options: --max-old-space-size=SIZE
Node.js - Command line options: NODE_OPTIONS=options...

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

友情鏈接更多精彩內(nèi)容