有次突然觀察到線上很多獲取圖片出現(xiàn)響應(yīng)碼500的錯(cuò)誤。由于歷史遺留原因，以前的圖片文件保存在MongoDB的GridFS里面。這次突然出現(xiàn)獲取圖片500的錯(cuò)誤，很有可能MongoDB出問(wèn)題了。

用mongo命令連接到PRIMARY節(jié)點(diǎn)上去查看副本集的狀態(tài)，如下：

ptcupload:PRIMARY> db.version()
2.4.8
ptcupload:PRIMARY> rs.status()
{
    "set" : "ptcupload",
    "date" : ISODate("2016-04-11T15:56:08Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "ptcupload-mongodb-001:27018",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 6355878,
            "optime" : Timestamp(1460390155, 1),
            "optimeDate" : ISODate("2016-04-11T15:55:55Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "ptcupload-mongodb-002:27018",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 45351,
            "optime" : Timestamp(1459863715, 2),
            "optimeDate" : ISODate("2016-04-05T13:41:55Z"),
            "lastHeartbeat" : ISODate("2016-04-11T15:56:08Z"),
            "lastHeartbeatRecv" : ISODate("2016-04-11T15:56:08Z"),
            "pingMs" : 0,
            "lastHeartbeatMessage" : "syncThread: 10334 BSONObj size: 1853163520 (0x0008756E) is invalid. Size must be between 0 and 16793600(16MB) First element: que: ?type=105",
            "syncingTo" : "ptcupload-mongodb-001:27018"
        },
        {
            "_id" : 2,
            "name" : "ptcupload-mongodb-003:37018",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 5992973,
            "lastHeartbeat" : ISODate("2016-04-11T15:56:07Z"),
            "lastHeartbeatRecv" : ISODate("2016-04-11T15:56:07Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}

整個(gè)副本集由一個(gè)PRIMARY、一個(gè)SECONDARY和一個(gè)ARBITER節(jié)點(diǎn)組成。從上面輸出的副本集狀態(tài)可以看到SECONDARY節(jié)點(diǎn)同步出現(xiàn)了問(wèn)題：

"lastHeartbeatMessage" : "syncThread: 10334 BSONObj size: 1853163520 (0x0008756E) is invalid. Size must be between 0 and 16793600(16MB) First element: que: ?type=105"

還觀察到SECONDARY節(jié)點(diǎn)的uptime為45351秒（12.59小時(shí)），遠(yuǎn)小于PRIMARY節(jié)點(diǎn)的uptime. 由此，懷疑12.59小時(shí)前SECONDARY節(jié)點(diǎn)宕機(jī)自動(dòng)重啟過(guò)。

我司MongoDB數(shù)據(jù)庫(kù)有專門(mén)的數(shù)據(jù)庫(kù)管理員管理，我們業(yè)務(wù)使用方?jīng)]有查看服務(wù)器相關(guān)日志的權(quán)限，因此不方便調(diào)查故障原因。不過(guò)，對(duì)于這種情況，有一種修復(fù)辦法，即將有問(wèn)題的SECONDARY節(jié)點(diǎn)撤掉，重新加一個(gè)新的SECONDARY節(jié)點(diǎn)。