Mongodb secondary上面replSet error RS102 too stale to catch up

1 http://101.130.92.32:28018/_replSet 监控网站上报错:replSet error RS102 too stale to catch up
Tue Aug 13 05:18:06 [rsSync] replSet See http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+Replica+Set+Member
05:18:06 [rsSync] replSet error RS102 too stale to catch up
05:20:07 [rsSync] replSet syncing to: 101.130.92.323:27018
05:20:07 [rsSync] replSet error RS102 too stale to catch up, at lbgst from 101.130.92.323:27018
05:20:07 [rsSync] replSet our last optime : Aug 6 19:18:05 52014bed:2
05:20:07 [rsSync] replSet oldest at 101.130.92.323:27018 : Aug 10 10:58:26 52061cd2:17
05:20:07 [rsSync] replSet See http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+Replica+Set+Member
05:20:07 [rsSync] replSet error RS102 too stale to catch up
05:22:08 [rsSync] replSet syncing to: 101.130.92.323:27018
05:22:08 [rsSync] replSet error RS102 too stale to catch up, at lbgst from 101.130.92.323:27018
05:22:08 [rsSync] replSet our last optime : Aug 6 19:18:05 52014bed:2
05:22:08 [rsSync] replSet oldest at 101.130.92.323:27018 : Aug 10 10:59:17 52061d05:3e

2 察看状态,发现是recovring状态
[bguser@346437-bgstore_db1 ~]$ /db/mongodb/bin/mongo --port 27018
MongoDB shell version: 2.0.1
connecting to: 127.0.0.1:27018/test
RECOVERING> db.printReplicationInfo();
configured oplog size: 90000MB
log length start to end: 553610secs (153.78hrs)
oplog first event time: Wed Jul 31 2013 09:31:15 GMT+0000 (UTC)
oplog last event time: Tue Aug 06 2013 19:18:05 GMT+0000 (UTC)
now: Tue Aug 13 2013 06:03:40 GMT+0000 (UTC)
RECOVERING> rs.status();
{
"set" : "rpl",
"date" : ISODate("2013-08-13T06:03:49Z"),
"myState" : 3,
"syncingTo" : "101.130.92.323:27018",
"members" : [
{
"_id" : 0,
"name" : "101.130.92.32:27017",
"hbglth" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 8433,
"optime" : {
"t" : 1376373827000,
"i" : 16
},
"optimeDate" : ISODate("2013-08-13T06:03:47Z"),
"lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
"pingMs" : 0
},
{
"_id" : 4,
"name" : "101.130.92.32:27019",
"hbglth" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 8435,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
"pingMs" : 0
},
{
"_id" : 5,
"name" : "101.130.92.32:27018",
"hbglth" : 1,
"state" : 3,
"stateStr" : "RECOVERING",
"optime" : {
"t" : 1375816685000,
"i" : 2
},
"optimeDate" : ISODate("2013-08-06T19:18:05Z"), --同步时间也不对,是8月6日的,今天以及国内13日了
"self" : true
},
{
"_id" : 6,
"name" : "101.130.92.323:27017",
"hbglth" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 8433,
"optime" : {
"t" : 1376373827000,
"i" : 17
},
"optimeDate" : ISODate("2013-08-13T06:03:47Z"),
"lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
"pingMs" : 0
},
{
"_id" : 7,
"name" : "101.130.92.323:27018",
"hbglth" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 8435,
"optime" : {
"t" : 1376373827000,
"i" : 17
},
"optimeDate" : ISODate("2013-08-13T06:03:47Z"),
"lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
"pingMs" : 0
}
],
"ok" : 1
}
RECOVERING>

3 去官网上面察看资料,发现如下
http://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#replica-set-resync-by-copying

•Restart the mongod with an empty data directory and let MongoDB’s normal initial syncing fbgture restore the data.
This is the more simple option, but may take longer to replace the data.

重新启动 步骤如下:
(1) stop the failed mongod 停止正在运行的同步异常的mongodb服务
(2) delete all data in the dbpath (including subdirectories) 删除mongodb目录下所有的数据文件包括其他的子目录
(3) start it and it will automatically resynchronize itself from primary 启动mongodb服务，服务会自动从primary上重新同步数据过来。

4 重新sync后，状态一直是 RECOVERING没有变成secondary，不过数据都已经同步过去了。
RECOVERING> rs.status();
{
"set" : "rpl",
"date" : ISODate("2013-08-13T07:58:41Z"),
"myState" : 3,
"syncingTo" : "101.130.92.323:27018",
"members" : [
{
"_id" : 0,
"name" : "101.130.92.322:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 5629,
"optime" : {
"t" : 1376380719000,
"i" : 84
},
"optimeDate" : ISODate("2013-08-13T07:58:39Z"),
"lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
"pingMs" : 0
},
{
"_id" : 4,
"name" : "101.130.92.322:27019",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 5629,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
"pingMs" : 0
},
{
"_id" : 5,
"name" : "101.130.92.322:27018",
"health" : 1,
"state" : 3,
"stateStr" : "RECOVERING",
"optime" : {
"t" : 1376380721000,
"i" : 31
},
"optimeDate" : ISODate("2013-08-13T07:58:41Z"),
"self" : true
},
{
"_id" : 6,
"name" : "101.130.92.323:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 5629,
"optime" : {
"t" : 1376380719000,
"i" : 66
},
"optimeDate" : ISODate("2013-08-13T07:58:39Z"),
"lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
"pingMs" : 0
},
{
"_id" : 7,
"name" : "101.130.92.323:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 5629,
"optime" : {
"t" : 1376380719000,
"i" : 66
},
"optimeDate" : ISODate("2013-08-13T07:58:39Z"),
"lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
"pingMs" : 0
}
],
"ok" : 1
}
RECOVERING>

还需要继续观察一段时间，看是否最终会从RECOVERING变成secondary。

观察了1天之后,还是RECOVRING, 日志里面也没有报任何异常信息.只好尝试着重新启动了下

kill 12292

/db/mongodb/bin/mongod -f /etc/mongodb/27018.conf

结果status变成了secondary了.