[Disability Report] 30 Oct., 2017: OmniPath network failure

2017.10.31

Fault occured in OminiPath network as follows and now resolved.

1.Period

30 Oct., 2017 (Mon.) 19:42 - 21:53

2.Influence

Around 30 Oct. (Mon.) 19:42, a failure occured in the Omini-Path network. During this time, could not access between about 340 computation nodes and storage (Luster, NFS). In login0 and login1, could be accessed normally.

We are verifying version 10.6 released on 27 Oct. (Fri.) of Fabric Manager which controls Omni-Path to fix this.

3.Recovery

Restart the Fabric Manager which controls Omni-Path.

4.Jobs that mey have been affected (jobs that were running during the failure period).

136032,136032,136032,136032,136032,136032,136032,136032,136032,136032,137515,137528,137754,137755,137756,137757,137758,137759,137760,137761,137762,137763,137764,137765,137766,137767,137768,137769,137770,137826,137827,137952,138115,138252,138257,138265,138266,138268,138274,138275,138276,138454,138456,138458,138459,138461,138462,138491,138492,138493,138499,138505,138533,138556,138560,138570,138571,138574,138575,138576,138643,138677,138707,138710,138718,138744,138748,138756,138767,138778,138843,138886,138995,139182,139186,139198,139222,139254,139255,139256,139182,139186,139198,139222,139254,139255,139256,139257,139307,139323,139324,139327,139331,139335,139533,139592,139593,139822,139896,139923,139928,139929,139930,140053,140054,140055,140065,140110,140119,140124,140126,140128,140132,140134,140137,140138,140142,140143,140144,140145,140148,140149,140151,140153,140154,140158,140160,140161,140162,140163,140164,140165,140166,140167,140168,140180,140185,140194,140197,140199,140201,140202,140206,140338,140342,140347,140357,140383,140385,140386,140387,140388,140392,140393,140397,140447,140453,140464,140465,140489,140495,140527,140540,140543,140544,140548,140549,140554,140555,140556,140557,140566,140571,140572,140574,140575,140576,140585,140589,140597,140600,140602,140603,140604,140605,140607,140608,140609,140610,140611,140618,140620,140621,140627,140630,140631,140632,140633,140635,140636,140637,138865,139717,140051,140491,140493,140502,140512,140513,140580