"trouble report" 2019.9.9 : all of compute nodes stopped due to cooling system trouble

2019.9.20

Compute nodes stopped and recovered as follows.

We are sorry for the late posting this information.

1.period

from 2019/9/9 4:56 to 2019/9/10 9:00

2.content

Around 4:56 on September 9, 2019, the rooftop cooling tower leaked due to the typhoon and cooling stopped.

As the result, the compute nodes were sequentially stopped to protect the system from rising temperatures, and all running jobs were stopped.

In addition, the TSUBAME portal was also manually stopped so that the operation would not cause any error because all compute nodes were stopped.

Judging from the situation, it was judged that there was a leakage due to rain, and we checked for leakage, insulation, and equipment damage.

Two of the three cooling towers restarted due to wiping off moisture, etc., and cooling started, but considering the damage if it stops again at night, we confirmed that the cooling was stable, next 2019 The operation of the compute node resumed at 9:00 am on September 10th.
The troubled second machine is investigated in detail to investigate the cause and prevent recurrence.

3.affected jobs

TSUBAME points consumed by the job affected by the failure will be automatically returned at a later date.

reservations:4436, 4466, 4477, 4511

5651422, 5671371, 5690016, 5708739, 5708755, 5708772, 5718502, 5721708, 5724270, 5724274, 5724686, 5724708, 5724719, 5724732, 5725491, 5727081, 5727210, 5727537, 5728581, 5728582, 5730037, 5730038, 5730039, 5730040, 5730041, 5730068, 5730399, 5730406, 5730407, 5731736, 5732284, 5732317, 5732322, 5732337, 5732414, 5732442, 5732445, 5732447, 5732450, 5732453, 5732454, 5732461, 5732470, 5732472, 5732473, 5732490, 5732518, 5732521, 5732526, 5732531, 5732614, 5732619, 5732623, 5732627, 5732680, 5732817, 5732827, 5732876, 5732879, 5732880, 5732881, 5732882, 5732883, 5732923, 5733044, 5733257, 5733271, 5733430, 5733433, 5733434, 5733588, 5733740, 5733805, 5733878, 5733879, 5733880, 5733881, 5733882, 5733884, 5733885, 5733886, 5733887, 5733888, 5733889, 5733890, 5733891, 5733892, 5733893, 5733894, 5733895, 5733896, 5733897, 5733898, 5733899, 5733900, 5733903, 5733904, 5733905, 5733906, 5733907, 5733908, 5733909, 5733910, 5733911, 5733912, 5733913, 5733914, 5734019, 5734020, 5734022, 5734024, 5734051, 5734054, 5734057, 5734076, 5734077, 5734079, 5734082, 5734100, 5734101, 5734103, 5734117, 5734118, 5734119, 5734120, 5734121, 5734122, 5734123, 5734133, 5734163, 5734179, 5734195, 5734211, 5734227, 5734243, 5734259, 5734275, 5734291, 5734307, 5734323, 5734356, 5734372, 5734388, 5734405, 5734422, 5734438, 5734454, 5734470, 5734486, 5734502, 5734518, 5734535, 5734551, 5734552, 5734566, 5734568, 5734569, 5734571, 5734576, 5734590, 5734591, 5734592, 5734593, 5734594, 5734595, 5734596, 5734619, 5734623, 5734626, 5734630, 5734631, 5734632, 5734633, 5734647, 5734649, 5734650, 5734651, 5734652, 5734653, 5734654, 5734655, 5734656, 5734657, 5734658, 5734660, 5734661, 5734662, 5734676, 5734677, 5734678, 5734679, 5734680, 5734681, 5734682, 5734683, 5734684, 5734685, 5734686, 5734687, 5734688, 5734689, 5734690, 5734691, 5734692, 5734693, 5734706, 5734708, 5734709, 5734710, 5734713, 5734715, 5734716, 5734717, 5734730, 5734731, 5734732, 5734733, 5734734, 5734735, 5734736, 5734737, 5734738, 5734739, 5734740, 5734741, 5734742, 5734743, 5734759, 5734761, 5734762, 5734763, 5734764, 5734767, 5734768, 5734769, 5734771, 5734772, 5734773, 5734774, 5734775, 5734776, 5734778, 5734795, 5734797, 5734798, 5734799, 5734800, 5734801, 5734802, 5734803, 5734804, 5734805, 5734806, 5734807, 5734808, 5734809, 5734810, 5734811, 5734812, 5734813, 5734814, 5734815, 5734816, 5734817, 5734818, 5734819, 5734820, 5734821, 5734822, 5734823, 5734824, 5734825, 5734826, 5734827, 5734828, 5734829, 5734830, 5734831, 5734832, 5734833, 5734834, 5734835, 5734836, 5734837, 5734838, 5734839, 5734840, 5734841, 5734842, 5734843, 5734844, 5734845, 5734846, 5734847, 5734848, 5734849, 5734850, 5734851, 5734852, 5734853, 5734854, 5734855, 5734856, 5734857, 5734858, 5734859, 5734861, 5734862, 5734863, 5734864, 5734865, 5734866, 5734872, 5734873, 5734874, 5734875, 5734876, 5734877, 5734878, 5734879, 5734880, 5734881, 5734882, 5734883, 5734901, 5734902, 5734903, 5734904, 5734905, 5734906, 5734907, 5734908, 5734909, 5734910, 5734911, 5734912, 5734913, 5734925, 5734926, 5734927, 5734928, 5734929, 5734930, 5734931, 5734932, 5734933, 5734934, 5734935, 5734936, 5734937, 5734938, 5734939, 5734940, 5734941, 5734942, 5734943, 5734944, 5734945, 5734946, 5734947, 5734948, 5734949, 5734950, 5734951, 5734952, 5734953, 5734954, 5734955, 5734956, 5734957, 5734958, 5734959, 5734960, 5734961, 5734962, 5734963, 5734964, 5734965, 5734966, 5734967, 5734968, 5734969, 5734970, 5734971, 5734972, 5734976, 5734977, 5734978, 5734979, 5734981, 5734982, 5734988, 5734989, 5734991, 5734992, 5734993, 5734994, 5734995, 5734996, 5735002, 5735006, 5735007, 5735008, 5735009, 5735010, 5735011, 5735013, 5735014, 5735044, 5735045, 5735046, 5735047, 5735048, 5735049, 5735050, 5735051, 5735052, 5735053, 5735054, 5735055, 5735056, 5735057, 5735058, 5735059, 5735060, 5735061, 5735062, 5735063, 5735064, 5735065, 5735066, 5735067, 5735068, 5735069, 5735070, 5735071, 5735072, 5735073, 5735074, 5735075, 5735076, 5735077, 5735078, 5735079, 5735080, 5735081, 5735082, 5735083, 5735084, 5735174, 5735175, 5735176, 5735183, 5735184, 5735188, 5735189, 5735190, 5735191, 5735192, 5735193, 5735208