2019.9.20
The compute nodes stopped and recovered as follows.
１．period
from 2019/9/19 14:08 to 2019/9/20 10:00
２．content
When we proceeded with the recovery work for Unit 2 of the cooling tower that had been stopped due to a 9/9 failure, all the cooling towers stopped abnormally. As a result, all compute nodes were stopped as well, and the TSUBAME portal was also stopped manually.
After that, it was found that the relay circuit of the No. 2 sprinkler pump was damaged, and not only the No. 2 unit but also all three relays were replaced, and the No. 1 and No. 3 units were restarted to stabilize cooling. The operation of the compute node was resumed at 10:00 on September 20, 2019.
It seems that there is no problem with the second machine, but the re-operation will be performed during the planned maintenance scheduled for November.
３．affected jobs
TSUBAME points consumed by the job affected by the failure will be automatically returned at a later date.
AR: 4547, 4546, 4556
5737502, 5752664, 5756926, 5756941, 5758572, 5759803, 5759841, 5759851, 5759861, 5759871, 5760242, 5760243, 5760244, 5760245, 5760246, 5760247, 5760248, 5760249, 5761674, 5761730, 5763049, 5763050, 5763052, 5763169, 5763348, 5763349, 5763352, 5763387, 5763389, 5763401, 5763435, 5763436, 5763437, 5763438, 5763493, 5763495, 5763496, 5763822, 5764239, 5764327, 5764408, 5764413, 5764416, 5764424, 5764429, 5764432, 5764433, 5764436, 5764438, 5764441, 5764442, 5764445, 5764446, 5764447, 5764450, 5764500, 5764664, 5764668, 5764706, 5764709, 5764713, 5764717, 5764755, 5764765, 5764773, 5764895, 5764901, 5764905, 5764906, 5765678, 5765839, 5765840, 5765841, 5765842, 5765844, 5765845, 5765846, 5765848, 5765849, 5765850, 5765957, 5765958, 5765959, 5765961, 5765962, 5765964, 5765968, 5766072, 5766073, 5766074, 5766075, 5766076, 5766077, 5766079, 5766080, 5766081, 5766082, 5766083, 5766084, 5766085, 5766086, 5766087, 5766088, 5766089, 5766090, 5766091, 5766092, 5766093, 5766094, 5766095, 5766096, 5766097, 5766098, 5766099, 5766102, 5766109, 5766110, 5766111, 5766113, 5766114, 5766115, 5766116, 5766117, 5766123, 5766124, 5766125, 5766126, 5766127, 5766128, 5766129, 5766130, 5766132, 5766133, 5766134, 5766135, 5766137, 5766138, 5766139, 5766140, 5766155, 5766156, 5766160, 5766161, 5766167, 5766176, 5766177, 5766178, 5766443, 5766458, 5766460, 5766461, 5766463, 5766465, 5766466, 5766471, 5766521, 5766532, 5766533, 5766535, 5766556, 5766557, 5766570, 5766575, 5766577, 5766582, 5766583, 5766633, 5766643, 5766650, 5766651, 5766652, 5766667, 5766671, 5766677, 5766686, 5766696, 5766704, 5766971, 5767606, 5767607, 5767608, 5767609, 5767610, 5767611, 5767612, 5767613, 5767614, 5767615, 5767616, 5767617, 5767618, 5767619, 5767620, 5767621, 5767622, 5767623, 5767624, 5767625, 5767626, 5767627, 5767628, 5767629, 5767630, 5767631, 5767632, 5767633, 5767634, 5767635, 5767636, 5767637, 5767638, 5767639, 5767640, 5767641, 5767642, 5767643, 5767644, 5767645, 5767647, 5767648, 5767649, 5767650, 5767651, 5767652, 5767653, 5767654, 5767655, 5767656, 5767657, 5767658, 5767659, 5767660, 5767661, 5767662, 5767663, 5767664, 5767665, 5767666, 5767667, 5767668, 5767669, 5767670, 5767740, 5767741, 5767742, 5767800, 5767801, 5767802, 5767803, 5767804, 5767805, 5767806, 5767807, 5767808, 5767809, 5767813, 5767815, 5767817, 5767819, 5767822, 5767824, 5767825, 5767826, 5767827, 5767828, 5767829, 5767830, 5767831, 5767832, 5767833, 5767834, 5767835, 5767836, 5767837, 5767838, 5767839, 5767840, 5767841, 5767842, 5767843, 5767844, 5767845, 5767846, 5767847, 5767848, 5767849, 5767850, 5767851, 5767852, 5767853, 5767854, 5767855, 5767856, 5767857, 5767858, 5767859, 5767861, 5767866, 5767869, 5767871, 5768059, 5768065, 5768066, 5768067, 5768068, 5768069, 5768070, 5768071, 5768072, 5768073, 5768074, 5768075, 5768079, 5768080, 5768102, 5768117, 5768143, 5768286, 5768288, 5768289, 5768296, 5768299, 5768302, 5768320, 5768331, 5768334, 5768439, 5768442, 5768443, 5768445, 5768447, 5768456, 5768457, 5768458, 5768459, 5768461, 5768463, 5768466, 5768468, 5768479, 5768480, 5768481, 5768482, 5768483, 5768484, 5768485, 5768486, 5768487, 5768488, 5768489, 5768490, 5768491, 5768492, 5768493, 5768494, 5768495, 5768496, 5768497, 5768498, 5768499, 5768500, 5768501, 5768502, 5768503, 5768504, 5768505, 5768506, 5768507, 5768508, 5768509, 5768510, 5768511, 5768512, 5768513, 5768514, 5768515, 5768516, 5768517, 5768518, 5768519, 5768520, 5768521, 5768522, 5768523, 5768524, 5768525, 5768526, 5768527, 5768528, 5768529, 5768531, 5768533, 5768542, 5768558, 5768570, 5768586, 5768611, 5768613, 5768615, 5768675, 5768677, 5768698, 5768708, 5768718, 5768729, 5768747, 5768753, 5768754, 5768756, 5768776, 5768781, 5768788, 5768793, 5768797, 5768836, 5768839, 5768844, 5768889, 5768894, 5768895, 5768897, 5768898, 5768899, 5768900, 5768901, 5768902, 5768903, 5768904, 5768905, 5768906, 5768907, 5768908, 5768909, 5768910, 5768912, 5768913, 5768914, 5768915, 5768916, 5768917, 5768918, 5768919, 5768920, 5768921, 5768922, 5768923, 5768924, 5768925, 5768926, 5768927, 5768928, 5768929, 5768930, 5768937, 5768982, 5768990, 5768993, 5768994, 5768995, 5768998, 5769002, 5769003, 5769004, 5769007, 5769008, 5769011, 5769014, 5769017, 5769021, 5769023, 5769034, 5769035, 5769037, 5769038, 5769040, 5769041, 5769043, 5769045, 5769047, 5769093, 5769097, 5769104, 5769106, 5769138, 5769139, 5769157, 5769174, 5769187, 5769196, 5769200, 5769201, 5769213