下記の通り障害が発生し復旧しました。
１．概要
/gs/hs1へのアクセスが滞る。失敗する。
２．期間
2021年5月22日(土)10:00頃 から 17:10頃 その後縮退中
３．詳細 (5/25追記)
5/22 10:00頃に/gs/hs1の領域を構成する、LustreファイルシステムにおけるOSSの一つであるossb0にて障害が発生しました。休日だったため16:35頃にossb0がハングしたような状況に至っていることを確認しました。障害が発生すると自動で検知しフェイルオーバー処理が行われますが、ossb0が完全にダウンとならず、かろうじて稼働しているような状態となっていたため、フェイルオーバーが作動しておりませんでした。
対処としてossb0を強制再起動致しました。その際HFIカードのエラーが見られたため、Lustreにおけるディスクの集合体であるOSTはossb1へフェイルオーバーさせ、サービスはペアとなるossb1側に片寄せ状態に縮退して17:10頃仮復旧しました。
現在、原因の調査、切り分けを行っており、完全な復旧時期は未定です。
(5/25追記)ストレージ障害の原因がわかり部品の交換を行いました。現在、縮退状態の影響でパフォーマンスが低下が見られることから本日16:00に本来の構成に戻す作業（テイクオーバー）を行います。ファイルシステムの負荷が低い場合は5分程度で完了しますが、負荷が高い場合は数十分要する事が考えられます。その間、ファイルシステム全体は停止しませんが、対象のディスクグループは応答がなくなり、完了後、I/Oは継続されます。
(5/25 16:30追記) 上記メンテナンスは無事完了しました。現在は本来の性能で動作しております。
４．影響があったジョブ
障害が影響した可能性の高いジョブは以下の通りです。1336件のジョブの内、/gs/hs1領域を使用していたジョブに影響があったと推測されます。
9119893 9125581 9125582 9125583 9125584 9125585 9125586 9127460 9127468 9127472 9127474 9127477 9127484 9127487 9127489 9127491 9127661 9127662 9127664 9127665 9127666 9127667 9127668 9127669 9127670 9127671 9127672 9127673 9127674 9127675 9127676 9127678 9127679 9127680 9127681 9127683 9127684 9127685 9127686 9127687 9127688 9128002 9128003 9128004 9128006 9128008 9128009 9128010 9128011 9128012 9128014 9128015 9128016 9128017 9128018 9128020 9128021 9128022 9128023 9128024 9128026 9128027 9128028 9128029 9128030 9128031 9128032 9128033 9128034 9128035 9128036 9128037 9128038 9128039 9128040 9128041 9128042 9128043 9128044 9128045 9128046 9128047 9128048 9128049 9128050 9128051 9128052 9128053 9128054 9128055 9128056 9128057 9128058 9128059 9128060 9128061 9128062 9128063 9128064 9128065 9128066 9128067 9128068 9128069 9128070 9128071 9128072 9128073 9128074 9128075 9128076 9129061 9129062 9129064 9129065 9129066 9129074 9129078 9129079 9129080 9129081 9129083 9129084 9129095 9129096 9129097 9129098 9129099 9129100 9129103 9129104 9129105 9129106 9129107 9129108 9129109 9129110 9129119 9129122 9129123 9129124 9129126 9129127 9129128 9129130 9129131 9129132 9129133 9129134 9129135 9129137 9129139 9129140 9129141 9129142 9129143 9130066 9130092 9130093 9130094 9130095 9130096 9130097 9130098 9130099 9130100 9130101 9130102 9130103 9130104 9130105 9130106 9130107 9130108 9130109 9130110 9130111 9130112 9130113 9130114 9130115 9130116 9130117 9130118 9130119 9130120 9130121 9130122 9130123 9130124 9130503 9131018 9131019 9131020 9131021 9131022 9131023 9131024 9131025 9131026 9131027 9131028 9131029 9131030 9131031 9131032 9131284 9131285 9132722 9132726 9132782 9132784 9132787 9132791 9132800 9132821 9132822 9132823 9132824 9132825 9132827 9132840 9132841 9132842 9132843 9132844 9132863 9132865 9133053 9133056 9133059 9133089 9133094 9133100 9133213 9133219 9133230 9133298 9133473 9133712 9133844 9133851 9133961 9133962 9133963 9133964 9133965 9134182 9134865 9134873 9134883 9134884 9134885 9134886 9134887 9135442 9135546 9135659 9135660 9135695 9135699 9135711 9135713 9135716 9135720 9135722 9135729 9135730 9135731 9135732 9135733 9135737 9135899 9135900 9135901 9135902 9135903 9135904 9135905 9135986 9135987 9135988 9135989 9135990 9135991 9135992 9135993 9135994 9135995 9136117 9136118 9136124 9136126 9136127 9136140 9136146 9136147 9136149 9136150 9136151 9136152 9136153 9136155 9136159 9136160 9136162 9136164 9136165 9136168 9136170 9136172 9136176 9136183 9136190 9136191 9136192 9136193 9136194 9136195 9136197 9136198 9136199 9136201 9136202 9136212 9136214 9136215 9136216 9136217 9136218 9136219 9136220 9136221 9136222 9136223 9136224 9136225 9136226 9136335 9136336 9136338 9136339 9136340 9136341 9136342 9136343 9136344 9136345 9136346 9136347 9136348 9136349 9136350 9136351 9136352 9136353 9136354 9136355 9136440 9136490 9136506 9136507 9136508 9136509 9136510 9136512 9136513 9136514 9136515 9136516 9136520 9136522 9136523 9136524 9136525 9136528 9136586 9136741 9136758 9136760 9136763 9136764 9136765 9136766 9136767 9136768 9136769 9136770 9136771 9136774 9136791 9136799 9136806 9136811 9136814 9136820 9136829 9136839 9136847 9136901 9137018 9137243 9137972 9137984 9137985 9137986 9137987 9137988 9137989 9137990 9137991 9137992 9137993 9139508 9139597 9139599 9139600 9139601 9139603 9139604 9139605 9139606 9139608 9139609 9139611 9139612 9139614 9139615 9139616 9139617 9139619 9139620 9139621 9139622 9139759 9139821 9139893 9139938 9139952 9139953 9139954 9139955 9139956 9139957 9139958 9139959 9139960 9139961 9139962 9139963 9139964 9139965 9139966 9139967 9139968 9139969 9139970 9139971 9139972 9139973 9139974 9139975 9139976 9139977 9139978 9139979 9139980 9139981 9139982 9139983 9139984 9139985 9139986 9139987 9139988 9139989 9139990 9139991 9139992 9139993 9139994 9139995 9139996 9139997 9139998 9139999 9140000 9140001 9140002 9140003 9140004 9140005 9140006 9140007 9140008 9140009 9140010 9140011 9140012 9140013 9140014 9140015 9140016 9140017 9140018 9140019 9140020 9140021 9140022 9140023 9140024 9140025 9140026 9140027 9140028 9140029 9140030 9140031 9140032 9140033 9140034 9140035 9140036 9140037 9140038 9140039 9140040 9140041 9140042 9140043 9140044 9140045 9140046 9140047 9140048 9140049 9140050 9140051 9140052 9140053 9140054 9140055 9140056 9140057 9140058 9140059 9140060 9140061 9140062 9140063 9140064 9140065 9140066 9140067 9140068 9140069 9140070 9140071 9140072 9140073 9140074 9140075 9140076 9140077 9140078 9140079 9140080 9140081 9140082 9140083 9140084 9140085 9140086 9140087 9140088 9140089 9140090 9140091 9140092 9140093 9140094 9140095 9140096 9140097 9140098 9140099 9140100 9140101 9140102 9140103 9140104 9140105 9140106 9140107 9140108 9140109 9140110 9140111 9140112 9140113 9140114 9140115 9140116 9140117 9140118 9140119 9140120 9140121 9140122 9140123 9140124 9140125 9140126 9140127 9140128 9140129 9140130 9140131 9140132 9140133 9140134 9140135 9140136 9140137 9140138 9140139 9140140 9140141 9140142 9140143 9140144 9140145 9140146 9140147 9140148 9140149 9140150 9140151 9140152 9140160 9140164 9140165 9140166 9140168 9140169 9140170 9140171 9140172 9140173 9140174 9140175 9140176 9140177 9140178 9140179 9140180 9140181 9140182 9140183 9140184 9140185 9140186 9140187 9140188 9140189 9140190 9140191 9140192 9140193 9140194 9140195 9140196 9140197 9140198 9140199 9140201 9140202 9140203 9140205 9140208 9140209 9140210 9140211 9140212 9140213 9140214 9140215 9140216 9140217 9140218 9140219 9140220 9140221 9140222 9140223 9140224 9140225 9140226 9140227 9140228 9140229 9140230 9140231 9140232 9140233 9140234 9140235 9140236 9140237 9140238 9140239 9140240 9140241 9140242 9140243 9140244 9140245 9140246 9140247 9140248 9140249 9140250 9140251 9140252 9140253 9140254 9140255 9140256 9140257 9140258 9140259 9140260 9140261 9140285 9140286 9140287 9140288 9140289 9140290 9140291 9140292 9140293 9140294 9140295 9140296 9140297 9140298 9140299 9140300 9140301 9140302 9140303 9140304 9140306 9140307 9140308 9140310 9140311 9140317 9140318 9140319 9140320 9140321 9140322 9140323 9140324 9140325 9140326 9140327 9140328 9140329 9140330 9140331 9140332 9140333 9140334 9140335 9140336 9140337 9140338 9140339 9140340 9140341 9140342 9140343 9140344 9140368 9140369 9140370 9140371 9140372 9140373 9140374 9140375 9140376 9140377 9140378 9140379 9140380 9140381 9140382 9140383 9140384 9140385 9140386 9140387 9140388 9140389 9140390 9140391 9140392 9140393 9140394 9140399 9140400 9140401 9140402 9140403 9140404 9140405 9140406 9140407 9140408 9140409 9140410 9140411 9140412 9140413 9140414 9140415 9140416 9140417 9140418 9140419 9140449 9140450 9140451 9140452 9140453 9140454 9140455 9140456 9140457 9140458 9140459 9140460 9140461 9140462 9140463 9140464 9140465 9140466 9140467 9140468 9140469 9140470 9140471 9140472 9140473 9140474 9140475 9140476 9140477 9140478 9140479 9140480 9140481 9140482 9140483 9140484 9140485 9140486 9140487 9140488 9140489 9140490 9140491 9140492 9140493 9140494 9140495 9140496 9140497 9140498 9140499 9140500 9140501 9140502 9140503 9140504 9140520 9140521 9140522 9140544 9140545 9140546 9140547 9140548 9140549 9140550 9140551 9140552 9140553 9140554 9140555 9140556 9140557 9140559 9140560 9140561 9140562 9140563 9140564 9140565 9140566 9140567 9140568 9140569 9140570 9140571 9140572 9140573 9140574 9140575 9140576 9140577 9140578 9140579 9140580 9140581 9140582 9140584 9140585 9140586 9140587 9140588 9140589 9140590 9140591 9140592 9140593 9140594 9140595 9140596 9140597 9140598 9140599 9140600 9140601 9140602 9140603 9140604 9140605 9140606 9140607 9140608 9140609 9140610 9140611 9140612 9140613 9140614 9140615 9140616 9140617 9140618 9140619 9140620 9140621 9140622 9140623 9140624 9140625 9140626 9140627 9140628 9140629 9140630 9140632 9140633 9140634 9140636 9140637 9140638 9140639 9140640 9140641 9140642 9140643 9140644 9140645 9140646 9140647 9140648 9140649 9140650 9140651 9140652 9140653 9140654 9140655 9140656 9140657 9140658 9140659 9140660 9140661 9140662 9140663 9140664 9140665 9140666 9140667 9140668 9140669 9140670 9140671 9140672 9140673 9140674 9140675 9140676 9140677 9140678 9140679 9140680 9140681 9140682 9140683 9140684 9140685 9140686 9140687 9140688 9140689 9140690 9140691 9140692 9140693 9140694 9140695 9140696 9140697 9140698 9140699 9140700 9140701 9140702 9140703 9140704 9140705 9140706 9140707 9140708 9140709 9140710 9140711 9140712 9140713 9140714 9140715 9140716 9140717 9140718 9140719 9140720 9140721 9140722 9140723 9140724 9140725 9140726 9140727 9140728 9140729 9140730 9140731 9140732 9140733 9140734 9140735 9140736 9140737 9140738 9140739 9140740 9140741 9140742 9140743 9140744 9140745 9140746 9140747 9140748 9140749 9140750 9140751 9140752 9140753 9140754 9140755 9140756 9140757 9140758 9140759 9140760 9140761 9140762 9140763 9140767 9140768 9140769 9140770 9140771 9140772 9140773 9140774 9140775 9140776 9140777 9140778 9140779 9140780 9140781 9140782 9140783 9140784 9140785 9140786 9140787 9140788 9140789 9140793 9140794 9140795 9140796 9140797 9140798 9140799 9140800 9140802 9140803 9140804 9140808 9140809 9140810 9140811 9140812 9140813 9140814 9140815 9140816 9140817 9140818 9140819 9140820 9140821 9140822 9140823 9140824 9140825 9140826 9140827 9140829 9140830 9140833 9140835 9140839 9140841 9140842 9140845 9140846 9140848 9140849 9140853 9140854 9140855 9140857 9140858 9140859 9140860 9140861 9140862 9140863 9140864 9140865 9140867 9140868 9140869 9140870 9140871 9140875 9140876 9140877 9140878 9140879 9140880 9140881 9140885 9140886 9140887 9140888 9140889 9140890 9140891 9140892 9140893 9140894 9140895 9140896 9140897 9140898 9140899 9140900 9140901 9140902 9140903 9140904 9140905 9140906 9140907 9140908 9140909 9140910 9140911 9140912 9140913 9140914 9140915 9140916 9140917 9140918 9140919 9140920 9140921 9140922 9140923 9140924 9140925 9140926 9140927 9140928 9140929 9140930 9140931 9140932 9140933 9140934 9140935 9140936 9140937 9140938 9140939 9140940 9140941 9140942 9140943 9140944 9140945 9140946 9140947 9140948 9140949 9140950 9140951 9140952 9140953 9140954 9140955 9140956 9140957 9140958 9140959 9140960 9140961 9140962 9140963 9140964 9140965 9140966 9140967 9140968 9140969 9140970 9140971 9140972 9140973 9140974 9140975 9140976 9140977 9140978 9140979 9140980 9140981 9140982 9140983 9140984 9140985 9140986 9140987 9140992 9140993 9140994 9140995 9140997 9140998 9141002 9141005 9141006 9141007 9141008 9141009 9141010 9141011 9141013 9141015 9141016 9141017 9141018 9141020 9141021 9141022 9141023 9141024 9141025 9141027 9141028 9141029 9141030 9141031 9141034 9141037 9141038 9141040 9141041