Read-only storage for Machine Learning

We offer databases for machine learning etc. with dedicated SSD server connected to TSUBAME.

(2023.3.2) We temporally suspended this feature due to operational reasons. The service will be back in April.

Note: This is an experimental service and may change or be discontinued without prior notice.

Notes

  • There are some differences from data on Lustre parallel filesystem (/gs/hs*/) and the performance might be worse than Lustre in some conditions.
    • Pros: Data is stored on RAID-0 SSD (/gs/hs*/ is RAID-6 HDD)
    • Cons: SSD, server, and network are not parallelized and the performance will be reduced under contention.
  • Users cannot write into the storage. If you want something to be hosted, please refer to the section at the bottom of this page.
  • Longer downtime is expected when the SSD fails.

Available databases

  • Alphafold2 database
    • /gs/ss0/alphafold/2.1.1/data/ $ALPHAFOLD_DATA_DIR
    • Set this path to ALPHAFOLD_DATA_DIR environment variable, after invoking module load alphafold
  • ILSVRC2012 dataset(also known as ImageNet): Academic use only
    • /gs/academic/ss0/ILSVRC2012/ /gs/hs0/GSIC/ILSVRC2012/
    • Please register at ImageNet official site before use.
    • Check the directory content to know how the data is organized.

We restrict access to the data marked as "Academic use only" to users in Tokyo Tech for license reasons. If an academic user outside of Tokyo Tech want to access the contents, please send an inquiry。

Request for serving new databases

If you want some databases to be served, please send an inquiry to us.
Please note that not all requests will be satisfied for various reasons.

  • The database must be public and widely used.
    • The database which is used by only one research group will likely be rejected.
    • If the complicated terms of use, unlike CC-BY-SA, are applied, we cannot put the data in public storage.
  • The database size must be suitable to be served with dedicated SSD.
    • A database smaller than 1GB can easily fit into home directory or group disk.
    • A database which does not fit into SSD (15TB with RAID-0) cannot be served in this storage.