This article describes the main difference between the TSUBAME2.5 and TSUBAME3.0. Please refer to "TSUBAME 3.0 User's Guide" and "TSUBAME portal User's Guide" for details.
What you can do with the login node
When you logged in to TSUBAME 2.5, the terminal was connected to the interactive node, which is the same constituent node as the compute node, and it was able to compile and debug the application on it.
In TSUBAME 3.0, the login node is a different configuration from the compute node (e.g. GPU is not connected), so it is not assumed to execute the application including debugging.
Although it is not a problem to transfer and deploy files and compile small programs at the login node, avoid using loads on login nodes such as debugging and running large-scale programs. Please correspond by using computing nodes interactively by the method described in the next section.
How to use interactive use
In order to use interactive use connecting directly to a compute node and inputting commands, please use the following command:
$ qrsh –g [TSUBAME3 group] -l [resource type]=[number] -l h_rt=[elapsed time]
Once the compute node is secured, the shell in the state logged in to the compute node will be displayed. When you exit this shell, the interactive usage is terminated and the compute nodes are released.
If -g is omitted, the TSUBAME points are not consumed, but the restriction such as execution time (within 10 minutes) as "trial execution" is applied.
When using the GUI application, it can not be executed from the shell executed by qrsh. Therefore, after starting interactive use of the resource type f_node in the above procedure, you can use it by connecting ssh -Y from the login node to the compute node at another terminal which is different from the one where qrsh is running. Please be aware that you need to specify the -Y option both of to the login node from the terminal, and ssh from the login node to the compute node.
How to run applications (setting of PATH etc.)
In TSUBAME 2.5, most applications were executable at the time of logging in. executing specified environment setting scripts when switching some applications, versions, or MPI environment respectively I switched the environment. When changing version, changing MPI environment, in some applications, the environment was switched by executing the specified environment setting script.
when logged in to TSUBAME 3.0, environment variables for most applications are not set. It can be executed by explicitly loading the module file corresponding to the application to be used.
For example, when using the Intel compiler, CUDA, OpenMPI, it must be set before compiling the application and executing the job as described below:
$ module load intel cuda openmpi
Job restrictions
Please check "Various limit value list" about the current limit.
In TSUBAME 2.5, instead of paying more TSUBAME points, it was permitted to run jobs longer than 24 hours as a premier option. The option to extend execution time has been abolished in TSUBAME 3.0. The maximum execution time for all jobs is 24 hours.
It was able to perform large reservation execution of 16 nodes or more by using H queue on a daily basis.
In In TSUBAME 3.0, it is currently in preraration so that reservation execution of 1 node or more, 1 hour unit can be performed.