High Performance Computing (HPC) Systems
The Summit system architecture is comprised of general compute nodes, GPU compute nodes, high-memory compute nodes and Phi nodes. For detailed information on the system, see the CU Boulder Research Computing resources page.
Allocations of time on Summit are based on Service Units (SU) and CPU core-hours as follows:
1 SU = 1 CPU core-hour (i.e. a single fully utilized CPU core on 1 compute node for 1 hour)
CSU users as a whole have an allocation of roughly 25% of the total SUs/yr for Summit (21,996,360 SU/yr)
All new accounts will receive an Initial allocation of 50,000 SU/yr. The Initial allocation will expire after 1 year.
If users require more time beyond the Initial allocation, they can apply for a Project allocation.
To apply for a project allocation, submit the Summit Project Allocation Request Form
The Summit Management and Allocations Review Committee will review and assess Project applications and respond within 1 week on the status of the request.
The Summit system offered an optional Condominium Computing Model (“Condo Model”). In the Condo Model, costs are split between researchers and Central IT. Researchers purchase their own compute nodes and Central IT provides the hosting environment and support services for those nodes.
Due to the age of the system, the final opportunity to buy into the Condo Model on Summit concluded in January 2018. We expect to offer Condo Model buy in opportunities on future HPC systems.
Condo jobs have the following privileges
- can request longer run times (up to 168 hrs. (7 D))
- get queue priority boost (equal to 1 D boost)
- can access all compute nodes
To properly activate Condo shares, Condo users should send the following info to email@example.com.
- full name
- condo group ID (see table below)
You will receive an email note when your condo group ID assignment is complete. You’ll then be able to submit jobs using your Condo allocation.
The table below shows the condo group ID that has been assigned to each principal investigator and their department affiliation.
|PI||Dept.||Condo group ID|
|Asa Ben-Hur||Computer Science||hal|
|Stephen Guzik||Mechanical Engineering||cfd|
|Chris Weinberger||Mechanical Engineering||crw|
Condo Job Submission
To submit jobs using your Condo allocation, include the following lines in your Slurm batch job file
#SBATCH --qos condo #SBATCH -A csu-summit-xxx
where “xxx” is your condo group ID from the table above. Note the double-dash for the “qos”parameter. When you submit a Slurm batch job file with these parameters, the job will run with the additional privileges described above.
The following statement may be used for Summit Acknowledgements:
“This work utilized the RMACC Summit supercomputer, which is supported by the National Science Foundation (awards ACI-1532235 and ACI-1532236), the University of Colorado Boulder and Colorado State University. The RMACC Summit supercomputer is a joint effort of the University of Colorado Boulder and Colorado State University.”
The following document includes information that may be used for grants, RFP’s and other solicitations:
CSU User's Guide
Logging into Summit requires DUO two factor authentication. To set up DUO, follow the instructions at http://authenticate.colostate.edu
Access to Research Computing resources is available by way of the Secure Shell, or ssh, protocol. Access is provided via a dedicated login node.
ssh command can be run from the Linux and OS X command-line.
where you should replace csu_eID with your eid.
When logging in from Windows, we recommend the PuTTY application.
If you are using the DUO smartphone app: When it asks for your password, type in your CSU password followed by a comma followed by the word “push”.
Don’t forget the comma! The DUO app on your phone will ask you to approve the request.
Alternatively: you can ask the DUO app to generate a 6-digit code called a DUO key. Use your CSU password, followed by a comma, followed by the 6 digit number generated by your app.
Don’t forget the comma!
NOTE: The DUO_key mentioned above cycles every 15 seconds. If you do not log in within 15 seconds of generating the key, it will expire and you’ll have to generate another key.
If you are having trouble logging in and you suspect DUO is the issue, contact the Central IT Support Helpdesk for DUO support
Transferring files to Summit is typically facilitated by sftp or Globus.
ssh File transfer
ssh File transfer (sftp) is recommended for smaller files. A good rule of thumb is that if you’re willing to sit through the file transfer, sftp is a good option. You can use sftp from the command line or other software. The login credentials are the same ones you use to ssh into the login node. Please see the section on remote login for more details.
Other sftp clients
You can also use other file transfer software like
- FileZilla, a multi-protocol, multi-platform file-transfer application.
- WinSCP, a basic SCP/SFTP file-transfer application for Windows
Generally, the information you need to use this method is as follows:
- Host: login.rc.colorado.edu
- Protocol: choose “SFTP – SSH File Transfer Protocol”
- User and password: the same credentials that you use to ssh into Summit (see remote login section)
Note: Because Research Computing uses one-time passwords for authentication, you must disable password retention / saving in your file-transfer client if you are using the DUO key authentication method. Failure to do so may cause your account to be temporarily disabled after the client attempts and fails to authenticate repeatedly in the background.
Globus software addresses deficiencies in secure copy requests by automating large data transfers, by resuming failed transfers, and by simplifying the implementation of high-performance transfers between computing centers. For more information about using Globus, see the documentation provided by CU Boulder research computing.
Summit users have 3 main directories: Home, Projects and Scratch.
The home directory (/home/csu_eID@colostate.edu) has 2 GB of storage that is backed up locally into a hidden directory (.snapshot/) at 2 hour, daily, and weekly intervals and to a second site for disaster recovery nightly. THe home directory is not on high-performance storage, so you should not be used to be written to from compute jobs.
The projects directory (/projects/csu_eID@colostate.edu) is intended to store software builds and smaller datasets and to share data and software with other users. It has 250 GB of storage that is backed up locally into a hidden directory (.snapshot/) at 6 hour, daily, and weekly intervals and to a second site for disaster recovery nightly. Like the home directory, projects is not on high-performance storage, so you should not be used to be written to from compute jobs.
Summit Scratch directory
The Summit scratch directory (/scratch/summit/csu_eID@colostate.edu) is intended for output from compute jobs running on Summit usinguses GPFS (General Parallel File System) for fast parallel I/O. Each user is limited to 10 TB of storage and a total of 20 million files and directories. If you need a larger allocation, email firstname.lastname@example.org. Files in the scratch directory are purged after 90 days and are NOT backed up and could be purged at any time. Transfer your data to the projects directory or to permanent data storage soon after your job completes.
For more in depth information, see the CU Boulder RC User’s Guide topic on Storage and Filesystems.
Summit uses a module system that allows the installation of multiple versions of common software packages that users can switch between. To use the software, the module must be loaded first. Loading a module will alter aspects of your environment, such as the $PATH variable.
Summit uses a module system to publish software. For a list of all software currently installed on Summit, see the CU Boulder RC User Guide. Note that Commercial application licenses only cover CU Boulder users. However, we are working on applying CSU’s MatLab site license to Summit soon.
Installing custom software
Users should install custom software in the project directory. Users may also take advantage of the module system to publish local module files to configure a running environment for the software. These modules could be adopted as a centrally supported module if it has wide community use.
For information on how to compile your software on Summit, please see the CU Boulder RC Summit User Guide.
For more information about using Summit, see the following Resources:
- Examples and tutorials from CU Boulder Research computing User Guide
- CU Boulder Research Computing Tutorial Wiki
- Parallelization workshop (R, Python, Matlab)
Note that the CU examples will include information about how they log in to Summit. Please substitute the information from our Remote login section.
Help & Additional Information:
Submit Summit support requests to email@example.com
Submit Cray support requests to firstname.lastname@example.org
To receive Summit system updates and other announcements, send a message to email@example.com
To receive information on training and events associated with Summit, email firstname.lastname@example.org.
To receive information on Cray system status, email email@example.com.