Last update: January 10, 2022
The Extreme Science and Engineering Discovery Environment (XSEDE) is a virtual collaboration funded by the National Science Foundation that facilitates free, customized access to advanced digital resources, consulting, training, and mentorship. XSEDE helps the nation's most creative minds discover breakthroughs and solutions for some of the world's greatest scientific challenges.
XSEDE's virtual cyberinfrastructure allows scientists to interactively share computing resources, data, and expertise. XSEDE resources may be broadly categorized as follows: High Performance Computing, High Throughput Computing, Visualization, Storage, and Data Services. Many resources provide overlapping functionality across categories.
Take a look at this brief introductory video "What is XSEDE?".
This recently recorded webinar presentation provides an overview of XSEDE and it's services and capabilities. The second half of the presentation details each of XSEDE's compute resources.
|XSEDE New User Training |
Recorded: September 20, 2020
Run time: 2 hours 1 minute
Your Campus Champion is your local source of knowledge about high-performance computing opportunities and resources. Your Campus Champion can direct researchers and educators to national high-performance computing resources and services that are available to you and that will help you to advance scientific discovery.
|Get to know your campus representative! |
Search the campus champions directory to find your local Campus Champion
Your Campus Champions representative will:
- Provide you with information about high-performance computing resources to researchers and educators, including resources available from XSEDE
- Assist you to quickly get access to allocations of high-performance computing resources
- Facilitate awareness and training workshops about the use of high-performance computing resources and services
- Provide you with contacts within the high-performance computing community for quick problem resolution
- Provide you with timely information on new resources, services, and offerings
In order to get access to XSEDE resources you must create an XSEDE User Portal (XUP) account. The XSEDE User Portal (XUP) is a web-based, single point-of-entry to the XSEDE system providing:
- Allocation request submission capability
- Access to all your XSEDE accounts and allocated resources
- Up-to-date information on your accounts and and allocation usage
- Interfaces for data transfers, data collections, and other user tasks and resources
- Subscriptions to user news & announcements
- Training class registration and schedule
- Access to the XSEDE Help Desk and User Forums
See the CyberInfrastructure Tutor's video "Using the XSEDE User Portal" for a tour of capabilities.
You are responsible for your account and for protecting your passwords.
- Do not share passwords, do not write passwords down where they can be easily found, and do not use tools that expose passwords on the network. This includes private keys: make sure they are password-protected.
- Close SSH terminals and log out of the XSEDE User Portal when finished
- Report Suspicious Activity. If you have any suspicion that your account or personal computer has been compromised, email firstname.lastname@example.org or call, 24/7, 1-866-907-2383 immediately.
After creating your XSEDE User Portal (XUP) account and logging in to accept the User Responsibilities, you have the choice of logging in with your XUP credentials, or you may use other login credentials such as your university credentials. Click on the blue "Other Sign In Options" button to link your XSEDE account with other identity providers.
It is important that you manage your XSEDE User Profile and publicatons to keep your contact information up to date and account information accurate. This can all be done by managing your user profile in XSEDE. Initially your profile information will include the information you supplied when creating your account.
To view and manage your profile please visit My XSEDE -> Profile or click on the "Edit Profile" link under the avatar.
Your profile information includes basic information such as email address, organization, mailing address and more. If you would like to share more information about yourself you also have the option to include information such as twitter username, facebook URL, LinkedIn URL, biography and interests. You also have the option to upload an image of yourself. You can also set privacy settings on your personal information. To do this click edit profile and proceed to the privacy setting tab and click save.
Users can also view their Distinguished Names (DNs) and add/remove new DNs. By default each user will receive three DNs associated with their accounts (NCSA, PSC, NICS). Additional DNs can be added as long as they are from the following providers:
/C=SE/O=AddTrust AB/* /DC=ch/DC=cern/* /C=ch/O=CERN/* /C=GB/ST=Greater Manchester/* /C=DE/O=DFN-Verein/* /C=US/O=DigiCert Inc/* /DC=com/DC=DigiCert-Grid/* /C=US/O=DigiCert Grid/* /C=IT/O=INFN/CN=INFN CA* /C=US/O=Internet2/OU=InCommon/* /C=US/O=National Center for Supercomputing Applications/* /DC=net/DC=ES/* /C=US/O=Pittsburgh Supercomputing Center/* /C=UK/O=eScienceCA/* /C=UK/O=eScienceRoot/* /DC=org/DC=cilogon/* /DC=es/DC=irisgrid/*
To add a DN please visit your profile and click "Manage DN" button which will bring up your existing registered DNs and the ability to add a DN subject. Paste your Certificate Subject into the text box, then click the "Add DN" button. This will register your certificate with XSEDE. (Note: It may take up to one business day for XSEDE service providers to process your registration.)
To remove a DN please click on the trash icon next to the DN you wish to remove. You cannot remove the default DNs associated with your account.
Users can add publications to their XSEDE profile and associate them with their XSEDE projects. Users can also view all publications entered into the publication service.
This will greatly simplify keeping user profiles up to date, and will make it easier when applying for renewal allocations (which require such information from prior allocations). By having these publications readily available in user profiles, it may also stimulate collaborations among current XSEDE users. This helps users keep track of the impact XSEDE services are having on their science by keeping an up to date their list of publications. You can also tag a publication with an SP site, specific resource and other general tags such as training and ECSS.
The XUP now also includes publication-discovery feature enabling users to easily add publications in the NSF publication database to their XUP Profile. This feature will greatly simplify keeping user profiles up to date and will make it easier for PIs when applying for renewal allocations, which require publication information. If you have publications in the NSF database A "New Publications Found" flag will appear above the profile publication section. Users will be presented with their list of publications and will have the option of:
- Adding the publication to their profile
- Declining to add the publication to their profile
- Skipping the publication if it's already included in their profile
Compute time on XSEDE resources is granted via the allocations process. You may apply for a new allocation or get added to an existing allocation. Please consult the detailed information in the XSEDE User Portal in the Allocations section.
A user may apply for one of the following allocation types:
Trial: Some resources offer trial allocations.
Startup: The fastest way to get started on XSEDE, Startup allocations require minimum documentation, are reviewed all year long, and are valid for one year.
Multi-year Startup Allocations: Startup allocations up to three years may be available.
Education: Also lasting one year, an Education allocation provides time for academic or training classes.
Research: Research allocation requests are reviewed quarterly and require more formal documentation. Research allocations will be granted for one year and may be renewed or extended.
In addition to requesting a new allocation, a Principal Investigator (PI) may add users to an existing allocation through the portal. Click "Add User" under the "My XSEDE" tab. Once your Startup or Education allocation is submitted, you should hear back from the Allocations committee within two weeks.
This recently recorded presentation details all the steps and requirements needed for successful research allocation requests. Topics covered include Computation Methodologies, Justification for SUs requested and many others.
|Writing and Submitting a Successful XSEDE Proposal |
Recorded: December 15, 2021
Run time: 1 hour 21 minutes
Once an allocation has been granted, user accounts are usually created within 24 hours on the allocated resource. Users may login to their accounts in one of two ways: The XSEDE Single Sign On Hub allows access to all your allocated resources using just your XUP username and password created in Step 2. while direct access via SSH requires a resource-specific password. Both methods require an SSH client. Linux and Unix users may use the built-in Unix
ssh command. Windows users may download one of many SSH clients, free ones include "PuTTY".
XSEDE's Single Sign On Login Hub is the easiest way to access all your accounts. The hub provides a single point of entry to all your allocated XSEDE resources. Login to the hub at login.xsede.org with your XUP username and password, using your favorite SSH client. Once logged in to the hub, use
gsissh to login to XSEDE resources. Globus' GSI-OpenSSH protocol implementation,
gsissh, is a modified version of OpenSSH adding support for Grid Security Infrastructure authentication and credential forwarding.
Please read more and see sample sessions on the XSEDE Single Sign On Login Hub page.
Users may also bypass the hub and login directly to XSEDE resources via SSH. This will require a resource-specific password. Look up your local username in the XUP under My XSEDE->Accounts then submit a Help Desk Ticket requesting a local password for the specific resource provider.
Once you have your resource-specific password,
ssh directly to a resource like so:
mylaptop$ ssh -l username machinenamee.g.
mylaptop$ ssh -l slindsey stampede2.tacc.utexas.edu
Once your accounts are set up, you'll probably want to start transferring your project files to your allocated resource. XSEDE offers several easy, reliable file transfer mechanisms briefly detailed below. For more in depth information please consult the XSEDE Data Transfers & Management page.
Globus is a web client that implements the GridFTP protocol, providing reliable data movement service between resources. Researchers with no IT background can easily move large quantities of files, or move files of large size, using the Globus clients, and developers who want to automate workflows can use the Globus command-line interface (GO CLI).
Here's a sample transfer from PSC's Bridges to TACC's Stampede2 optimized for large files:
globus-url-copy is a command-line implementations of the GridFTP protocol which underlies all XSEDE transfer mechanisms. Use these commands to transfer large files.
login1$ globus-url-copy -stripe -tcp-bs 8388608 \ gsiftp:// gridftp.bridges.psc.xsede.org/scratcha/joeuser/file \ gsiftp:// gridftp.stampede2.tacc.xsede.org/scratch/joeuser
Visit the Data Transfers & Management page to get the GridFTP endpoints for each site.
You may also use the
sftp command-line utilities to transfer files between XSEDE resources and/or your local machine. From Linux or Mac, you can run these commands directly from the terminal. From Windows, use your ssh client ("PuTTY" is freely available). Both
sftp are easy to use and secure.
A computing environment refers to the components of a compute resource such as the operating system, basic hardware infrastructure, and software characteristics.
Where your data resides on XSEDE and the appropriate storage is your responsibility. In general, all resources provide the following file systems:
- HOME: Permanent space, but small. A good choice for building software and working file collections of small to medium sized files, where a medium sized file is less than 50 MB.
- SCRATCH: More space, but temporary; use while you are running your jobs. Scratch space is temporary; it is not backed up and has limited redundancy, and is periodically purged of old files!
- Archival Storage: Long term storage of large amounts of data (often tape); slower access, accessible from all sites.
For more detailed information on file systems and storage please consult the user guide and the XSEDE Storage page.
The environment management package, Modules, is a command-line interface providing commands for setting and modifying shell environment variables. These environment variables define values used by the shell (e.g., bash, tcsh,…) and by programs that you execute on the shell command line. For example, the PATH environment variable commonly defines a list of directories that the shell should search to find an executable program that you have referred to on the command line - this allows you to execute that program without having to type out the entire directory path to the executable file for that program. Modules makes it easier for you to manage collections of environment variables associated with various software packages, and to automatically modify environment variables as needed when switching between software packages.
module avail- lists available modules
module list- lists currently loaded modules
module help foo- help on module
module whatis foo- brief description of module
module display foo- displays the changes that are made to the environment by loading module foo without actually loading it.
module load foo- load module
module unload foo- unloads module
fooand removes all changes that it made in the environment
All XSEDE resources employ a job scheduler (sometimes called "batch" scheduler), software that monitors and runs programs in the background, ideally achieving optimum utilization of that resource, including processors, memory, and disk space. All schedulers basically work the same way:
- specify number/type of nodes you need
- specify how long you need to run
- specify where your output files go
- submit the job script containing these specifications
Check each resource's user guide for exact information needed such as production queues, node types, modules and job launch mechanisms. A typical job script will look similar to the following:
For the most part, XSEDE systems use the Simple Linux Utility for Resource Management (Slurm) job scheduler. Each resource will have one or more queues, each with a pool of jobs waiting to run. A user submits jobs to a queue via job scheduler commands. Some common Slurm commands are listed below.
sbatch myjobscript- submit the file ‘myjobscript' to start a job
squeue -u username- display the status of all your jobs
scancel jobid- delete the job from the queue using the jobid
#SBATCH -J myjob # Job name #SBATCH -o myjob.%j.out # define stdout filename; %j expands to jobid #SBATCH -e myjob.%j.err # define stderr filename; skip to combine stdout and stderr #SBATCH --email@example.com #SBATCH --mail-type=ALL #SBATCH -A TG-01234 # specify the project or allocation number #SBATCH -p normal # specify queue #SBATCH -N 2 # Number of nodes, not cores (16 cores/node) #SBATCH -n 32 # Total number of MPI tasks (if omitted, n=N) #SBATCH -t 00:30:00 # set maximum run time of 30 minutes module load fftw3 # Load any necessary modules (these are examples) mpirun ./myprogram # TACC systems use "ibrun", not "mpirun"
We realize this is a lot to take in, so XSEDE has lots of help options. You might begin by searching for your XSEDE Campus Champion. Your local champion can direct to the staff or resources you need. This robust community of research computing professionals is spread out over 300 member institutions.
Still not sure if XSEDE is right for you? The XSEDE Help Desk staff will answer your question or put you in touch with the experts who can. XSEDE has numerous ways to search for information and get help on a variety of topics. Our staff can help with problems such as choosing a platform or resource, choosing a software package, improving file transfer performance, and help with visualization. Send your questions to firstname.lastname@example.org.
Additional help resources:
User News - keep up to date with what is happening with XSEDE and the services you use. XSEDE staff regularly sends announcements, machine status notices, notifications of upcoming training classes and more. Please login to XSEDE User Portal to manage your subscriptions here.
User Forums - ask questions and interact with other users on the user forums available within the XSEDE User portal.
Extended Collaborative Support (ECSS) is available to all XSEDE users. You may request XSEDE's advanced ECSS support at any time though the Allocations tab in the XSEDE User Portal. ECSS requires written justification and a project plan. Our highly experienced staff can help with:
- porting applications to new resources
- providing help for portal and gateway development
- implementing algorithmic enhancements
- implementing parallel math libraries
- improving scalability of codes to higher processor counts
- optimizing codes to efficiently utilize specific resources
- assisting with visualization, workflow, data analysis, and data transfer
- performance analysis
Need additional info? Visit XSEDE Extended Collaborative Support.
XSEDE also provides a mobile interface to access XSEDE services and information from the convenience of your mobile device. The XSEDE User Portal mobile interface offers both public content and select user information and service via a new and improved XSEDE Mobile Web site.
- Browse all XSEDE systems
- View and subscribe to user news
- View and register for training courses
- View your allocations and account information
- Monitor XSEDE resources and job status
- Receive consulting support by viewing and submitting helpdesk tickets
Check out XSEDE Mobile User Portal from the convenience of your mobile device anytime and anywhere.