It can be daunting when you’re just starting out using the Linux command line to access the University’s HPC resources. This post is intended to be a collection of helpful tutorials that will give you the essentials you need to get your analysis up and running on a Linux-based computing cluster.
Pleast yeet me an email if I’ve forgotten anything or if you notice a dead link so I can update this page.
Connecting to an external server
This section applies if you are connecting to a Linux server, rather than directly working on a Linux-based PC. There are two types of connections that you will need to establish.
- SSH: will let you pull up a command line terminal on the server
- FTP: will let you transfer files between the server and your computer
MobaXterm will handle both of these connections for you and is your best choice.
The Documentation is thorough, but the live demo gives a good overview of the features https://mobaxterm.mobatek.net/demo.html
Linux command line
Ubuntu has a nice tutorial which walks through the basics of navigating the filesystem and basic operations: https://ubuntu.com/tutorials/command-line-for-beginners#1-overview
There are plenty of tutorials available; here are a few:
https://ryanstutorials.net/linuxtutorial/ (Covers most of the important stuff in excellent detail)
https://www.educative.io/blog/bash-shell-command-cheat-sheet (Handy quick reference guide for some common commands)
https://www.guru99.com/linux-commands-cheat-sheet.html (A much bigger cheat sheet)
There are many useful programs for performing a plethora of functions, and when they are strung together in ‘pipes’ allow you to do some pretty powerful processing of your files. Here is one excellent introduction to most of these commands: https://dev.to/awwsmm/101-bash-commands-and-tips-for-beginners-to-experts-30je
You wont have the necessary permissions to install software in the system directories. Instead, you should aim to use conda as much as possible.
There are many benefits to using tmux but the biggest is that it allows you to shutdown your computer without having to end your session on the server.
If it’s not already installed and you’re using conda, then you can install it like so:
conda install tmux
The wiki page has everything you need to know https://github.com/tmux/tmux/wiki/Getting-Started but a couple of commands are all you need to get started:
tmuxwill start a new session
- Ctrl + b, then d detatch from the session
tmux lswill list the current sessions (by default they’re numbered 0-9)
tmux a -t 0will attach to session number 0
exitwill close the session you’re in
DeepThought uses slurm. Most of the time you will be given a template submission script by the HPC’s sysadmin and you’ll just modify that whenever you want to submit something. Rob has covered the basics here:, and a more thorough explanation of the tools is covered in the Slurm quickstart guide, and the tutorial series.
If your HPC uses the Sun Grid Engine (SGE) instead of slurm you’ll want something like this: https://www.uibk.ac.at/zid/systeme/hpc-systeme/common/tutorials/sge-howto.html