Introduction to SSH & Data Transfer
Attention
Note that all port numbers, IP addresses, and SSH keys included in this chapter are placeholders and do not refer to known systems. They are used purely for illustrative purposes, and users should modify the commands to correspond to their actual systems.
A Secure SHell (SSH) tunnel creates an encrypted connection between two computer systems. This secure connection allows users to access and use a remote system via the command line on their local machine. SSH connections can also be used to transfer data securely between two systems. Many HPC platforms, including NOAA Level 1 systems, are accessed via SSH from the user’s own computer.
Attention
Note that the instructions on this page assume that users are working on a UNIX-like system (i.e., Linux or MacOS). They may not work as-is on Windows systems, but users can adapt them for Windows or use a tool such as Cygwin, which enables the use of UNIX-like commands on Windows. Users may also consider installing a virtual machine such as VirtualBox.
Creating an SSH Tunnel
Create an SSH Key
To generate an SSH key, open a terminal window and run:
ssh-keygen -t rsa
Hit enter three times to accept defaults, or if customization is desired:
Enter the file in which to save the key (for example:
~/.ssh/id_rsa
)Enter passphrase (empty for no passphrase)
Enter same passphrase again
To see the SSH public key contents, run:
cat id_rsa.pub
SSH Into a Remote Machine
This process differs somewhat from system to system. However, this section provides general guidance.
Create/Edit an SSH Configuration File (~/.ssh/config
)
If it exists, the SSH config
file is located in the hidden .ssh
directory. If it does not exist, opening it will create the file. In a terminal window, run:
vi ~/.ssh/config
Press i
to edit the file, and add an entry in the following format:
Host <name_of_your_choice>
Hostname <host_name_or_IP_address>
User <Username>
IdentityFile ~/.ssh/<key_name>
When finished, hit the esc
key and type :wq
to write the data to the file and quit the file editor.
Note
The IdentityFile
line is not required unless the user has multiple SSH keys. However, there is no harm in adding it.
Concretely, a user logging into an AWS cluster might enter something similar to the following.
Host aws
Hostname 50.60.700.80
User Jane.Doe
IdentityFile ~/.ssh/id_rsa
Users attempting to authenticate via SSH on GitHub might create the following code block instead:
Host github
Hostname github.com
User git
IdentityFile ~/.ssh/id_ed25519
SSH Into the Remote System
To SSH into the remote system, simply run:
ssh <Host>
where <Host>
is the “name_of_your_choice” that was added to the config
file. For example, a user logging into the AWS cluster above would type:
ssh aws
This will create an SSH tunnel between the user’s local system and the AWS cluster. The user will be able to work on the AWS cluster by running commands in the terminal window.
In some cases, the user may be asked if they want to connect:
The authenticity of host '50.60.700.80 (50.60.700.80)' can't be established.
ECDSA key fingerprint is SHA256:a0ABbC4cdeDEfFghi+j3kGHlO5mnIJKLMop7NOqPrQR.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Enter yes
to continue connecting. The user is responsible for verifying that they are connecting to the correct system.
Data Transfer via SSH
Introduction
Users who are working on a remote cloud or HPC system may want to copy files (e.g., graphics plots) to or from their local system. Users can run the scp
command in a new terminal/command prompt window to securely copy these files from their remote system to their local system or vice versa. The structure of the command is:
scp [OPTION] [user@]SRC_HOST:]file1 [user@]DEST_HOST:]file2
Here, SRC_HOST
refers to the system where the files are currently located. DEST_HOST
refers to the system that the files will be copied to. file1
is the path to the file or directory to copy, and file2
is the location that the file or directory should be copied to on the DEST_HOST
system.
Download the Data from a Remote System to a Local System
Note
Users should transfer data to or from non-Level 1 platforms using the recommended approach for that platform. This section outlines some basic guidance, but users may need to supplement with research of their own. On Level 1 systems, users may find it helpful to refer to the RDHPCS CommonDocs Wiki.
To download data using scp
, users can typically adjust one of the following commands for use on their system:
scp username@your-IP-address:/path/to/file_or_directory_1 /path/to/file_or_directory_2
# OR
scp -P 12345 username@localhost:/path/to/file_or_directory_1 path/to/file_or_directory_2
To copy an entire directory, use scp -r
instead of scp
.
Users who know the IP address of their remote system can use the first command. For example:
scp Jane.Doe@10.20.300.40:/contrib/Jane.Doe/expt_dirs/test_community/2019061518/postprd/*.png /Users/janedoe/plots
This command will copy all files ending in .png
from the remote test_community/2019061518/postprd/
experiment subdirectory into Jane Doe’s local plots
directory.
Users who know their localhost
port number should use the second command and, if requested, enter the password to the remote system. For example:
scp -P 3355 Jane.Doe@localhost:/lustre/Jane.Doe/expt_dirs/test_community/2019061518/postprd/*.png .
This command will copy all files ending in .png
from the test_community/2019061518/postprd/
experiment subdirectory on a remote HPC system into Jane Doe’s present working directory (.
).
Attention
Note that all port numbers, IP addresses, and SSH keys included in this chapter are placeholders and do not refer to known systems. They are used purely for illustrative purposes, and users should modify the commands to correspond to their actual systems.