Skip to main content

SSH and uploading Data via SFTP

Whilst there are a number of ways to get a terminal on the ondemand website (including using jupyterlab, which probably has the best terminal) sometime it is useful to directly ssh or use ssh from your desktop tools.

The information below will configure your machine's ssh configuration to be able to connect to the service.

Uploading data

For small files Ondemand, the remote desktop and Jupyter Lab all offer file transfer options in their toolbars.

When you have a large number of files or large file (greater than 1G - one gigabyte) you will need to use other methods

If your data is on your local machine or a drive attached to it then using an SFTP client, such as cyberduck, can copy files to and from our storage.

If your data is in another cloud service, such as OneDrive/Sharepoint;g-drive;dropbox then rclone can be used to sync the data or mount it to a server. Please note that tools that attempt to sync files such as the one drive client or dropbox can not be used as they are designed to work on a single computer rather than a cluster of servers.

SSH Access and SFTP clients​

We need two pieces of information to allow you to connect to the service

  • SSH Keypair and send us the public half
  • and, if you are not at the LSE campus, will will also need to know your public IP address which can be found at https://checkip.amazonaws.com/.
Application in development

We are developing an application to allow you to manage keys yourself and that will detect your public IP address. Until that is ready please send us your details.

If you already have an SSH Keypair please send us the public half of that. If not then follow the next section to generate one.

Open a terminal and use it to run the following commands.

ssh-keygen

When creating your key you can protect it with a passphrase (password) of your choice. You will either have to use that passphrase when you use the key or configure your machine to store and use your passphrase for you.

Once we have let you know that your access is in place you need to configure your computer to access the service.

We are going to add some lines of config to your $HOME/.ssh/config file. If you know how to do that already simply copy the config sections from below, otherwise follow these steps.

The lines of config you need can be found in the ondemand application SSH Config you may need to initialise the app the first time you use it.

We can now test that we have access. Running the command ssh fab should give you a terminal session on the login server (type exit to leave it). Configuring your SFTP client to connect to server fabfiles and port 22619 should show you your files and allow you to transfer files in both directions.

Users with lots of files to sync should investigate the rsync or rclone tools

rclone​

Rclone can be configure via a GUI tool however here we cover the command line setup to connect to and sync files from Microsoft OneDrive, which all LSE members have access to.

First you will need to install rclone on your desktop or use our remote desktop service. This set is needed to authorize your connection.

Firstly add rclone to your environment with module add rclone

running rclone config will walk you through an interactive setup process:

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> myonedrive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[snip]
XX / Microsoft OneDrive
   \ "onedrive"
[snip]
Storage> onedrive
Microsoft App Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Microsoft App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.
y) Yes
n) No
y/n> n
Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
	rclone authorize "onedrive"
Then paste the result.
Enter a value.

Now we run rclone authorize "onedrive" either on the remote desktop or your local machine. This will open a web browser and log you into Microsoft service. Switch back to your terminal and you will have a code to paste into rclone on Fabian.

Choose a number from below, or type in an existing value
 1 / OneDrive Personal or Business
   \ "onedrive"
 2 / Sharepoint site
   \ "sharepoint"
 3 / Type in driveID
   \ "driveid"
 4 / Type in SiteID
   \ "siteid"
 5 / Search a Sharepoint site
   \ "search"
config_type> 1

Option config_driveid.
Select drive you want to use
Choose a number from below, or type in your own string value.
Press Enter for the default (b!5ijNWH_a-series-of-letters-and-digits_R3jVAnuc).
 1 / OneDrive (business)
   \ (b!5ijNWH_a-series-of-letters-and-digits_R3jVAnuc)
config_driveid> 1

Drive OK?

Found drive "root" of type "business"
URL: https://lsecloud-my.sharepoint.com/personal/email_address_lse_ac_uk/Documents

y) Yes (default)
n) No
y/n>
--------------------
[remote]
type = onedrive
token = {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"}
drive_id = b!a-series-of-letters-and-digits
drive_type = business
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Keep this "myonedrive" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote

y/e/d>

Current remotes:

Name                 Type
====                 ====
myonedrive           onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Once configured you can then use rclone like this,

List directories/folders in top level of your OneDrive

rclone lsd myonedrive:

List all the files in your OneDrive

rclone ls myonedrive:

To copy a local directory to an OneDrive directory called backup

rclone copy source myonedrive:backup

To sync a local directory called code with an OneDrive directory called code

rclone sync code myonedrive:code

You can use the same approach with Sharepoint Site including those that hold the files of a teams site.

Documentation for other cloud storage providers is available on the rclone website.

If the version of rclone you are using has changed please follow the official documentation.

Advanced configuration​

Configure your machine to store and use your passphrase​

You can configure your mac to store your ssh key's passphrase in our Keychain

add the following section to your .ssh/config file

Host *
  AddKeysToAgent yes
  UseKeychain yes

run ssh-add -K once with your passphrase