Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Here are a couple of ways of accessing S3 buckets in the SCITAS clusters.

Table of Contents

s3fs

Allows you to use an S3 bucket as a Filesystem. It is installed on the cluster login nodes (must be mounted before running the jobs on the login node).

...

Code Block
themeMidnight
fusermount -u ${HOME}/mybucket

boto (python)

Install the library

Code Block
themeMidnight
module load gcc python
pip3 install --user boto

...

Code Block
themeMidnight
chmod u+x s3.py
./s3.py
test	6	2021-06-23T08:29:48.409Z 

rclone

Edit ~/.rclone.conf

Code Block
themeMidnight
[private]
type = s3
access_key_id = put_your_access_key_here
secret_access_key = put_your_secret_key_here
region = other-v2-signature
endpoint = https://s3.epfl.ch/

...