s3 connection using rclone (Windows)
To connect to a bucket from the DigitalArchive with an s3 client you need
- old ECS: Endpoint: https://digitalarchiv.s3.fds.rwth-aachen.de:9021
- new HCP: Endpoint: https://digitalarchiv.hcp-carl.dasi.rwth-aachen.de:443
- Access Key
- Secret Key
- Bucket Name
You can view the endpoint (ECS or HCP) on the data project info page. You will get this from the admins of the corresponding data project in SeviRe to which your s3-bucket belongs.
- Download
Download the client software rclone: https://rclone.org/downloads/
Rclone is a single executable file (rclone or rclone.exe on Windows) that you can download as a zip archive and extract to a location of your choice.
- Installation and Configuration
Call the command line. To do this, either enter "cmd" in the Windows search bar and confirm with Enter or use the Windows key + "R" and then type "cmd" and confirm with Enter.
Now use the command line to call up the directory in which you have stored the rclone file.
Note:
- You change the directory with the command cd<path> for example: cd C:\FolderA\Folder1\rclone\.
- Note here that CMD does not support UNC paths (examples: "Downloads", "Documents",...). Therefore, please make sure that the downloaded rclone folder is located locally on the computer and the correct path is specified to it.
Run the rclone config command.
For example: C:\Program Files\rclone-v1.57.0-windows-amd64>rclone config
Now you've started the configuration of rclone. Create a remote access to the object storage by following the configuration steps.
Note: The commands may differ slightly depending on the rclone version, so pay less attention to the number you enter in the following steps and more attention to what you have to select.
# New remote
-> n
# Name (=RemoteName)
-> free selectable (Here it is advisable to choose a name that corresponds to the name chosen for your data project in Sevire)
# Storage type
-> (Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, SeaweedFS, and Tencent COS)
# Provider
-> any other S3 compatible provider
# credentials
-> 1 (Enter AWS credentials in the next step.)
# access_key_id
-> [Access Key] (zugehörig zu Ihrem Bucket auf SeviRe)
# secret_access_key
-> [Secret Access Key] (zugehörig zu Ihrem Bucket auf SeviRe)
# region
-> 2 (other-v4-signature)
# endpoint
-> https://digitalarchiv.s3.fds.rwth-aachen.de:9021
# location_constraint
-> Do not enter anything and press Enter
# ACL
-> 1 (Owner gets FULL_CONTROL.)
# advanced config
-> n (No)
# server_side_encryption
-> Do not enter anything and press Enter
# sse_kmskey_id
-> Do not enter anything and press Enter
# okay?
-> y (Yes, this is ok)
# quit
-> q
This completes the configuration and you have now created a connection to your bucket.
- Upload files
Starting point: From the command line, call the directory where you placed the software file and run the rclone copy command.
- Generic
rclone copy --s3-no-check-bucket [file(-path)] [Remotename]:[Bucketname]\
- Concrete
rclone copy --s3-no-check-bucket testfile MyDigiatlarchiv:ABC-1234567\
- With more spending
rclone copy --s3-no-check-bucket testfile MyDigiatlarchiv:ABC-1234567\ -v
- Multithread
Rclone offers the possibility of a multithread upload/download.
Note: When dividing, the size of the sections depends on the load and equipment of the server. In the following examples we have chosen a size of 4 MB. The specification of the --s3-upload-concurrency also depends on this.
- Generic
rclone copy --s3-no-check-bucket [file] [RemoteName]:[Bucketname] \ --multi-thread-cutoff [section size] --multi-thread-streams [size] --s3-upload-concurrency [size]
- Concrete
rclone copy --s3-no-check-bucket testfile MyDigiatlarchiv:ABC-1234567 \ --multi-thread-cutoff 4M --multi-thread-streams 4 --s3-upload-concurrency 4
- With more spending
rclone copy --s3-no-check-bucket testfile MyDigiatlarchiv:ABC-1234567 \ --multi-thread-cutoff 4M --multi-thread-streams 4 --s3-upload-concurrency 4 -v
- Download files
To download objects, first specify their bucket name and then the location where you want to save their objects.
- Generic
rclone copy [RemoteName]:[Bucketname]\[Object, you want to download] [storage location]\
- Concrete
rclone copy MyDigiatlarchiv:ABC-1234567\testfile C:\Downloads\
- With more spending
rclone copy MyDigiatlarchiv:ABC-1234567\testfile C:\Downloads\ -v
Note: Run rclone ls [RemoteName]:[Bucketname] to get a listing of the objects that are in your bucket.