Data transferring

Transferring Data to B2DROP¶

⚠️ If you plan to integrate file transfers to B2DROP within automated workflows, jobs, or pipelines, please inform the BSC Data Management team in advance. Provide details about the expected data volume, frequency, and timing of transfers. This helps ensure optimal performance and prevents service degradation for other users.

Method 1: Accessing Files via WebDAV¶

Another way to access files is through the WebDAV protocol, which allows you to interact with remote files as if they were local files. You can access B2DROP files using WebDAV, and this method is supported by various file managers and applications.

For more information on accessing files via WebDAV in Nextcloud, refer to the official documentation:
📄 Accessing Nextcloud Files Using WebDAV

Method 2: Uploading Large Files (Chunked Upload)¶

If you experience problems uploading large files via the B2DROP web interface, you can use chunked uploads via the command line. This is especially useful when transferring files over unstable networks or when automating uploads.

A Bash script may be available to assist with the upload from your local machine.
It is recommended to generate an application-specific password in your B2DROP account settings (under Security), use it for this upload process, and delete it afterward.
You can implement chunked uploads using the WebDAV API, as described in the official Nextcloud documentation:

📄 Chunked File Upload (WebDAV API)

A reference script that has been used by other users for uploading large files to B2DROP is provided below. It splits the file into chunks and uploads each chunk separately:

#!/bin/bash
# Script to upload a big file to B2DROP
# Usage: upload_to_b2drop.sh <file_to_upload> <b2drop_destination> <b2drop_username> <b2drop_password>

# Check if the number of arguments is correct
if [ $# -ne 4 ]; then
    echo "Usage: upload_to_b2drop.sh <file_to_upload> <b2drop_destination> <b2drop_username> <b2drop_password>"
    exit 1
fi

# Store the arguments in variables
file_to_upload=$1
b2drop_destination=$2
b2drop_username=$3
b2drop_password=$4
chunk_size=5000000000 # 5G

# Create a UUID folder for the file
uuid_folder=$(uuidgen)

# Create the folder in the B2DROP account /uploads
b2drop_upload_folder="https://b2drop.bsc.es/remote.php/dav/uploads/$b2drop_username/$uuid_folder"
curl -X MKCOL -u $b2drop_username:$b2drop_password $b2drop_upload_folder
echo "Created folder $b2drop_upload_folder"

# Split the file in 5G chunks
split -b $chunk_size -a 10 $file_to_upload $file_to_upload.chunk
echo "Splitted $file_to_upload in chunks of $chunk_size"

# Upload the chunks to the B2DROP account, remove the chunks as they are uploaded
for chunk in $file_to_upload.chunk*; do
    curl -T $chunk -u $b2drop_username:$b2drop_password $b2drop_upload_folder/$chunk && echo "Uploaded $chunk"
done
echo "All chunks uploaded"

# Remove the chunks
rm $file_to_upload.chunk*

# Create the file in the B2DROP account
b2drop_file_destination=https://b2drop.bsc.es/remote.php/dav/files/$b2drop_username/$b2drop_destination
curl -X MOVE -u $b2drop_username:$b2drop_password --header \'Destination:$b2drop_file_destination\' $b2drop_upload_folder/.file

echo "File uploaded to B2DROP to $b2drop_file_destination"

Method 3: Mounting Remote Directories with SSHFS¶

You can mount a remote directory (e.g., from a BSC system) onto your local machine using sshfs, allowing you to easily browse and transfer files to B2DROP.

Steps:

Install the sshfs package on your computer.
Create a local directory to mount the remote data:

bash mkdir ~/b2drop_transfer
Mount the remote directory with the following command (replace placeholders accordingly):

bash sshfs -o idmap=user,reconnect,ro your_username@remote_host:/path/to/data ~/b2drop_transfer
Transfer files to B2DROP (see the Large Files section below if handling big datasets).
When finished, unmount the directory:

bash fusermount -u ~/b2drop_transfer

Additional Resources (Nextcloud)¶

B2DROP is built on Nextcloud, and the following documentation may also be useful:

💻 Command-Line Interface (Nextcloud Desktop Client)
https://docs.nextcloud.com/desktop/3.4/advancedusage.html