See PDF tutorial 'Backup Files to Amazon S3 with S3Express'.
To calculate the total size of all objects contained in a S3 bucket, use the following command:
ls my_bucket_name -s -sum
The output will be similar to the following:
You can configure S3Express to work with Minio by setting the following options in S3Express.
A) Set the S3Express end-point to Minio IP and port number. For example, assuming Minio is running on IP 22.214.171.124 and port 9000:
Using S3Express you can easily list all non-private objects in a bucket.
ls my_bucket_name -s -cond:"s3_acl_is_private = false"
This command will list all non-private objects in a S3 bucket.
If you prefer to just see a summary of the total amount of objects present in a bucket that are not private, instead of listing each object's name, add the flag -sum, e.g.:
ls my_bucket_name -s -cond:"s3_acl_is_private = false" -sum
Depending on the amount of objects to check in the bucket, it may take some time for the above command to complete, because each object's ACL must be queried by S3Express, even if the querying is done by S3Express in a multithreaded, concurrent fashion.
Sometimes it can be useful to check if there are publicly accessible objects in a specific S3 bucket. Using S3Express you can easily list all public objects in a bucket. The command to use is the following:
ls my_bucket_name -s -cond:"s3_acl_is_public_read = true"
This command will list all public objects in a S3 bucket.
If you prefer to just see a summary of the total amount of objects present in a bucket that are publicly accessible, instead of listing each object's name, add the flag -sum, e.g.:
ls my_bucket_name -s -cond:"s3_acl_is_public_read = true" -sum
Other options for the filtering condition -cond are s3_acl_is_private or s3_acl_is_public_read_write , see S3Express Manual for more details.
Depending on the amount of objects in the bucket, it may take some time for the above command to complete, because each object's ACL must be queried by S3Express, even if the querying is done by S3Express in a multithreaded fashion.
*** Requires S3Express 1.5 or newer *** Release History
S3Express can be used to move files to Amazon S3. Moving files means that local files are deleted after/only if they are successfully uploaded to S3. To do that, use the option -move of the put command. The -move option instructs S3Express to immediately delete local files after/only if they are successfully uploaded.
- Are not selected to be uploaded (e.g. due to the option -onlydiff or -onlynewer).
- Delete local files after having verified they are already on S3, so local files are deleted in a successive run of S3Express, not immediately after they are uploaded like with the -move option.
Note 2: If the condition COND is not specified, that is, only -localdelete is used, then all local files that have a corresponding matching file on S3 will be deleted, regardless of age, size, time, etc.
Restoring a small number of objects from AWS S3 Glacier is not a problem: you can use the AWS Web Console and restore them in the GUI. But if you need to restore a LOT of objects, the manual approach is not feasible.
With S3Express, it is easy to restore multiple objects using the restore command. The restore command in S3Express fully supports file masks and conditions, so you can restore objects based on name, extension, size, ACL, metadata, time, age and much more. See S3Express Restore Command for all details.
Some examples of commands that can be issued are:
restore mybucket/a.txt -days:5 -tier:Expedited
restore *.jpg -s -days:10 -tier:Bulk
restore mybucket/* -s -days:1 (restore all objects in mybucket and subfolders for 1 day)
restore "mybucket/my folder/.*\.txt|.*\.vsn" -r -days:2 (restore all objects with extension txt or vsn in mybucket/my folder/ for 2 days)
restore mybucket/^r.* -r -days:2 (restore all objects starting with 'r' in mybucket for 2 days)
restore mybucket -cond:"name starts_with 'a'" - days:7 (restore all objects in mybucket (non-recursive) if name starting with a)
restore mybucket -s -cond:"extract_value(cache-control,'max-age') > 0" -days:1 (restore all objects in mybucket (recursive, include subfolders) for 1 day if cache-control:max-age > 0 in the metadata)
See S3Express Restore Command for all details.
You can instruct S3Express to automatically retry in case of network error, by setting the global options -retry and -retrywait with the command setopt : www.s3express.com/help/vv_options.html
The option -retry sets the number of retries performed by S3Express in case of a network error. By default S3Express retries 3 times.
The option -retrywait sets the wait time, in seconds, before a retry. The default value is 5 seconds.
If you do not want S3Express to retry on network error, set -retry to 0.
You can limit the maximum bandwidth used by S3Express during file uploads.
The maximum bandwidth can be set via the flag -maxb of the put command, see manual: www.s3express.com/help/vv_put.html
For example, using -maxb:100 will instruct S3Express to use maximum 100KB/sec to upload files.
The following is an example of the put command that can be used to upload only changed or new files since the last upload to an Amazon S3 bucket:
put c:\myfolder\ my_bucket_name -s -onlydiff
c:\myfolder\ : the local folder to upload.
-onlynewer : only upload files that are newer compared to the matching files that are already on S3. Newer files are files that have the same path and the same name but a newer modified time. Newer files are also files that are not yet uploaded to S3. So using the '-onlynewer' flag uploads files that are not yet on S3 plus all the files whose timestamp is newer compared to files already on S3.
-onlydifferent : only upload files that are new, that is not yet on S3. Using -onlynew only uploads files that are not yet on S3.
-onlyexisting : only upload files that are already existing on S3. Using -onlyexisting only uploads files that already have a corresponding matching file with same name and path on S3.
Using Amazon S3 and S3Express command line you can upload very large files to a S3 bucket efficiently (e.g. several megabytes or even multiple gigabytes).
The main issues with uploading large files over the Internet are:
You can instruct S3Express to upload a number of chunks in parallel using the flag -t.
If the upload of one single chunk fails, for whatever reason, or if the upload is interrupted, you can simply restart the uncompleted upload and S3Express will restart from the last successful chunk instead of having to re-upload the entire file. If you do not want to restart an unfinished multipart upload, you can use the command rmupl to remove the uncompleted upload.
Once all chucks are uploaded, the file is reconstructed at the destination to exaclty match the origin file. S3Express will also recaclulate and apply the correct MD5 value.
The multipart upload feature in S3Express makes it very convenient to upload very large files to Amazon S3, even over less reliable network connections, using the command line.
When using local encryption (-le), the MD5 value of the files that are uploaded to S3 will no longer match the local MD5 value, because the files are encrypted before being uploaded. This will prevent the -onlydiff switch from working properly. However there are alternatives:
A printable version of the entire FAQ and Knowledge Base is also available.
For further queries, please contact us by e-mail at email@example.com