where:
my_bucket_name is the name of your bucket
-s is used to include all subfolders (e.g. recursive)
-sum is used to show just a summary without listing all objects.

The output will be similar to the following:

Bucket: my_bucket_name
5417 Files (10.74GB = 11533607629B)

How to configure S3Express for Minio S3 compatible service

Requires S3Express version 1.5.9 or newer.

You can configure S3Express to work with Minio by setting the following options in S3Express.

A) Set the S3Express end-point to Minio IP and port number. For example, assuming Minio is running on IP 191.168.1.10 and port 9000:

setopt -endpoint:192.168.1.10:9000

B) Enable the S3Express option to use path-style requests:

setopt -usepathstyle:on

C) If your Minio installation only supports HTTP and not HTTPs, then set S3Express to use the HTTP protocol:

setopt -protocol:http

D) Enter your Minio authorization credentials in S3Express as you would do for S3, using the saveauth command.

You are now ready to use S3Express with Minio.

How to list all non-private objects in a bucket

Using S3Express you can easily list all non-private objects in a bucket.
The command to use is the following:

ls my_bucket_name -s -cond:"s3_acl_is_private = false"

where:
my_bucket_name is the name of the bucket
-s is used to include subfolders (e.g. recursive)
-cond is the filtering condition to only list objects which do not have private ACL.

This command will list all non-private objects in a S3 bucket.

If you prefer to just see a summary of the total amount of objects present in a bucket that are not private, instead of listing each object's name, add the flag -sum, e.g.:

ls my_bucket_name -s -cond:"s3_acl_is_private = false" -sum

Depending on the amount of objects to check in the bucket, it may take some time for the above command to complete, because each object's ACL must be queried by S3Express, even if the querying is done by S3Express in a multithreaded, concurrent fashion.

How to list all public objects in a bucket

Sometimes it can be useful to check if there are publicly accessible objects in a specific S3 bucket. Using S3Express you can easily list all public objects in a bucket. The command to use is the following:

ls my_bucket_name -s -cond:"s3_acl_is_public_read = true"

where:
my_bucket_name is the name of the bucket
-s is used to include subfolders (e.g. recursive)
-cond is the filtering condition to only list objects which have public-access ACL.

This command will list all public objects in a S3 bucket.

If you prefer to just see a summary of the total amount of objects present in a bucket that are publicly accessible, instead of listing each object's name, add the flag -sum, e.g.:

ls my_bucket_name -s -cond:"s3_acl_is_public_read = true" -sum

Other options for the filtering condition -cond are s3_acl_is_private or s3_acl_is_public_read_write , see S3Express Manual for more details.

Depending on the amount of objects in the bucket, it may take some time for the above command to complete, because each object's ACL must be queried by S3Express, even if the querying is done by S3Express in a multithreaded fashion.

How to move files to S3 (difference between -move and -localdelete)

*** Requires S3Express 1.5 or newer *** Release History

S3Express can be used to move files to Amazon S3. Moving files means that local files are deleted after/only if they are successfully uploaded to S3. To do that, use the option -move of the put command. The -move option instructs S3Express to immediately delete local files after/only if they are successfully uploaded.

A slightly different capability is given by the option -localdelete:COND. This option instructs S3Express to delete local files if they:

- Are not selected to be uploaded (e.g. due to the option -onlydiff or -onlynewer).

- Have a corresponding matching file already on S3 with same path and name.

- Satisfy the condition COND. COND is a condition that follows the general condition rules.

By using the option -localdelete:COND, a more sophisticated move operation can be setup, because differently from the basic -move option, -localdelete will:

- Delete local files after having verified they are already on S3, so local files are deleted in a successive run of S3Express, not immediately after they are uploaded like with the -move option.

- Delete local files according to a condition e.g. -localdelete:”age_days > 30” or -localdelete:”size_mb > 50” to delete local files selectively.

- Delete local files that are already on S3 even if these files were not uploaded by S3Express itself.

Note 1: uploading has priority over -localdelete. During the execution of the put command, firstly files are selected for uploading and after that the remaining files are selected for local deletion if the option -localdelete is specified.

Note 2: If the condition COND is not specified, that is, only -localdelete is used, then all local files that have a corresponding matching file on S3 will be deleted, regardless of age, size, time, etc.

How to restore multiple objects from AWS S3 Glacier with one command

Restoring a small number of objects from AWS S3 Glacier is not a problem: you can use the AWS Web Console and restore them in the GUI. But if you need to restore a LOT of objects, the manual approach is not feasible.

With S3Express, it is easy to restore multiple objects using the restore command. The restore command in S3Express fully supports file masks and conditions, so you can restore objects based on name, extension, size, ACL, metadata, time, age and much more. See S3Express Restore Command for all details.

Some examples of commands that can be issued are:

restore mybucket/a.txt -days:5 -tier:Expedited

restore *.jpg -s -days:10 -tier:Bulk

restore mybucket/* -s -days:1 (restore all objects in mybucket and subfolders for 1 day)

restore "mybucket/my folder/.*\.txt|.*\.vsn" -r -days:2 (restore all objects with extension txt or vsn in mybucket/my folder/ for 2 days)

restore mybucket/^r.* -r -days:2 (restore all objects starting with 'r' in mybucket for 2 days)

restore mybucket -cond:"name starts_with 'a'" - days:7 (restore all objects in mybucket (non-recursive) if name starting with a)

restore mybucket -s -cond:"extract_value(cache-control,'max-age') > 0" -days:1 (restore all objects in mybucket (recursive, include subfolders) for 1 day if cache-control:max-age > 0 in the metadata)

See S3Express Restore Command for all details.

How to set 'retry values' for network errors

You can instruct S3Express to automatically retry in case of network error, by setting the global options -retry and -retrywait with the command setopt : www.s3express.com/help/vv_options.html

The option -retry sets the number of retries performed by S3Express in case of a network error. By default S3Express retries 3 times.

The option -retrywait sets the wait time, in seconds, before a retry. The default value is 5 seconds.

If you do not want S3Express to retry on network error, set -retry to 0.

How to throttle bandwidth (for file uploads to S3)

You can limit the maximum bandwidth used by S3Express during file uploads.

The maximum bandwidth can be set via the flag -maxb of the put command, see manual: www.s3express.com/help/vv_put.html

For example, using -maxb:100 will instruct S3Express to use maximum 100KB/sec to upload files.

How to upload only changed or new files since the last upload to S3 (-onlydiff)

The following is an example of the put command that can be used to upload only changed or new files since the last upload to an Amazon S3 bucket:

put c:\myfolder\ my_bucket_name -s -onlydiff

where:

c:\myfolder\ : the local folder to upload.
my_bucket_name : the S3 bucket name.
-s : include subfolders too (= recursive).
-onlydiff : only upload files that are different compared to the matching files that are already in the S3 bucket. Different files are files that have the same path, and the same name, but a different MD5 value. Different files are also files that are not yet uploaded to S3, i.e. new files. So using the '-onlydiff' flag will upload files that are not yet on S3 plus all the files whose content has changed compared to the files already on S3.

The -onlydiff flag can be used to perform incremental uploads / backups to Amazon S3 from the command line. S3Express will firstly compare the local files vs the corresponding remote files and will then only upload files whose MD5 value is different or files that do not already have a corresponding matching file on S3 (that is, new files).

Other flags that could be used instead of -onlydiff are :

-onlynewer : only upload files that are newer compared to the matching files that are already on S3. Newer files are files that have the same path and the same name but a newer modified time. Newer files are also files that are not yet uploaded to S3. So using the '-onlynewer' flag uploads files that are not yet on S3 plus all the files whose timestamp is newer compared to files already on S3.
The difference between -onlynewer and -onlydiff is that -onlydiff uses the MD5 value to compare files, e.g. files are different if the MD5 value is different, while -onlynewer uses the file timestamp, e.g. a file is different if the timestamp is different.

-onlydifferent : only upload files that are new, that is not yet on S3. Using -onlynew only uploads files that are not yet on S3.

-onlyexisting : only upload files that are already existing on S3. Using -onlyexisting only uploads files that already have a corresponding matching file with same name and path on S3.

How to upload very large files to Amazon S3 (efficiently)

Using Amazon S3 and S3Express command line you can upload very large files to a S3 bucket efficiently (e.g. several megabytes or even multiple gigabytes).

The main issues with uploading large files over the Internet are:

The upload could be involuntarily interrupted by a transient network issue and if that happens, the whole upload could fail and it would need to be restarted for the beginning. If the file is very large, it would result in wasted time and bandwitdh.
Being a large file, the upload could be voluntarily interrupted by the user with the intent of being continued at a later stage. In that case again, the whole upload would need to be restarted from the beginning.
Being the upload one big file, only one thread at a time can be used to upload the file and that would make the transfer quite slow.

All of the above issues are solved using multipart uploads.

By specifying the flag -mul of the command put when uploading files, S3Express will break the files into chunks (by default each chunk will be 5MB) and upload them separately.

You can instruct S3Express to upload a number of chunks in parallel using the flag -t.

If the upload of one single chunk fails, for whatever reason, or if the upload is interrupted, you can simply restart the uncompleted upload and S3Express will restart from the last successful chunk instead of having to re-upload the entire file. If you do not want to restart an unfinished multipart upload, you can use the command rmupl to remove the uncompleted upload.

Once all chucks are uploaded, the file is reconstructed at the destination to exaclty match the origin file. S3Express will also recaclulate and apply the correct MD5 value.

The multipart upload feature in S3Express makes it very convenient to upload very large files to Amazon S3, even over less reliable network connections, using the command line.

How to use the -onlydiff switch with local encryption (-le)

When using local encryption (-le), the MD5 value of the files that are uploaded to S3 will no longer match the local MD5 value, because the files are encrypted before being uploaded. This will prevent the -onlydiff switch from working properly. However there are alternatives:

One alternative is to use the -onlynewer switch instead of the -onlydiff switch.

The -onlynewer switch instructs S3Express to only upload files that have a newer timestamp locally compared to files that are already on S3 but are older. Checking timestamp of S3 objects is very fast because the S3 metadata of each S3 object do not need to be checked.

The other possibility is to use the original MD5 value that is stored in the S3 metadata of locally encrypted files. When uploading locally encrypted files to S3, S3Express stores the original MD5 value in the metadata header x-amz-meta-s3xpress-encrypted-orig-md5. So, by adding the following condition to the upload command, you can upload only files that have a changed MD5: -cond:"md5<>x-amz-meta-s3xpress-encrypted-orig-md5".

For instance: put c:\localfolder\ bucketname -s -le
-cond:"md5<>x-amz-meta-s3xpress-encrypted-orig-md5"

Note that the metadata of each S3 object must be check in this case which can take a long time if there are many files to check (it depends also on the available connection speed to Amazon S3).

A printable version of the entire FAQ and Knowledge Base is also available.

For further queries, please contact us by e-mail at support@s3express.com