Amazon S3 - Basics

S3 is the Amazon Web Services (AWS) cloud storage system. censhare uses Amazon S3 as an additional (or only) file system for storing asset files. This article explains how to work as an administrator with a file system that uses an Amazon S3 bucket.

Amazon Web Services (AWS) is a service provided by Amazon. The use of this Amazon service can result in additional costs that are invoiced directly by Amazon. censhare cannot influence or control these costs and therefore cannot be held responsible for them.

Context

• The XML of an asset provides information about the used file system for related asset files.

• The server log contains information which S3 bucket is used to store asset files.

• censhare provides manual and automatic server actions to move asset files to another file system. This can be necessary if the new domain of an asset belongs to another file system than the old domain.

Prerequisites

• The censhare Server is configured to use the desired Amazon S3 bucket as an asset file system. 

• Administration mode permission for the censhare Client

• Access to the censhare Server logs

• Permission to access the Module configuration section in the censhare Admin Client

Introduction

When you create a new asset from a file, the domain of an asset determines which file system censhare uses to store the file. This information is not shown when censhare creates the asset. The information about the used file system can be retrieved in various ways:

• censhare Client: Inspect the asset XML.

• censhare Admin Client: In the File systems table in the Master Data. 

• censhare Server log: When censhare saves a file, a log entry is created with information about the file system. If the file system is Amazon S3, this includes the name of the S3 bucket.

If you change the domain of an asset and the new domain uses a different file system, censhare does not automatically move the associated files to the respective file system. You must configure a manual or automatic server action to do this.

With a manual server action, you have full control over whether and when to move an asset file to the file system that is associated to the new asset domain. For example, this is useful during the testing phase of an Amazon S3 file system.

For certain events, you can also use asset automation to tell censhare to automatically check for differences between domains and the current file system. censhare may then move the files to the file system associated with the domain. For example, for every  Save and Close asset event, censhare can perform a check for the asset in question.

The manual and automatic server actions can be configured to either move all versions of the file to the new domain or to move only the current version. Moving all versions ensures consistent storage. Moving only the current version allows you to move the latest published version of a file between different locations.

For example, you have a subcontractor that takes care of detailed layout work. Using certain workflow steps of a workflow you can assign a layout to that subcontractor for further processing and at the same time move the current files to their location.

Create assets from files

If you create a new asset from a file, select the desired Amazon S3 domain, for example, "Domain=Root/Amazon S3". censhare saves the file in the Amazon S3 bucket assigned to that domain.

For every asset file, the XML representation of the asset shows the file system that is used to store the file. The XML also shows the relative path of the file. The asset attributes contain the domains to which the asset (and files) belong.

The XML view is available in the censhare Client:

1. Open the client.

2. Switch to administrator mode.

3. Search in the censhare Client for the asset you created.

4. Right-click the asset in the query window to open the associated menu.

5. Select the option Admin followed by Show XML.

6. An editor window opens with the XML view.

In the first line with the tag "<asset" you find information such as the asset domains. For example:

<asset ... domain="root.amazon_s3." domain2="root."...>

The asset is located in the domain with the ID "root.amazon_s3." and the second domain with the ID "root.".

For each of the asset's files there is a proprietary entry that starts with the tag "<storage_item". Here you can see which file system censhare uses for the asset (attribute  filesys_name) as well as the name of the relative path within the file system (attribute relpath). For example:

<storage_item asset_id="56656"... filesys_name="s3-assets"... 
     rel-path="file:28/41/284152.jpg"... />

censhare saved the file belonging to the asset with the ID 56656 on the file system with the ID "s3-assets". The associated domain can be looked up in the censhare Admin Client in the Master data/File systems table in the Master data section of the censhare Admin Client. The relative path there is "="28/41/". The name of the file itself is "284152.jpg".

The local cache file system for the selected Amazon S3 asset file system keeps a copy of the uploaded file. You specified the path in the Filesystem Service. 

You can also open the "AWS Management Console" for the S3 management area and open the associated bucket. The uploaded file is there as well. You can see their names in the log entries.

A log entry that was created when saving an asset file in an Amazon S3 bucket looks similar to the following example:

RMI TCP Connection(31)-127.0.0.1: S3FileSystemImpl: 
     s3-assets(bucketName: bucket2).getOutputStream of 28/44/284470.png, 
     output stream created and returned to caller. 

"s3-assets" is the asset file system ID. "bucketName: bucket2" contains the name of the bucket. Here, "bucket2". "284470.png" is the name of the file on Amazon S3. "output stream created and returned to caller" is part of the process of saving the file to the bucket.

T077: AssetManagementService: COMMAND-ID: computed hash code for file
     munich:s3-assets:https://bucket2.amazonaws.com/,28/44/284470.png in 4 ms:

"https://bucket2.amazonaws.com/,28/44/284470.png" is the full pathname for the uploaded file on Amazon S3. "https://bucket2.amazonaws.com" is the path of the bucket. COMMAND-ID is a unique ID. It is created to identify all log entries that belong to a command, for example, "munich.20160202.151333.805[censhare]". "Munich" is the name of the censhare Server. "censhare" is the user account under which the command was carried out.

T077: AssetManagementService: COMMAND-ID: The temp file provided to the 
     asset management: s3-assets, file:28/44/284470.png for asset with id 56832
     already is on an asset file system. The file is reused and not moved. 

"id 56832" contains the ID of the asset that censhare created when uploading the file. The log entry tells you that the asset management of the censhare Server has recognized that the asset temp and asset file system are the same. As a result, the file is not moved from asset temp to the asset file system. 

Moving files to a different asset file system

Overview

There are two options for the manual and automatic file system switch. Either censhare moves only the current version of the asset files or it moves all versions. The latter ensures that all asset files with all versions are available on the associated file system. The former allows you to move the current files between different locations.

For example, you have a subcontractor that takes care of detailed layout work. Using certain workflow steps of a workflow you can assign a layout to that subcontractor for further processing and at the same time, censhare can move the current files to their location.

The server actions

The server actions are available in the censhare Admin Client in the Configuration/Modules/Asset Files  folder:

The server action  Switch Filesystem for the current Asset Versions (automatic)  with the configuration file  switch-filesystem.xml  configuration file is deprecated.

Test a server action

How to test the move of asset files to another file system:

1. Select the server action  Switch Filesystem for all Asset Versions in the censhare Admin Client.

2. Double click to open it and select Enabled under  General setup.

3. Click Update Server Configuration in the toolbar.

4. In the censhare Client, select an asset and change the domain to one that is in another file system.

5. Select the asset and execute the server action  Switch Filesystem for all Asset Versions.

Check the serverlog

In the log file of the server, you can check what happens when you move the asset files. For example:

CommandExecutor: COMMAND-ID: 
  asset_file.aa-switch-filesystem-versions-action-20160301-1553-0 execute

This is where the server action starts.. "asset_file.aa-switch-filesystem-versions-action-20160301-1553-0" is the name of the configuration file for the module you changed on the server. For more information on the COMMAND-ID, see Creating an asset.

...
AASwitchFilesystemVersions.serverActionSetup: CommandExecutor: COMMAND-ID: 
  taking: Translation with memory.png[56832-v3-c0-tcn16-ccn18] 

"Translation with memory.png" is the name of the asset whose files are moved.

...
AASwitchFilesystemVersions.serverActionSetup: CommandExecutor: COMMAND-ID: 
  Copying file from 'SERVER:assets:work/assets/,29/03/290382.jpg' to 
  'SERVER:s3-assets:,29/45/294567.jpg' (size: 6781) 

This is where the asset file move starts. "SERVER:assets:work/assets/,29/03/290382.jpg" is the original location of the file. "assets" is the file system. "work/assets/" is the location of the file system and SERVER is the name of the censhare server. "29/03/290382.jpg" is the relative path within the file system. "SERVER:s3-assets:,29/45/294567.jpg" is the new file location. The file is moved to the file system "s3-assets" and now has the relative path "29/45/294567.jpg".

...
AASwitchFilesystemVersions.serverActionSetup: FileFactoryService: COMMAND-ID: 
     stream copy SERVER:assets:work/assets/,29/03/290382.jpg ->
     SERVER:s3-assets:https://bucket2.amazonaws.com/,29/45/294567.jpg 

Here you see the complete address for the new location: "https://bucket2.amazonaws.com/,29/45/294567.jpg". "bucket2" is the bucket used in the Amazon S3 file system.

...
CommandExecutor: COMMAND-ID: 
     asset_file.aa-switch-filesystem-versions-action-20160301-1553-0 
     completed all in 40336ms 

The censhare server has completed the entire operation.

Result

If the new domain of an asset also includes a new file system, the files of this asset are not automatically transferred: You know how to get more information about the used asset file system. You know how to manually or automatically trigger the switch of the file system.