org.duracloud.s3storage

Class S3StorageProvider

java.lang.Object

org.duracloud.storage.provider.StorageProviderBase

org.duracloud.s3storage.S3StorageProvider

All Implemented Interfaces:

StorageProvider

Direct Known Subclasses:

GlacierStorageProvider, SnapshotStorageProvider

public class S3StorageProvider
extends StorageProviderBase

Provides content storage backed by Amazon's Simple Storage Service.

Author:

Bill Branan

Nested Class Summary

Nested classes/interfaces inherited from class org.duracloud.storage.provider.StorageProviderBase

StorageProviderBase.SpaceDeleteWorker

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	MAX_ITEM_COUNT
protected com.amazonaws.services.s3.AmazonS3Client	s3Client

Fields inherited from class org.duracloud.storage.provider.StorageProviderBase

ACL_DELIM, ACL_GROUP_READ, ACL_GROUP_WRITE, ACL_USER_READ, ACL_USER_WRITE

Fields inherited from interface org.duracloud.storage.provider.StorageProvider

DEFAULT_MAX_RESULTS, DEFAULT_MIMETYPE, ISO8601_DATE_FORMAT, PROPERTIES_CONTENT_CHECKSUM, PROPERTIES_CONTENT_CREATOR, PROPERTIES_CONTENT_FILE_CREATED, PROPERTIES_CONTENT_FILE_LAST_ACCESSED, PROPERTIES_CONTENT_FILE_MODIFIED, PROPERTIES_CONTENT_FILE_PATH, PROPERTIES_CONTENT_MD5, PROPERTIES_CONTENT_MIMETYPE, PROPERTIES_CONTENT_MODIFIED, PROPERTIES_CONTENT_SIZE, PROPERTIES_COPY_SOURCE, PROPERTIES_COPY_SOURCE_STORE, PROPERTIES_SPACE_ACL, PROPERTIES_SPACE_ACL_GROUP, PROPERTIES_SPACE_ACL_PUBLIC, PROPERTIES_SPACE_COUNT, PROPERTIES_SPACE_CREATED, PROPERTIES_SPACE_SIZE, PROPERTIES_STREAMING_HOST

Constructor Summary

Constructors

Constructor and Description
S3StorageProvider(com.amazonaws.services.s3.AmazonS3Client s3Client, String accessKey, Map<String,String> options)
S3StorageProvider(String accessKey, String secretKey)
S3StorageProvider(String accessKey, String secretKey, Map<String,String> options)

Method Summary

Methods

Modifier and Type	Method and Description
String	addContent(String spaceId, String contentId, String contentMimeType, Map<String,String> userProperties, long contentSize, String contentChecksum, InputStream content) Adds content to a space.
String	copyContent(String sourceSpaceId, String sourceContentId, String destSpaceId, String destContentId) This method copies the content item found in source-space with the id of source-content-id into the dest-space, naming it to dest-content-id.
void	createSpace(String spaceId) Creates a new space.
void	deleteContent(String spaceId, String contentId) Removes content from a space.
protected String	doesContentExist(String bucketName, String contentId)
protected void	doSetSpaceProperties(String spaceId, Map<String,String> spaceProperties)
protected Map<String,String>	getAllSpaceProperties(String spaceId)
String	getBucketName(String spaceId) Gets the name of an existing bucket based on a space ID.
InputStream	getContent(String spaceId, String contentId) Gets content from a space.
Map<String,String>	getContentProperties(String spaceId, String contentId) Retrieves the properties associated with content.
protected String	getETagValue(String etag)
protected String	getNewBucketName(String spaceId)
Iterator<String>	getSpaceContents(String spaceId, String prefix) Provides access to the content files within a space.
List<String>	getSpaceContentsChunked(String spaceId, String prefix, long maxResults, String marker) Provides a listing of the content files within a space.
protected String	getSpaceCount(String spaceId, int maxCount)
protected String	getSpaceFree(String name) Replaces all spaces with "%20"
protected String	getSpaceId(String bucketName) Converts a bucket name into what could be passed in as a space ID.
Iterator<String>	getSpaces() Provides a listing of all spaces owned by a customer.
protected String	getWithSpace(String name) Converts "%20" back to spaces
protected boolean	isSpace(String bucketName) Determines if an S3 bucket is a DuraCloud space
protected Map<String,String>	removeCalculatedProperties(Map<String,String> contentProperties)
void	removeSpace(String spaceId)
void	setContentProperties(String spaceId, String contentId, Map<String,String> contentProperties) Sets the properties associated with content.
protected boolean	spaceExists(String spaceId)
protected void	wait(int seconds)

Methods inherited from class org.duracloud.storage.provider.StorageProviderBase

deleteSpace, deleteSpaceSync, getSpaceACLs, getSpaceDeleteWorker, getSpaceProperties, packACLs, setNewSpaceProperties, setSpaceACLs, setWrappedStorageProvider, throwIfSpaceExists, throwIfSpaceNotExist, throwIfSpaceNotExist, unpackACLs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_ITEM_COUNT

protected static final int MAX_ITEM_COUNT

See Also:

Constant Field Values

s3Client

protected com.amazonaws.services.s3.AmazonS3Client s3Client

Constructor Detail

S3StorageProvider

public S3StorageProvider(String accessKey,
                 String secretKey)

S3StorageProvider

public S3StorageProvider(String accessKey,
                 String secretKey,
                 Map<String,String> options)

S3StorageProvider

public S3StorageProvider(com.amazonaws.services.s3.AmazonS3Client s3Client,
                 String accessKey,
                 Map<String,String> options)

Method Detail

getSpaces

public Iterator<String> getSpaces()

Provides a listing of all spaces owned by a customer.

Returns:

Iterator listing spaceIds

getSpaceContents

public Iterator<String> getSpaceContents(String spaceId,
                                String prefix)

Provides access to the content files within a space. Chunking of the list is handled internally. Prefix can be set to return only content IDs starting with the prefix value.

Parameters:

spaceId - - ID of the space

prefix - - The prefix of the content id (null for no constraints)

Returns:

Iterator of contentIds

getSpaceContentsChunked

public List<String> getSpaceContentsChunked(String spaceId,
                                   String prefix,
                                   long maxResults,
                                   String marker)

Provides a listing of the content files within a space. The number of items returned is limited to maxResults (default is 1000). Retrieve further results by including the last content ID in the previous list as the marker. Set prefix to return only content IDs starting with the prefix value.

Parameters:

spaceId - - ID of the space

prefix - - Only retrieve content IDs with this prefix (null for all content ids)

maxResults - - The maximum number of content IDs to return in the list (0 indicates default (1000))

marker - - The content ID marking the last item in the previous set (null indicates the first set of ids)

Returns:

List of contentIds

spaceExists

protected boolean spaceExists(String spaceId)

Specified by:

spaceExists in class StorageProviderBase

createSpace

public void createSpace(String spaceId)

Creates a new space. Depending on the storage implementation, the spaceId may be changed somewhat to comply with the naming rules of the underlying storage provider. The same spaceId value used here can be used in all other methods, as the conversion will be applied internally, however a call to getSpaces() may not include a space with exactly this same name.

Parameters:

spaceId - - ID of the space

getNewBucketName

protected String getNewBucketName(String spaceId)

removeSpace

public void removeSpace(String spaceId)

Specified by:

removeSpace in class StorageProviderBase

getAllSpaceProperties

protected Map<String,String> getAllSpaceProperties(String spaceId)

Specified by:

getAllSpaceProperties in class StorageProviderBase

getSpaceCount

protected String getSpaceCount(String spaceId,
                   int maxCount)

doSetSpaceProperties

protected void doSetSpaceProperties(String spaceId,
                        Map<String,String> spaceProperties)

Specified by:

doSetSpaceProperties in class StorageProviderBase

addContent

public String addContent(String spaceId,
                String contentId,
                String contentMimeType,
                Map<String,String> userProperties,
                long contentSize,
                String contentChecksum,
                InputStream content)

Adds content to a space. Computes the checksum of the provided content and checks this against the checksum of the uploaded content to protect against loss or corruption during transfer.

Parameters:

spaceId - - ID of the space

contentId - - ID of the content in the space

contentMimeType - - the MIME type of the content being added

userProperties - - the metadata associated with the content

contentSize - - the file size (in bytes) of the content being added

contentChecksum - - the MD5 checksum of the content being added (null if no checksum is known)

content - - content to add

Returns:

The checksum of the provided content

doesContentExist

protected String doesContentExist(String bucketName,
                      String contentId)

wait

protected void wait(int seconds)

copyContent

public String copyContent(String sourceSpaceId,
                 String sourceContentId,
                 String destSpaceId,
                 String destContentId)

Description copied from interface: StorageProvider

This method copies the content item found in source-space with the id of source-content-id into the dest-space, naming it to dest-content-id.

Parameters:

sourceSpaceId - of content to copy

sourceContentId - of content to copy

destSpaceId - where copied content will end up

destContentId - given to copied content

Returns:

MD5 checksum of destination content item

getContent

public InputStream getContent(String spaceId,
                     String contentId)

Gets content from a space.

Parameters:

spaceId - - ID of the space

contentId - - ID of the content in the space

Returns:

the content stream

deleteContent

public void deleteContent(String spaceId,
                 String contentId)

Removes content from a space.

Parameters:

spaceId - - ID of the space

contentId - - ID of the content in the space

setContentProperties

public void setContentProperties(String spaceId,
                        String contentId,
                        Map<String,String> contentProperties)

Sets the properties associated with content. This effectively removes all of the current content properties and adds a new set of properties. Some properties, such as system properties provided by the underlying storage system, cannot be updated or removed. Some of the values which cannot be updated or removed: Content-MD5 ETag Last-Modified Content-Type cannot be removed, but it can be updated

Parameters:

spaceId - - ID of the space

contentId - - ID of the content in the space

contentProperties - - new content properties

removeCalculatedProperties

protected Map<String,String> removeCalculatedProperties(Map<String,String> contentProperties)

Overrides:

removeCalculatedProperties in class StorageProviderBase

getContentProperties

public Map<String,String> getContentProperties(String spaceId,
                                      String contentId)

Retrieves the properties associated with content. This includes both properties generated by the underlying storage system as well as custom properties. Use the PROPERTIES_CONTENT_* constants to retrieve standard properties values.

Parameters:

spaceId - - ID of the space

contentId - - ID of the content in the space

Returns:

content properties

getETagValue

protected String getETagValue(String etag)

getBucketName

public String getBucketName(String spaceId)

Gets the name of an existing bucket based on a space ID. If no bucket with this spaceId exists, throws a NotFoundException

Parameters:

spaceId - the space Id to convert into an S3 bucket name

Returns:

S3 bucket name of a given DuraCloud space

Throws:

NotFoundException - if no bucket matches this spaceID

getSpaceId

protected String getSpaceId(String bucketName)

Converts a bucket name into what could be passed in as a space ID.

Parameters:

bucketName - name of the S3 bucket

Returns:

the DuraCloud space name equivalent to a given S3 bucket Id

isSpace

protected boolean isSpace(String bucketName)

Determines if an S3 bucket is a DuraCloud space

Parameters:

bucketName - name of the S3 bucket

Returns:

true if the given S3 bucket name is named according to the DuraCloud space naming conventions, false otherwise

getSpaceFree

protected String getSpaceFree(String name)

Replaces all spaces with "%20"

Parameters:

name - string with possible space

Returns:

converted to string without spaces

getWithSpace

protected String getWithSpace(String name)

Converts "%20" back to spaces

Parameters:

name - string

Returns:

converted to spaces