S3Storage

Configuration of an S3 storage Like others, but in protocol S3 add:

  • SecureConnection: true if https, false otherwise
  • Aws_access_key_id
  • Aws_secret_access_key

if the Aws variables are not defined, it will try to go throught the S3Gateway

The key of the objects are the LFN without trailing path. The Path should be the BucketName

class DIRAC.Resources.Storage.S3Storage.S3Storage(storageName, parameters)

Bases: DIRAC.Resources.Storage.StorageBase.StorageBase

class StorageBase
DYNAMIC_OPTIONS = {}
PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']
__init__(storageName, parameters)

x.__init__(…) initializes x; see help(type(x)) for signature

changeDirectory(directory)

Change the directory to the supplied directory

constructURLFromLFN(lfn, withWSUrl=False)

Construct URL from the given LFN according to the VO convention for the primary protocol of the storage plagin

Parameters:
  • lfn (str) – file LFN
  • withWSUrl (boolean) – flag to include the web service part into the resulting URL
Return result:

result[‘Value’] - resulting URL

createDirectory(urls)
Create directory on the storage.
S3 does not have such a concept, but we return OK for everything
Parameters:urls – list of urls to be created on the storage
Returns:Always Successful dict {path : True }
createPresignedUrl(urls, *args, **kwargs)

Generate a presigned URL to share an S3 object

Parameters:
  • urls – urls for which to generate a presigned URL. If s3_method is put_object, it must be a dict <url:Fields> where fields are the metadata of the file (see https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_post) # pylint: disable=line-too-long # noqa
  • s3_method – name of the method for which to generate a presigned URL
  • expiration – Time in seconds for the presigned URL to remain valid
Returns:

Presigned URL as string. If error, returns None.

exists(urls)

Check if the urls exists on the storage

Parameters:urls – list of URLs
Returns:Failed dictionary: {url : error message} Successful dictionary: {url : bool} S_ERROR in case of argument problems
getCurrentDirectory()

Get the current directory

getCurrentURL(fileName)

Obtain the current file URL from the current working directory and the filename

Parameters:
  • self – self reference
  • fileName (str) – path on storage
static getDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

static getDirectoryMetadata(*_args, **_kwargs)

Generic method for unavailable method on S3

static getDirectorySize(*_args, **_kwargs)

Generic method for unavailable method on S3

getEndpoint()
This will get endpoint of the storage. It basically is the same as getURLBase()
but without the basePath
Returns:‘proto://hostname<:port>’
getFile(urls, localPath=False)

Make a local copy of the urls.

Parameters:
  • urls – list of urls on the storage
  • localPath – destination folder. Default is from current directory
Returns:

  • Successful dict: {path : size}
  • Failed dict: {path : errorMessage}
  • S_ERROR in case of argument problems

getFileMetadata(urls)

Get metadata associated to the file(s)

Parameters:urls – list of urls on the storage
Returns:
  • successful dict { path : metadata }
  • failed dict { path : error message }
  • S_ERROR in case of argument problems
getFileSize(urls)

Get the physical size of the given file

Parameters:urls – list of urls on the storage
Returns:
  • Successful dict {path : size}
  • Failed dict {path : error message }
  • S_ERROR in case of argument problem
getName()

The name with which the storage was instantiated

getOccupancy(**kwargs)

Get the StorageElement occupancy info in MB.

This generic implementation download a json file supposed to contain the necessary info.

Parameters:occupancyLFN – (mandatory named argument) LFN of the json file.
Returns:S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB
getParameters()

Get the parameters with which the storage was instantiated

getTransportURL(urls, protocols)
Get a transport URL for given urls
If http/https is requested, the URLs will be valid for 24hours
Parameters:
  • urls (dict) – s3 urls
  • protocols (list) –

    a list of acceptable transport protocols in priority order. In practice, besides ‘s3’, it can only be:

    • ’https’ if secureConnection is True
    • ’http’ othewise
Returns:

succ/failed dict url with required protocol

getURLBase(withWSUrl=False)

This will get the URL base. This is then appended with the LFN in DIRAC convention.

Parameters:
  • self – self reference
  • withWSUrl (bool) – flag to include Web Service part of the url
Returns:

URL

static isDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

isFile(urls)

Check if the urls provided are a file or not

In practice, if the object exists, it is necessarily a file

Parameters:urls – list of urls to be checked
Returns:
  • Failed dict: {path : error message}
  • Successful dict: {path : bool}
  • S_ERROR in case of argument problems
isNativeURL(url)

Check if URL :url: is valid for :self.protocol:

Parameters:
  • self – self reference
  • url (str) – URL
isOK()
isURL(path)

Guess if the path looks like a URL

Parameters:
  • self – self reference
  • path (string) – input file LFN or URL
Returns boolean:
 

True if URL, False otherwise

static listDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

static notAvailable(*_args, **_kwargs)

Generic method for unavailable method on S3

pinFile(*parms, **kws)

Pin the file on the destination storage element

pluginName = 'S3'
prestageFile(*parms, **kws)

Issue prestage request for file

prestageFileStatus(*parms, **kws)

Obtain the status of the prestage request

static putDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

putFile(urls, sourceSize=0)

Upload a local file.

..warning:: no 3rd party copy possible

Parameters:
  • urls – dictionary { urls : localFile }
  • sourceSize – size of the file in byte. Mandatory for third party copy (WHY ???) Also, this parameter makes it essentially a non bulk operation for third party copy, unless all files have the same size…
Returns:

  • Successful dict: { path : size }
  • Failed dict: { path : error message }
  • S_ERROR in case of argument problems

releaseFile(*parms, **kws)

Release the file on the destination storage element

static removeDirectory(*_args, **_kwargs)

Generic method for unavailable method on S3

removeFile(urls)

Physically remove the file specified by keys

A non existing file will be considered as successfully removed

Parameters:urls – list of urls on the storage
Returns:
  • Successful dict {path : True}
  • Failed dict {path : error message}
  • S_ERROR in case of argument problems
resetCurrentDirectory()

Reset the working directory to the base dir

setParameters(parameterDict)

Set standard parameters, method can be overriden in subclasses to process specific parameters

setStorageElement(se)
updateURL(url, withWSUrl=False)

Update the URL according to the current SE parameters