HTCondorCEComputingElement

HTCondorCE Computing Element

Allows direct submission to HTCondorCE Computing Elements with a SiteDirector Agent Needs the condor grid middleware (condor_submit, condor_history, condor_q, condor_rm)

Configuration for the HTCondorCE submission can be done via the configuration system

WorkingDirectory: Location to store the pilot and condor log files
DaysToKeepLogs:  how long to keep the log files until they are removed
ExtraSubmitString: Additional option for the condor submit file, separate options with '\n', for example:
   request_cpus = 8 \n periodic_remove = ...
UseLocalSchedd: If False, directly submit to a remote condor schedule daemon,
then one does not need to run condor daemons on the submit machine

see HTCondorCE Parameters

class DIRAC.Resources.Computing.HTCondorCEComputingElement.HTCondorCEComputingElement(ceUniqueID)

Bases: DIRAC.Resources.Computing.ComputingElement.ComputingElement

HTCondorCE computing element class implementing the functions jobSubmit, getJobOutput

__init__(ceUniqueID)

Standard constructor.

available(jobIDList=None)

This method returns the number of available slots in the target CE. The CE instance polls for waiting and running jobs and compares to the limits in the CE parameters.

Parameters:jobIDList (list) – list of already existing job IDs to be checked against
getCEStatus()

Method to return information on running and pending jobs.

getDescription()

Get CE description as a dictionary.

This is called by the JobAgent for the case of “inner” CEs.

getJobOutput(jobID, _localDir=None)

TODO: condor can copy the output automatically back to the submission, so we just need to pick it up from the proper folder

getJobStatus(jobIDList)

Get the status information for the given list of jobs

getParameterDict()

Get the CE complete parameter dictionary

initializeParameters()

Initialize the CE parameters after they are collected from various sources

isProxyValid(valid=1000)

Check if the stored proxy is valid

isValid()

Check the sanity of the Computing Element definition

killJob(jobIDList)

Kill the specified jobs

loadBatchSystem()

Instantiate object representing the backend batch system

sendOutput(stdid, line)

Callback function such that the results from the CE may be returned.

setCPUTimeLeft(cpuTimeLeft=None)

Update the CPUTime parameter of the CE classAd, necessary for running in filling mode

setParameters(ceOptions)

Add parameters from the given dictionary overriding the previous values

Parameters:ceOptions (dict) – CE parameters dictionary to update already defined ones
setProxy(proxy, valid=0)

Set proxy for this instance

submitJob(executableFile, proxy, numberOfJobs=1, processors=1)

Method to submit job

writeProxyToFile(proxy)

CE helper function to write a CE proxy string to a file.

DIRAC.Resources.Computing.HTCondorCEComputingElement.condorIDAndPathToResultFromJobRef(jobRef)

Extract tuple of jobURL and jobID from the jobRef string. The condorID as well as the path leading to the job results are also extracted from the jobID.

Parameters:jobRef (str) – PilotJobReference of the following form: htcondorce://<ceName>/<condorID>:::<pilotStamp>
Returns:tuple composed of the jobURL, the path to the job results and the condorID of the given jobRef
DIRAC.Resources.Computing.HTCondorCEComputingElement.condorIDFromJobRef(*args, **kwargs)

Extract tuple of jobURL and jobID from the jobRef string.

param str jobRef:
 PilotJobReference of the following form: htcondorce://<ceName>/<pathToResult>-<condorID>
return:tuple composed of the jobURL and the condorID of the given jobRef

Deprecated: Please use condorIDAndPathToResultFromJobRef

DIRAC.Resources.Computing.HTCondorCEComputingElement.findFile(workingDir, fileName, pathToResult=None)

Find a file in a file system.

Parameters:
  • workingDir (str) – the name of the directory containing the given file to search for
  • fileName (str) – the name of the file to find
  • pathToResult (str) – the path to follow from workingDir to find the file
Returns:

list of paths leading to the file

DIRAC.Resources.Computing.HTCondorCEComputingElement.getCondorLogFile(pilotRef)

Return the location of the logFile belonging to the pilot reference.

DIRAC.Resources.Computing.HTCondorCEComputingElement.logDir(ceName, stamp)

Return path to log and output files for pilot.

Parameters:
  • ceName (str) – Name of the CE
  • stamp (str) – pilot stamp from/for jobRef