[WIP] Get CCDB time machine timestamp from CCDB by default in anchorMC.sh

Related JIRA ticket: https://its.cern.ch/jira/browse/O2-6543

This will use the script GRID/utils/getCCDBTimeMachineTimestamp.py in anchorMC.sh to obtain the CCDB time machine timestamp from the CCDB (or estimate it from the given anchored apass output files). This is now the default behavior. If the timestamp cannot be obtained, anchorMC.sh exits with an error. If set, the JDL vairable `ALIEN_JDL_CCDB_CONDITION_NOT_AFTER` can still be used to set the `--condition-not-after` timestamp by hand instead of obtaining it from CCDB.

The default object path for the time machine timestamp is `Users/a/aliprod/CCDBTimeMachineTS`. All objects are uploaded with validity `1:2`, and the metadata contains the pass, period, NotAfter and LPMProductionTag (to track the productions which uploaded the object). So there should be one object per reconstruction pass per period, valid for all runs of the given pass and period. Some test examples are already uploaded in http://ccdb-test.cern.ch:8080/browse/Users/e/ehellbar/CCDBTimeMachineTS.

This PR contains two commits:
1) in getCCDBTimeMachineTimestamp.py
- add a second CCDB query to check for existing objects right before the upload of a new object, to check if between the last empty query and now (after the estimation of the not-after timestamp from the reco output) another job was faster and already uploaded the object. This is an attempt to avoid duplicate objects for the same period/pass uploaded by a single LPMProductionTag. NB: a more elegent way to create these CCDBTimeMachineTS objects by default could be at the beginning or the end of the actual data reconstruction pass.
2) use getCCDBTimeMachineTimestamp.py in anchorMC.sh to obtain the CCDB time machine timestamp by default, keeping the current `ALIEN_JDL_CCDB_CONDITION_NOT_AFTER` option to enforce it by hand

I mark this as WIP since the following is still missing:
- treatment of MC specific CCDB objects: There is a handful of MC CCDB paths (TPC corrections, GLO/Param/MatLUT) for which the objects are uploaded independently of the data calibration objects, so the NotAfter as it is obtained here might invalidate the correct object to use. With Giulio, Ruben and Costin, we are working on an elegant way to handle this.
- I still need to run a small test on the grid to finally validate the implementation.

Author: ehellbar

MC/run/ANCHOR/anchorMC.sh

@@ -354,7 +354,29 @@ remainingargs="${ALIEN_JDL_ANCHOR_SIM_OPTIONS} ${remainingargs} --anchor-config
 ALIEN_JDL_O2DPG_ASYNC_RECO_FROMSTAGE=${ALIEN_JDL_O2DPG_ASYNC_RECO_FROMSTAGE:-RECO}
 remainingargs="${remainingargs} ${ALIEN_JDL_O2DPG_ASYNC_RECO_TAG:+--alternative-reco-software ${PWD}/env_async.env@${ALIEN_JDL_O2DPG_ASYNC_RECO_FROMSTAGE}}"
 # potentially add CCDB timemachine timestamp
-remainingargs="${remainingargs} ${ALIEN_JDL_CCDB_CONDITION_NOT_AFTER:+--condition-not-after ${ALIEN_JDL_CCDB_CONDITION_NOT_AFTER}}"
+if [[ -n ${ALIEN_JDL_CCDB_CONDITION_NOT_AFTER} ]]; then
+  # override CCDB time machine timestamp by hand, use e.g. 3385078236000 to effectively disable it
+  ARG_CONDITION_NOT_AFTER="--condition-not-after ${ALIEN_JDL_CCDB_CONDITION_NOT_AFTER}"
+else
+  # options to set CCDB url, disable upload and force overwrite of an existing timestamp; default to production CCDB, uploading the object and not overwriting
+  : ${CCDBTIMEMACHINE_CCDB_URL:=https://alice-ccdb.cern.ch}
+  [[ ${CCDBTIMEMACHINE_SKIP_UPLOAD:-} != "1" ]] && CCDBTIMEMACHINE_OPTIONS+=" --publish-to-ccdb"
+  [[ ${CCDBTIMEMACHINE_FORCE_OVERWRITE:-} == "1" ]] && CCDBTIMEMACHINE_OPTIONS+=" --force-overwrite"
+
+  # obtain CCDB time machine timestamp from CCDB or, if not available, estimate it from the creation time of the reconstruction output files
+  CCDBTIMEMACHINE_CMD="${O2DPG_ROOT}/GRID/utils/getCCDBTimeMachineTimestamp.py --ccdb-url $CCDBTIMEMACHINE_CCDB_URL ${CCDBTIMEMACHINE_OPTIONS:-}"
+  echo "Running command to obtain CCDB time machine timestamp: ${CCDBTIMEMACHINE_CMD}"
+  CCDBTIMEMACHINE_TS=$(${CCDBTIMEMACHINE_CMD})
+
+  # return an error if the timestamp could not be obtained
+  if [[ -n ${CCDBTIMEMACHINE_TS} ]]; then
+    ARG_CONDITION_NOT_AFTER="--condition-not-after ${CCDBTIMEMACHINE_TS}"
+  else
+    echo_error "Could not obtain CCDB time machine timestamp. Exiting."
+    exit 1
+  fi
+fi
+remainingargs="${remainingargs} ${ARG_CONDITION_NOT_AFTER:-}"
 
 echo_info "baseargs passed to o2dpg_sim_workflow_anchored.py: ${baseargs}"
 echo_info "remainingargs forwarded to o2dpg_sim_workflow.py: ${remainingargs}"