Add download from plougshare

[scarlehoff]

The pinecard in this case needs to be just an url to plougshare:

e.g., https://ploughshare.web.cern.ch/ploughshare/db/pinejet/pinejet-atlas-wm-arxiv-1109.5141/pinejet-atlas-wm-arxiv-1109.5141.tgz

with a metadata.txt and possibly a postrun.sh if there is something that needs to be modified to make it compatible with our data (e.g., fb -> pb or whatever)

[felixhekhorn]

maybe this is not a strict duplicate, but very close to #3

[scarlehoff]

Both things are necessary since we have the old grids, although #3 is actually quite well covered by @Radonirinaunimi's docs https://pinefarm.readthedocs.io/en/latest/run.html#converting-pre-existing-interpolation-tables

[scarlehoff]

ATLAS_2JET_7TEV_R06_M12Y https://ploughshare.web.cern.ch/ploughshare/record.php?dataset=applfast-atlas-dijets-v2-fc-fnlo-arxiv-1312.3524 @Radonirinaunimi

ATLAS_2JET_13TEV_DIF https://ploughshare.web.cern.ch/ploughshare/record.php?dataset=applfast-atlas-dijets-v2-fc-fnlo-arxiv-1711.02692 @andrpie

CMS_2JET_13TEV https://ploughshare.web.cern.ch/ploughshare/record.php?dataset=applfast-cms-dijets-v2-fc-fnlo-arxiv-2312.16669 NNPDF/nnpdf#2361 @andrpie

CMS_2JET_8TEV_3D https://ploughshare.web.cern.ch/ploughshare/record.php?dataset=applfast-cms-dijets-v2-fc-fnlo-arxiv-1705.02628 NNPDF/nnpdf#2418 @andrpie

CMS_2JET_7TEV_M12-Y https://ploughshare.web.cern.ch/ploughshare/record.php?dataset=applfast-cms-dijets-v2-fc-fnlo-arxiv-1212.6660 @andrpie

ideally, a pinecard with the dataset name and a plougshare.yaml with the link to plougshare and automatize download + conversion

1. Download + convert (pinefarm part)
2. Upload to theories_slim
3. Check chi2 with the grid and the different variants <--- by Amsterdam
4. Then create the fktable, then the fit etc

[achiefa]

Just a couple of comments regarding the grids on Ploughsahre for CMS dijet at 13 TeV.

1. For the 2D distribution, the last two bins in m_jj of the first grid (000) and the last bin in m_jj of the second grid (001) have to be trimmed to match the kinematics on Hepdata
2. For 3D distribution, the grids are 1D in m_jj and there is an independent grid for each combination of yb and ys. However, there are a couple of steps to be applied to the grids before they can be used. I suspect that the binning in yb and ys reported on Hepdata is wrong (possibly inverted). In the common data, the outer loop runs over yb, and then for each yb it loops over ys. On ploughshare, I have the impression that this is reversed.

I have produced a bash script (which I report below) that fixes the order, renames the grids appropriately, and adds another key to the 3D grids that contains information about the yb- and ys-binning. This should be included in the automation recently discussed. Feel free to change it accordingly, as I haven't tested it enough (it works on my machine).

Script

#!/bin/bash

ENV_NAME=$1

# conda activate is a shell function and requires conda.sh to be sourced first.
# Find the conda base directory, trying multiple common install locations.
if command -v conda &>/dev/null; then
    CONDA_BASE=$(conda info --base 2>/dev/null)
fi
if [ -z "$CONDA_BASE" ]; then
    for candidate in \
        "$HOME/miniforge3" \
        "$HOME/mambaforge" \
        "$HOME/miniconda3" \
        "$HOME/anaconda3" \
        "/opt/conda" \
        "/opt/miniforge3" \
        "/opt/miniconda3" \
        "/opt/anaconda3" \
        "/usr/local/miniconda3" \
        "/usr/local/anaconda3"
    do
        if [ -f "$candidate/etc/profile.d/conda.sh" ]; then
            CONDA_BASE="$candidate"
            break
        fi
    done
fi
if [ -z "$CONDA_BASE" ]; then
    echo "Error: Could not find conda installation" >&2
    exit 1
fi

# shellcheck source=/dev/null
source "$CONDA_BASE/etc/profile.d/conda.sh"
conda activate "$ENV_NAME"

# Create dir for adjusted grids
GRIDS_2D_PATH="CMS_2JET_13TEV_2D"
GRIDS_3D_PATH="CMS_2JET_13TEV_3D"
mkdir -p "$GRIDS_2D_PATH"
mkdir -p "$GRIDS_3D_PATH"

# Utility functions
delete_bins() {
    local bin_range="$1"
    local input="$2"
    local output="$3"
    pineappl write --delete-bins "$bin_range" "$input" "$output"
}

add_bin_info() {
    local bin_info="$1"
    local input="$2"
    pineappl write --set-key-value kin_bin "$bin_info" "$input" "${GRIDS_3D_PATH}/${input}"
}

# 1. Rename CMS_2JET_13TEV_00x_tabgz.pineappl.lz4 -> CMS_2JET_13TEV_2D_BINy.pineappl.lz4
for i in 0 1 2 3 4; do
    bin=$((i + 1))
    src="CMS_2JET_13TEV_00${i}_tabgz.pineappl.lz4"
    dst="CMS_2JET_13TEV_2D_BIN${bin}.pineappl.lz4"
    mv "$src" "$dst"
done

# 2. Wrapper function to delete bins from a grid
mv "CMS_2JET_13TEV_2D_BIN3.pineappl.lz4" "$GRIDS_2D_PATH/."
mv "CMS_2JET_13TEV_2D_BIN4.pineappl.lz4" "$GRIDS_2D_PATH/."
mv "CMS_2JET_13TEV_2D_BIN5.pineappl.lz4" "$GRIDS_2D_PATH/."
delete_bins 22-23 CMS_2JET_13TEV_2D_BIN1.pineappl.lz4 ${GRIDS_2D_PATH}/CMS_2JET_13TEV_2D_BIN1.pineappl.lz4
delete_bins 22    CMS_2JET_13TEV_2D_BIN2.pineappl.lz4 ${GRIDS_2D_PATH}/CMS_2JET_13TEV_2D_BIN2.pineappl.lz4

# 3. Rename 3D grids
mv "CMS_2JET_13TEV_005_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN1.pineappl.lz4"
mv "CMS_2JET_13TEV_010_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN2.pineappl.lz4"
mv "CMS_2JET_13TEV_014_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN3.pineappl.lz4"
mv "CMS_2JET_13TEV_017_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN4.pineappl.lz4"
mv "CMS_2JET_13TEV_019_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN5.pineappl.lz4"

mv "CMS_2JET_13TEV_006_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN1.pineappl.lz4"
mv "CMS_2JET_13TEV_011_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN2.pineappl.lz4"
mv "CMS_2JET_13TEV_015_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN3.pineappl.lz4"
mv "CMS_2JET_13TEV_018_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN4.pineappl.lz4"

mv "CMS_2JET_13TEV_007_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN3_ys_BIN1.pineappl.lz4"
mv "CMS_2JET_13TEV_012_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN3_ys_BIN2.pineappl.lz4"
mv "CMS_2JET_13TEV_016_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN3_ys_BIN3.pineappl.lz4"

mv "CMS_2JET_13TEV_008_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN4_ys_BIN1.pineappl.lz4"
mv "CMS_2JET_13TEV_013_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN4_ys_BIN2.pineappl.lz4"

mv "CMS_2JET_13TEV_009_tabgz.pineappl.lz4" "CMS_2JET_13TEV_3D_yb_BIN5_ys_BIN1.pineappl.lz4"

# 4. Add bin information to 3D grids
add_bin_info "yb<0.5 ys<0.5"             CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN1.pineappl.lz4
add_bin_info "yb<0.5 0.5<=ys<1.0"        CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN2.pineappl.lz4
add_bin_info "yb<0.5 1.0<=ys<1.5"        CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN3.pineappl.lz4
add_bin_info "yb<0.5 1.5<=ys<2.0"        CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN4.pineappl.lz4
add_bin_info "yb<0.5 2.0<=ys<2.5"        CMS_2JET_13TEV_3D_yb_BIN1_ys_BIN5.pineappl.lz4

add_bin_info "0.5<=yb<1.0 ys<0.5"        CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN1.pineappl.lz4
add_bin_info "0.5<=yb<1.0 0.5<=ys<1.0"   CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN2.pineappl.lz4
add_bin_info "0.5<=yb<1.0 1.0<=ys<1.5"   CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN3.pineappl.lz4
add_bin_info "0.5<=yb<1.0 1.5<=ys<2.0"   CMS_2JET_13TEV_3D_yb_BIN2_ys_BIN4.pineappl.lz4

add_bin_info "1.0<=yb<1.5 ys<0.5"        CMS_2JET_13TEV_3D_yb_BIN3_ys_BIN1.pineappl.lz4
add_bin_info "1.0<=yb<1.5 0.5<=ys<1.0"   CMS_2JET_13TEV_3D_yb_BIN3_ys_BIN2.pineappl.lz4
add_bin_info "1.0<=yb<1.5 1.0<=ys<1.5"   CMS_2JET_13TEV_3D_yb_BIN3_ys_BIN3.pineappl.lz4

add_bin_info "1.5<=yb<2.0 ys<0.5"        CMS_2JET_13TEV_3D_yb_BIN4_ys_BIN1.pineappl.lz4
add_bin_info "1.5<=yb<2.0 0.5<=ys<1.0"   CMS_2JET_13TEV_3D_yb_BIN4_ys_BIN2.pineappl.lz4

add_bin_info "2.0<=yb<2.5 ys<0.5"        CMS_2JET_13TEV_3D_yb_BIN5_ys_BIN1.pineappl.lz4
rm *.pineappl.lz4