Open Submission of Models for Large-Scale Benchmarking

Building upon the execution environment developed in García-Pedrajas et al. (2025), “A Massive Comparison of Classifiers for Real World Problems: An Incremental Approach” (submitted), we aim to open it to the broader research community by establishing a sustainable benchmarking platform based on Apptainer technology and deployed on our SLURM-based HPC cluster.

Researchers with representative models that have been validated through peer-reviewed publication are invited to submit their models for inclusion in the Large-Scale Benchmarking.
Once evaluated, we will provide the authors with a detailed performance report and publish the results on our project website.

Step-by-step instructions are provided below to help researchers build a single .sif file containing their model, along with guidelines for submission and evaluation.

This Apptainer-based approach ensures consistent execution and scientific reproducibility, independent of the original environment or programming language (e.g., C/C++, Java, Python, R, Go).

Once submitted, the .sif image will be executed on our SLURM-based HPC cluster for benchmarking.

Apptainer
Apptainer (formerly known as Singularity) is a containerization software for GNU/Linux systems that enables the encapsulation of complex software stacks within single-file container images (.sif file). Its primary goal is to ensure reproducibility of software execution across different machines from those where the software was originally developed, while also adhering to modern design principles such as security and performance.

Apptainer leverages several advanced features of the Linux kernel that have been progressively introduced in recent years to support secure, isolated, and flexible execution environments. These include FUSE (Filesystem in Userspace), Unprivileged User Namespaces, OverlayFS, and Control Groups (cgroups), which are foundational components of modern container technologies.

In addition, Apptainer is specifically designed for high-performance computing (HPC) environments. The software developer can tailor the container image to the needs of their application and build a single .sif file, making it straightforward to share and deploy computational workloads seamlessly across systems — from local workstations to large-scale HPC clusters.

Installing Apptainer
You will need a Linux system to run Apptainer. Other option is to try a Linux Virtual Machine. Recent versions of Windows 10 and Windows 11 include the Windows Subsystem for Linux (WSL2) that is tightly integrated with the Windows environment and it is straighforward to install Apptainer inside WSL2 (Ubuntu is the default Linux distribution in WSL2). Then you can install Apptainer from source or from Debian Packages (more on https://learn.microsoft.com/en-us/windows/wsl/install and https://apptainer.org/docs/admin/1.4/installation.html). Also Apptainer is available for Mac via Lima (https://lima-vm.io/). The default Linux distribution used by Lima for Apptainer template is Ubuntu.

Apptainer has a simple and robust command line interface and boasts excellent online documentation.
We can check the installed version:

    $ apptainer --version
    apptainer version 1.4.5

Apptainer has been created specifically for High Performance Computing (HPC) and scientific environments. Apptainer greatly facilitates scientific reproducibility by simplifying the creation and execution of containers ensuring software components are encapsulated for portability and reproducibility. Once the scientist creates the container with all the software components necessary to run their model, that container can be easily executed as a user without additional privileges on a SLURM cluster configured for that purpose.

To construct a reproducible containerized execution environment, the first step is to select a suitable base image from publicly available OCI-compliant container registries (e.g., Docker Hub). This base image serves as the foundation for constructing a fully self-contained execution environment, encapsulating all software dependencies required to ensure portability and reproducibility across heterogeneous computing infrastructures.

A minimal base container image can be easily built using, for example, the popular Alpine Linux distribution available on Docker Hub:

    $ apptainer build alpine.sif docker://alpine
    INFO:    Starting build...
    INFO:    Fetching OCI image...
    ... 
    INFO:    Creating SIF file...
    INFO:    Build complete: alpine.sif

Apptainer has been created specifi

$ apptainer build alpine.sif docker://alpine INFO: Starting build… INFO: Fetching OCI image… … INFO: Creating SIF file… INFO: Build complete: alpine.sif

Tutorial: A Decision Tree Example Using Scikit-learn in Python
For this tutorial we use a simple command-line Decision Tree classifier in Python using Scikit-learn. It accepts training and test datasets in ARFF format and trains a decision tree classifier. After training, the model generates predictions and class probabilities for the test set, which are written to output files (.preds and .probs). The source code is available at https://cib.uco.es/decisiontree/.

First step is to select an appropriate container image to serve as the foundation of the model’s execution environment. For this example, the container image python:3.x-slim is selected as the base runtime environment.This image is officially maintained by the Python project and distributed through Docker Hub. It provides a minimal Debian-based runtime environment with a specific Python 3.x interpreter preinstalled, while excluding non-essential packages in order to reduce image size. At the same time, it preserves compatibility with glibc, ensuring broad support for precompiled Python wheels (e.g., NumPy, SciPy, and Scikit-learn).

Rather than constructing the container image directly from the command line, we define it using an Apptainer definition file (https://apptainer.org/docs/user/latest/definition_files.html), in which all software dependencies required by the model are explicitly specified. The content of this file for our example is presented below:

    # Apptainer definition file: apptainer.def
    Bootstrap: docker
    From: python:3.10-slim

    %post
        # Install required Python dependencies
        pip install --upgrade pip    
        python -m pip install --no-cache-dir \
            numpy \
            scikit-learn \
            scipy \
            liac-arff \
            joblib

        # Create application directory inside the container
        mkdir -p /app

    %files
        # Copy the model script into the container
        decisiontree.py /app/decisiontree.py

    %runscript    
        # Default execution command when running:
        # apptainer run <image>.sif ...
        exec python3 /app/decisiontree.py "$@"

    %help    
        Container image for a Scikit-learn decision tree model.
        Usage:
        apptainer run sklearn-decisiontree.sif <trainfile> <testfile> <outputfile>

To create the SIF image, execute:

    $ apptainer build sklearn-decisiontree.sif apptainer.def

At this point, the newly created image containing our model (sklearn-decisiontree.sif) can be executed locally on any system where Apptainer is installed. For example, using the files glass-train.arff, glass-test.arff, and specifying glass-out as the output prefix, the container can be run as follows:

    $ apptainer run sklearn-decisiontree.sif glass-train.arff glass-test.arff glass-out

Files glass-out.preds and glass-out.probs will be generated in the current directory.

Once the the correct execution of the model using the Apptainer SIF image they created has been verified, the image can be submitted to the authors of this work for inclusion in the large-scale benchmarking framework.

Please fill out the following form to contact the authors and initiate the benchmarking process.