Compute and storage quota

Objectives

  • Understand what quota is

  • Be aware about how quota is consumed

  • Know what issues may arise from exceeding the allocated quota

  • Be able to resolve issues revolving from exceeding quotas

  • Be able to use quota effictively by planning ahead

Demo/discussion time

We have great documentation about compute and storage quota and there is no point repeating it here. Let’s go through the documentation and examples together:

Big-picture overview of storage areas

We also have a more detailed overview page but here is the big picture:

  • /cluster/home/

    • very limited quota

    • for scripts, source code, small data files

  • /cluster/projects/

    • much more quota

    • run your calculations here

    • keep your conda environments here

  • /cluster/work/

    • basically unlimited

    • files older than 21 days are automatically deleted

    • good for temporary files and testing

Instructor note

  • Teaching: 20 min

  • Discussion: 10 min

Demonstrating going beyond space quota:

  • Instructor shows dusage directly on the cluster. The reason is that the output is colored and we can then also try to troubleshoot problems together.

  • The instructor prepares a situation by creating a big file before the course which exceeds quota so that we can see how error messages look and also together discuss how to solve the problem. This can be done for instance using head -c 1GB /dev/urandom > large-file.bin

  • On Saga and Fram it can be interesting to demonstrate that moving the files may not be enough. This can be demonstrated with moving, followed by a comparison of du vs. dusage mismatch if group permissions have not been adjusted.

Demonstrating going beyond inode quota:

  • The instructor can use the following environment.yml file:

    name: myproject
channels:
  - defaults
dependencies:
  - python=3.10
  - numpy
  - pandas
  - scipy
  - matplotlib

  • Check which version of mamba is available on the system and adjust the script below accordingly:

    $ module avail mamba

  • And the following install script:

    #!/usr/bin/env bash

# settings to catch errors in bash scripts
set -euf -o pipefail

module load Mamba/23.11.0-0

# this is not a good setting but we use it to demonstrate a problem
my_conda_storage=${HOME}/tmp/conda

export CONDA_PKGS_DIRS=${my_conda_storage}/package-cache
mamba env create --prefix ${my_conda_storage}/myproject --file environment.yml

  • This hopefully installs in seconds.