Structure of SCITAS filesystems
- The structure and purpose of each filesystem is described here File systems
$WORKare shared across the site, while
$SCRATCHis local to each cluster
$SCRATCHautomatic deletion of files older than 2 weeks may happen without notice
- Production jobs should use
What to do when CPU time is significantly less than WALL time?
$SCRATCHis a GPFS parallel filesystem which is designed to perform well with parallel I/O
- In certain cases a big number of files is produced at runtime. Such I/O patterns put stress on the
$SCRATCHfilesystem metadata service, and are generally much slower than using a local disk.
- A long term solution would require changing the code to use external libraries like HDF5 or ADIOS. Those libraries give more flexibility in the way data is saved/handled.
- A workaround is to use the local filesystem
$TMPDIRis visible only once resources are allocated.
If you query the value of
$TMPDIRin a login node:
However, within a job allocation:
$SCRATCHare set by the SLURM scheduler while preparing the environment for each job
How to use the $TMPDIR in your simulations
- The following example show how to use the
$TMPDIRwith Quantum-ESPRESSO (QE).
- QE relies on fortran namelists to read certain parameters used during the simulation.
The only change that has to be done to a standard pw input is related to the outdir in the &CONTROL namelist. For example, in the input below the outdir is set equal to a placeholder 'fakeoutdir':
The submission script would look like:
After the sed command the CONTROL namelist looks like:
For a single 100GB file, all results in MB/s, <write into TMPDIR> : <copy from TMPDIR to /scratch>:
Cluster Architecture write into
Deneb E5v2 76 74 Deneb E5v3 109 103 Fidis E5v4 529 498