Announcements

23 February 2021

The next generation DAS, DAS-6, will be available shortly! DAS-6 is currently being installed and tested. It is expected to become available for regular use in March/April 2021.

May, 2016

IEEE Computer publishes paper about 20 years of Distributed ASCI Supercomputer. See the DAS Achievements page.

Special compute nodes

DAS-6 is built from single- and dual-socket compute nodes. The most common single-socket compute node has a 24-core AMD EPYC-2 (Rome) 7402P CPU. The most common dual-socket compute node has two 16-core AMD EPYC-2 (Rome) 7282 CPUs. All nodes have at least 128 GB main memory. In addition, several DAS-6 sites include non-standard node types for specific research purposes.

To get a quick overview of the queues (partitions) and available nodes on a site, use the "sinfo" command:

[fs0] sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite     32   idle node[001-027,030-034]
fatq         up   infinite      2   idle node[028-029]

Compute nodes are by default allocated from queue (partition) "defq" containing the standard node types. To allocate a special resource in SLURM or prun, a so-called "constraint" for a required node property should be specified as follows:

  • -C cpunode
    regular node type that only offers cpus
  • -C fatnode
    a cpu node with non-default CPU and often extra memory; typically this will be in a special partition to avoid mixing it with other node types.
  • -C gpunode
    regular node type that offers GPUs; the exact GPU types will be specified by other properties. Note that initially DAS-6 is not yet equipped with GPUs, but many nodes in fact are GPU-capable and the GPU status will change in due time.

This resource selector should be added as

#SLURM -C resource
in a SLURM job script, or passed as
-native '-C resource'
option to prun/preserve.

To allocate a GPU on a node, besides specifying the GPU type, the option "--gres=gpu:1" should be added as well. Examples can be found on the DAS-6 GPU page.

Nodes that have a different CPU or node architecture than the default are typically in a different queue (SLURM calls them "partitions") to avoid unpredictable performance. To run a job on node in partition "part", add the following:

#SLURM -p part
in a SLURM job script, or passed as
-native '-p part'
option to prun/preserve. When specifying multiple constraints or partitions, group them all together as argument to the single -native option in prun as follows:
-native '-p part -C resource1,resource2'

Vrije Universiteit Amsterdam (VU)

fs0.das6.cs.vu.nl has 32 regular single-socket GPU-capable nodes. In addition 2 of the nodes are fat nodes, with dual 24 cores and 2 TB memory

The following command provides details on partition names, sockets and cores per node, and features:

[fs0] sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite     32   idle node[001-027,030-034]
fatq         up   infinite      2   idle node[028-029]

[fs0] sinfo -e -o  "%9P %.6D %4X %4Y %24N %16f"
PARTITION  NODES SOCK CORE NODELIST                 AVAIL_FEATURES
defq*         32 1    24   node[001-027,030-034]    cpunode
fatq           2 2    24   node[028-029]            fatnode

Leiden University

fs1.das6.liacs.nl has 16 regular CPU nodes and 2 regular dual CPU nodes:

[fs1] sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite     16   idle node[101-116]
dual         up   infinite      2   idle node[117-118]

[fs1] sinfo -e -o  "%9P %.6D %4X %4Y %24N %16f"
PARTITION  NODES SOCK CORE NODELIST                 AVAIL_FEATURES
defq*         16 1    24   node[101-116]            cpunode
dual           2 2    16   node[117-118]            cpunode

University of Amsterdam (UvA), SNE cluster

fs2.das6.science.uva.nl has 8 regular dual-socket CPU nodes, 4 of which have a large number of disks for specific IO experiments. These nodes are in fatq.

[fs2] sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
fatq         up   infinite      4   idle node[201-204]
defq*        up   infinite      4   idle node[205-208]

[fs2] sinfo -e -o  "%9P %.6D %4X %4Y %24N %16f"
PARTITION  NODES SOCK CORE NODELIST                 AVAIL_FEATURES
fatq           4 2    16   node[201-204]            fatnode
defq*          4 2    16   node[205-208]            cpunode

Delft University of Technology

fs3.das6.tudelft.nl will have 16 regular dual-socket CPU nodes and 4 dual-socket GPU-capable nodes

University of Amsterdam (UvA), AI cluster

fs4.das5.science.uva.nl has 12 regular GPU-capable nodes, each with two NVMe disks.

[fs4] sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite     12   idle node[401-412]

[fs4] sinfo -e -o  "%9P %.6D %4X %4Y %24N %16f"
PARTITION  NODES SOCK CORE NODELIST                 AVAIL_FEATURES
defq*         12 1    24   node[401-412]            cpunode

ASTRON

fs5.das6.astron.nl has 4 regular dual-socket GPU-capable nodes and a dual-socket (2*64 cores) fat node:

[fs5] sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite      4   idle node[501-504]
fatq         up   infinite      1   idle node505

[fs5] sinfo -e -o  "%9P %.6D %4X %4Y %24N %16f"
PARTITION  NODES SOCK CORE NODELIST                 AVAIL_FEATURES
defq*          4 2    16   node[501-504]            cpunode
fatq           1 2    64   node505                  fatnode