Restricted Queues

There are several queues on Frank which are restricted to use by only select groups. These queues run on hardware that was carried from a previous cluster, Fermi, which was purchased using CMMS funds.

If you are not a member of one of these CMMS research groups, you will not be able to submit a job to these queues. The error upon job submission will look something like the following:

All queues have a maximum walltime of 6 days and are not charged service units.

$ qsub -I -q kohn
qsub: Unauthorized Request  MSG=group ACL is not satisfied: user user@login0.frank.sam.pitt.edu, queue kohn

dist_amd

The dist_amd queue contains 28 AMD Interlagos nodes each with 64 cores, 128 GB (2 GB/core) of RAM and 2 TB of local scratch. Separate limitations on the number of cores are in place depending on the length of time requested.

Number of Nodes Max Walltime Max Processors in use per user
0 - 5 7 days 896 (14 nodes)
0 - 28 12 hours 1792 (28 nodes)

Users of this queue should add the following acknowledgement to their papers

Computational resources were also provided by the Center for Simulation and Modeling at the University of Pittsburgh and NSF MRI award 1229064.

shared_amd

The shared_amd queue contains 2 AMD Interlagos nodes each with 64 cores, 252 GB (3.9 GB/core) of RAM and 1826 GB of local scratch. The maximum number of cores in use per user is 64. The max walltime is 7 days.

Only one job is allowed to run on each node at a time.

Users of this queue should add the following acknowledgement to their papers

Computational resources were also provided by the Center for Simulation and Modeling at the University of Pittsburgh and NSF MRI award 1229064.

amd_compbio

The amd_compbio queue contains 24 AMD Interlagos nodes each with 64 cores and 2 TB of local scratch. 8 nodes have 126 GB of RAM and the remaining 16 nodes have 252 GB of RAM. Please use the -l mem flag to select the amount of memory required for the job.

  • The default mem value is 2600mb. The maximum is 252gb.

  • The default ddisk value is 38gb. The maximum is 1826gb.

  • The queue will only allow up to 48 cores to be used due to to these cores only having 32 floating-point cores.

Priorities in this queue are determined by membership in the Core quality of service. The quality of service is assigned automatically. The max walltime is 10 days.

QoS Destination queue Core Limits
Core amd_compbio_c Soft limit: 384
Hard Limit: 1008
Non-core amd_compbio_n Soft limit: 128
Hard Limit: 384

Users of this queue should add the following acknowledgement to their papers

Computational resources were also provided by the Center for Simulation and Modeling at the University of Pittsburgh and NSF MRI award 1229064.

jordan

The jordan queue runs on the Magny Cours nodes, and is only available to members of the Jordan research group.

There is no core limit per user on this queue.

Each node has 48 cores.

westmere

Jobs submitted to the westmere queue will run on nodes with the Intel Xeon (Westmere) CPUs with 12 cores each. There are 8 of these nodes, and all have 48 GB installed. Please refer to the Section System Architecture for more details on these westmere nodes (n163-n170).

There is no core limit per user on this queue

mem48g

Jobs submitted to the mem48g queue will run on nodes with the Intel Xeon (Nehalem) CPUs. All eight of these nodes have 8 cores, 48 GB of memory and 2 TB of local disk installed. This queue intended for jobs that require at least 4 GB of memory per core. Please refer to the Section System Architecture for more details on these mem48g Nehalem nodes (n59-n66).

  • There is no core limit per user on this queue
  • The maximum walltime is 2 days

mem24g

Jobs submitted to the mem24g queue will run on nodes with the Intel Xeon (Nehalem) CPUs. Each node has 8 cores. All forty of these nodes have 24 GB of memory and 1 TB of local disk installed. This queue intended for jobs that require at least 2 GB of memory per core. Please refer to the Section System Architecture for more details on these mem24g Nehalem nodes (n113-n152).

There is no core limit per user on this queue

ib

Jobs submitted to the ib queue will run on nodes with the Intel Xeon (Nehalem) CPUs. There are 92 of these nodes altogether, and they have a variable amount of memory installed. Each node has 8 cores. Jobs are limited to a maximum of 64 cores in this queue. Please refer to the Section System Architecture for more details on these ib nodes (67-112, 153-162, 171-176)

The core limit per user on this queue is 128.

one_day

Jobs submitted to the one_day queue will run on nodes with the Intel Xeon (Nehalem) CPUs. There are 90 nodes assigned to this queue, which spans nodes included in other Fermi queues and jobs must compete for resources. The maximum wall time for this queue is 24 hours.

Due to differences in memory and CPU speed between the nodes, jobs will only be run in the following sets. A single job cannot span nodes from different sets.

Node Set Number of nodes
ib 52
mem24g (A) 20
mem24g (B) 20
  • Nodes in the mem24g (A) node set are reserved for use by the Jordan group only.

The core limit per user on this queue is 128.