What is the DCC?

The Duke Compute Cluster is a general purpose high performance/high-throughput installation, and it is fitted with software used for a broad array of scientific projects. With a few notable exceptions, applications on the cluster are generally Free and Open Source Software.

Quick facts:

• 1360 nodes which combined have more than 45,000 vCPUs, 1,000 GPUs and 270TB of RAM

• Interconnects are 10 Gbps or 40 Gbps

• Utilizes a 7 Petabyte Isilon file system

• Runs Alma 9 and SLURM is the job scheduler

Cluster Appropriate Use

The DCC exists to support the research mission of Duke University, and users agree to only run jobs related to that mission.

Use of the cluster for any of the following is prohibited:

• Financial gain

• Commercial or business use

• Unauthorized use or storage of copyright-protected or proprietary resources

• Unauthorized mining of data on or off campus (including many web scraping techniques)

Sharing resources responsibly

The DCC is a shared environment and all users are expected to follow some basic best practices:

• All computational work must be submitted through the SLURM scheduler. Running jobs on the login nodes is considered an abuse of the system.

• Use common partition resources judiciously, only request the CPUs, memory, and time your job actually needs.

• Long running jobs are discouraged on the common and scavenger partitions. Save intermediate results (checkpoint) so an interrupted job does not mean starting over.

• Shared storage is for data under active computation, not long term storage. Clean up and remove your data when your computation concludes.

AI tools on the DCC

Data Security & Privacy

• Users are responsible for any data they introduce to the cluster and must follow all applicable Duke (including IRB), school, and departmental policies on data management and data use.

Data Classification Standard

Duke classifies all university data into three levels based on risk, and that classification determines where data may be stored and processed.

• Sensitive - data that Duke is either required by law to protect, or which Duke protects to mitigate institutional risk. Explicit institutional approval is needed to access Sensitive data.

• Restricted - data that is not necessarily for public consumption, but also does not fit into the Sensitive category. Disclosure would not significantly harm the university.

• Public - all other data, which can be accessible to the general public.

If data has not been classified, it defaults to Restricted, so when in doubt treat data as Restricted until its data steward tells you otherwise.

Classification examples

Data & the DCC

• The DCC’s security and compliance provisions are sufficient for Public and Restricted data only.

• Sensitive data (e.g. legally protected data such as PHI or FERPA) and data bound by certain restrictions in data use agreements are not allowed on the cluster.

• Appropriately de-identified or obfuscated data can potentially be used on the cluster, but confirm with your IRB protocol or data use agreement first.

• If your project involves Sensitive data it needs to live in an environment designed for it (e.g. the Protected Research Data Network), talk to your advisor and see Regulated Research Data Support.

• Before putting data on any Duke service, check the Duke Services and Data Classification chart for what is approved at each level.

Data & Other Services (AI)

• Duke’s data protections do not follow your data - Restricted or Sensitive data should only go into a third party service if Duke has a data protection agreement (or similar contract) in place with that vendor.

• A useful rule of thumb: if you did not log in with your NetID via Duke’s Shibboleth SSO, the service very likely does not have such an agreement.

• It is your responsibility to know what services you are using and what data you are sharing with them. Getting this wrong can have real consequences for both you and Duke.

• For example, uploading your PIs unpublished research data or student records to a personal ChatGPT or Claude account is not ok - use Duke’s AI tools (ChatGPT for Education, DukeGPT, the AI Gateway) instead, which are covered by Duke agreements.

Some Principals for AI use

• Data classification governs data handling

• Users are accountable for outputs they adopt

• Attribution follows the norms of the venue

• Credentials and authentication secrets stay out of Generative AI tools

• Agents and automated workflows require architectural constraints

• Understand third-party terms.

Storage usage

Since you’ve been added to the statdept group you will have access /hpc/group/statdept - this is a shared resource across the department use sparringly. If you have heavier needs you should be working with a PI - use their group’s allocation.

Accessing the DCC

All access to the DCC compute nodes is managed through the SLURM job scheduler.

Traditionally, users would log in to the cluster via SSH and then submit jobs to the scheduler.

This is still the preferred approach for large scale jobs but is beyond the scope of today’s workshop.

If this is of interest to you, I strongly recommend RC training.

Open OnDemand

For interactive use the current recommended approach is to use the OnDemand web-based interface.

• Open OnDemand gives access to tools like Jupyter Lab, RStudio, Positron, and VS Code on the DCC through your browser

• Currently there are two flavors of OnDemand available:

  • RC OnDemand - https://dcc-ondemand-01.oit.duke.edu/
  • Stats OnDemand - https://dcc-stat-ondemand-01.oit.duke.edu

Accessing OnDemand

Accounts & Partitions

Your account will belong to different groups and those groups will have access to different partitions on the DCC.

OnDemand generally will let you know which partitions you have access to, but this is a common reason a job may fail to run.

Limitations

Interactive partition:

• 10 CPUs, 64 GB memory, runtime <1 day

All other partitions (limit total for all jobs):

• 1.5 TB memory
• 400 CPUs
• 400 queued jobs

Important Job Options

Stats OnDemand has been customized and looks a little bit different but the core job details are basically the same.

After account and partition, the other important job options are:

• Session duration - how long the job will run before it is killed, better to pick longer than you think you need and then cancel the job if you finish early.

• Session size - how many CPUs and GBs of memory to ask for. Keep in mind partition limits and larger jobs may take longer to schedule. We’ve provided defaults that should work for most use cases, but custom values can be set under “Advanced Options”.

App Container

This is another advanced option - for all of the tools provided we have pre-built containers with a preconfigured environment that includes Python, R, and Julia (along with common packages).

The hope is that these will meet most of your needs, but you can also create your own containers if something more specialized is needed.

If you would like to explore this see RC Containers documentation for details or come chat with me.