High Performance Computing for Deep Learning - NLDL 2023 Winter School

Important

• Some of the material here are adopted from HPC Carpentries https://carpentries-incubator.github.io/hpc-intro/

• Learners are expected to have basic UNIX knowledge as a prerequisite. We have prepared the following training for those who needs this https://training.pages.sigma2.no/tutorials/unix-for-hpc/

• This is a shortened version of the introduction course we deliver for NRIS user, you could access the full course here:

  • Introduction ful course: https://training.pages.sigma2.no/tutorials/hpc-intro/

  • GPUs full course: https://documentation.sigma2.no/getting_started/tutorials/gpu.html

  • Upcoming training events: https://documentation.sigma2.no/training/events.html

High Performance Computing for ML/AI (Sabry Razick (UiO/NRIS)

• Why use a Cluster?
• Working on a remote HPC system
• Accessing software
• Scheduling jobs

Monitoring GPU accelerated deep learning (Hicham AGueny, UiB,NRIS)

https://github.com/HichamAgueny/Profiling-GPU-accelerated-DL

• Overview of GPU Architecture (10 min)

• Basic tools for GPU usage (10 min)

• Profiling tools for Deep Learning (30)

• Demo on NRIS HPC system

Distributing Deep learning on HPC (Vetle Hofsøy-Woie, UiT, NRIS)

downloads/Distributing_Deeplearning_on_HPC.pdf

• Distributing Deep learning

• Why distribute(10 min)

• Types deep learning distribution(20 min)

  • Data distribution

  • Model distribution

• Example: Distribute a deep learning example (20)

Presenters

Sabry Razick: Sabry Razick received his PhD in information technology from University of Oslo (UiO). He is currently a chief engineer at UiO working on NRIS HPC systems and also acts as the manager of NAIC (Norwegian artificial intelligence cloud). https://no.linkedin.com/in/sabry-razick

Hicham Agueny: Hicham Agueny received his PhD in theoretical and computational physics and chemistry. He is currently a senior engineer in scientific computing at the UiB (IT-department). Previously, he worked as a researcher for about seven years at the UiB (physics department). His particular interest lies in heterogenous computing involving GPU-acceleration.

Vetle Hofsøy-Woie: Vetle Hofsøy-Woie is currently working on his master’s degree in computer science. Besides his master’s studies, Vetle is a senior engineer at UiT working in NRIS with GPU programming. During his studies, Vetle has had a specific interest in deep learning and has specialized in deep learning from a computer science perspective.