This course is an introduction to high-performance computing. High-performance computing (HPC) is the practice of leveraging parallel computing power, from computer clusters and supercomputers, to solve complex computational problems faster. In this course students will learn the main programming models, such as openMP and MPI, to write highly optimized code for modern multi-core processors, clusters, and modern computing architecture (e.g., GPUs). Furthermore, this course will explore the concepts and challenges of performance scalability, portability, and energy efficiency of HPC applications and systems.

Lecturer: Dr. Steve Petruzza TA: Parker Bywater

Topics include: multithread programming, openMP, distributed programming, MPI, performance profiling, parallel and distributed algorithms, parallel I/O, in situ analytics, GPU programming.

Prerequisites: a good C/C++ programming background. Most of the assignment will be in C/C++. In particular, a student needs to have knowledge of data structures, arrays, pointers and memory allocation in C/C++. If you are not too familiar with C/C++ here is a tutorial that could be useful, the “C++ basic” section is a good place to start. If you are not familiar with the Unix terminal:

• you can try it online or install it on your machine (see here )
• some basic commands that you can use can be found here
• if you use Windows you can install a Linux subsystem following this guide here.

Resources

• [IPP] An Introduction to Parallel Programming. Peter Pacheco
• [UMPI] Using MPI: Portable Parallel Programming with the Message-Passing Interface. Anthony Skjellum, William Gropp, and Ewing Lusk
• [GPU] Programming Massively Parallel Processors, 2nd Edition. David B. Kirk, Wen-mei W. Hwu
• [CHPC] Accessing CHPC Resources

Gradings

• Homework (30%), 5 assignments
• Mid-term exam (25%)
• Final exam (25%)
• Project (20%)

Class schedule

Day	Topic	References	Homework
Aug 31	Introduction	IPP Ch. 1
Sep 2	Computing architecture and memory	IPP Ch. 2
Sep 7	Parallel hardware and interconnects	IPP Ch. 2
Sep 9	Parallel software and scalability	IPP Ch. 2	HW1
Sep 14	Shared memory programming: PThreads	IPP Ch. 4
Sep 16	Shared memory programming: PThreads (2)	IPP Ch. 4
Sep 21	Shared memory programming: PThreads (3)	IPP Ch. 4
Sep 23	C++11 Threads and running on an HPC system	CHPC	HW2
Sep 28	OpenMP (1)	IPP Ch. 5
Sep 30	OpenMP (2)	IPP Ch. 5
Oct 5	OpenMP (3)	IPP Ch. 5
Oct 7	Introduction to GPU Programming and CUDA	GPU Ch.1,2,3	HW3
Oct 12	Review and GPU programming (2)	GPU Ch.4
Oct 14	Mid-term exam		Mid-term
Oct 19	GPU: execution model	GPU Ch. 4
Oct 21	GPU: memory model	GPU Ch.5
Oct 26	GPU: tiling and matrix multiplication	GPU Ch.6,7
Oct 28	GPU: convolution	GPU Ch.8	HW4
Nov 2	GPU: convolution analysis	GPU Ch.9
Nov 4	GPU: deep learning
Nov 9	Distributed programming: basics	IPP Ch. 3	HW5
Nov 11	Distributed programming: MPI	IPP Ch. 3
Nov 16	MPI (2)	IPP Ch. 3
Nov 18	MPI Parallel I/O (3)	IPP Ch. 3
Nov 23	Parallel I/O, Applications
Nov 25	Thanksgiving
Nov 30	Python for HPC
Dec 2	Profiling
Dec 7	Review
Dec 9	Review and Project presentations
Dec 14	No class.
Dec 16	Final exam		Final

The schedule above is subject to change

Assignments

• You will submit your assignment using the Canvas System. Assignment is due by 11:59 p.m. on the due date. Late assignment will be accepted for a maximum of 2 days beyond the submission deadline, a late penalty of 10% for each day will be applied (max 2 days).
• You are required to do your own work. You will be responsible to ensure that your assignment is submitted correctly. If you have multiple files to submit, you need to zip multiple files into one file before submission. Please verify the information in the confirmation email. If the assignment is submitted incorrectly, you can resubmit it. The system keeps track of the last file submitted for each assignment. If you do not get the confirmation information within a few minutes, you can assume the assignment has not been submitted.
• Homeworks, mid-term and final exams are a mix of quiz, open questions and short programming assignments. Most of the programming assignments are in C/C++.
• At the top of each assignment, please include comments that consist of your name, your contact email, and assignment name or number. For programming assignments please clearly add comments for each important variable and section.

Projects

Selection/proposal deadline: TBD, 2021

Submission deadline: TBD, 2021

Grading Scheme

A [93%, 100%] A- [87%, 93%) B+ [81%, 87%) B [75%, 81%) B- [70%, 75%) C+ [65%, 70%) C [60%, 65%) C- [58%, 60%) D+ [55%, 58%) D [53%, 55%) F Below 53%

Grading Policy and Grades

• Each programming assignment is graded based on the following factors: code efficiency, code structure, technique difficulty, code readability, etc.
• Your scores for each assignment and exam will be available through the Canvas System. You will be shown each individual score, your percent to date, and any comments. You will be notified via email whenever you receive a score or when a change is made to a score.
• If you want to dispute a score, contact the TA within two days of having received the original score. If the disagreement is not resolved after communicating with the TA, contact me on Canvas. I will review your submission, talk to the TA and then respond to you. I will not consider changes in scores if you do not contact me within one week of when the score is sent to you.

Important dates

• Late Adds: The last day to add this class is September 21 (Monday). Attending this class beyond that date, without being officially registered, will not be approved by the Dean’s Office. Students must be officially registered for this course. No assignments or tests of any kind will be graded for students whose names do not appear on the class list.
• Drop Date: The last day to drop classes is:
  1. Monday, September 21 - without a “W” notation on transcript.
  2. Monday, November 2 - with a “W” notation on transcript.

Withdrawal Policy, “I” Grade Policy, and Dropping Courses

If a student does not attend a class during the first week of the term or by the second class meeting, whichever comes first, the instructor may submit a request to have the student dropped from the course. (This does not remove responsibility from the student to drop courses which they do not plan to attend.) Students who are dropped from courses will be notified by the Registrar’s Office through their preferred e-mail account. Students may drop courses without notation on the permanent record through the first 20 percent of the class. If a student drops a course following the first 20 percent of the class, a W will be permanently affixed to the student’s record (check General Catalog for exact dates). Students with extenuating circumstances should refer to the policy regarding Complete Withdrawal from the University and the Incomplete (I) Grade policy in the General Catalog.

COVID-19 Classroom Safety Protocols

While not mandated, USU encourages and welcomes the wearing of masks in all university building, especially within 6 feet of others. Furthermore, it is strongly encouraged to take measures to mitigate risk as recommended by federal and state public health officials. These measures include getting fully vaccinated, staying home if you are sick (even with mild symptoms), and maintaining good hygiene including frequent hand washing. Testing will be provided, without charge, throughout the semester and the USU COVID Webpage will provide up-to-date information. Please remember; COVID can have significant impact on the health and safety of those around you so remain vigilant and respectful.

Academic Freedom and Professional Responsibilities

Academic freedom is the right to teach, study, discuss, investigate, discover, create, and publish freely. Academic freedom protects the rights of faculty members in teaching and of students in learning. Freedom in research is fundamental to the advancement of truth. Faculty members are entitled to full freedom in teaching, research, and creative activities, subject to the limitations imposed by professional responsibility. USU Policy 403 further defines academic freedom and professional responsibilities.

Academic Integrity - “The Honor System”

The University expects that students and faculty alike maintain the highest standards of academic honesty. The Code of Policies and Procedures for Students at Utah State University (Student Conduct) addresses academic integrity and honesty and notes the following: Academic Integrity: Students have a responsibility to promote academic integrity at the University by not participating in or facilitating others’ participation in any act of academic dishonesty and by reporting all violations or suspected violations of the Academic Integrity Standard to their instructors. The Honor Pledge: To enhance the learning environment at Utah State University and to develop student academic integrity, each student agrees to the following Honor Pledge: “I pledge, on my honor, to conduct myself with the foremost level of academic integrity”. Violations of the Academic Integrity Standard (academic violations) include, but are not limited to cheating, falsification, and plagiarism.

Plagiarism

Plagiarism includes knowingly “representing by paraphrase or direct quotation, the published or unpublished work of another person as one’s own in any academic exercise or activity without full and clear acknowledgment. It also includes the unacknowledged use of materials prepared by another person or agency engaged in the selling of term papers or other academic materials.” The penalties for plagiarism are severe. They include warning or reprimand, grade adjustment, probation, suspension, expulsion, withholding of transcripts, and denial or revocation of degrees.

Grievance Process

Students who feel they have been unfairly treated [in matters other than discipline, admission, residency, employment, traffic, and parking - which are addressed by procedures separate and independent from the Student Code] may file a grievance through the channels and procedures described in the Student Code: Article VII Grievances

Discrimination and Sexual Misconduct

USU strives to provide an environment for students and employees that is free from discrimination and sexual misconduct. If you experience sexual misconduct or discrimination at any point during the semester inside or outside of class, you are encouraged to contact the USU Title IX Coordinator via Old Main room 161 in Logan, 435-797-1266, titleix@usu.edu, or at equity.usu.edu/report. You can learn more about the USU resources available for individuals who have experienced sexual misconduct at sexualrespect.usu.edu. Resources for individuals who have experienced discrimination are listed at equity.usu.edu. Required Reporting of Sexual Misconduct The instructor is designated by USU as a “reporting employee.” This means that if you share information about sexual misconduct (sexual harassment, sexual assault, relationship violence, or sex-based stalking) with the instructor, they will report that information to the USU Title IX Coordinator. The instructor is also required to tell you about designated confidential resources, supportive measures, and how you can file a report with the USU Title IX Coordinator.

Students with Disabilities

USU welcomes students with disabilities. If you have, or suspect you may have, a physical, mental health, or learning disability that may require accommodations in this course, please contact the Disability Resource Center (DRC) as early in the semester as possible (University Inn # 101, 435‐797‐2444, drc@usu.edu). All disability related accommodations must be approved by the DRC. Once approved, the DRC will coordinate with faculty to provide accommodations. Withdrawal Policy, “I” Grade Policy and Dropping Courses If a student does not attend a class during the first week of the term or by the second class meeting, whichever comes first, the instructor may submit a request to have the student dropped from the course. (This does not remove responsibility from the student to drop courses which they do not plan to attend.) Students who are dropped from courses will be notified by the Registrar’s Office through their preferred e-mail account. Students may drop courses without notation on the permanent record through the first 20 percent of the class. If a student drops a course following the first 20 percent of the class, a W will be permanently affixed to the student’s record (check General Catalog for exact dates). Students with extenuating circumstances should refer to the policy regarding Complete Withdrawal from the University and the Incomplete (I) Grade policy in the General Catalog.

Assumption of Risk

All classes, programs, and extracurricular activities within the University involve some risk, and some involve travel. The University provides opportunities to participate in these programs on a voluntary basis. Therefore, students should not participate in them if they do not care to assume the risks. Students can ask the respective program leaders/sponsors about the possible risks a program may generate, and if students are not willing to assume the risks, they should not select that program. By voluntarily participating in classes, programs, and extracurricular activities, students do so at their own risk. General information about University Risk Management policies, insurance coverage, vehicle use policies, and risk management forms can be found at http://www.usu.edu/riskmgt/.

Mental Health

Mental health is critically important for the success of USU students. As a student, you may experience a range of issues that can cause barriers to learning, such as strained relationships, increased anxiety, alcohol/drug problems, feeling down, difficulty concentrating and/or lack of motivation. These mental health concerns or stressful events may lead to diminished academic performance or reduce your ability to participate in daily activities. Utah State University provides free services for students to assist them with addressing these and other concerns. You can learn more about the broad range of confidential mental health services available on campus at Counseling and Psychological Services (CAPS). Students are also encouraged to download the “SafeUT App” to their smartphones. The SafeUT application is a 24/7 statewide crisis text and tip service that provides real-time crisis intervention to students through texting and a confidential tip program that can help anyone with emotional crises, bullying, relationship problems, mental health, or suicide related issues.