Course Syllabus

Syllabus

CS 6960 Multimodal LLM Agents

     
  Instructor: Kenneth Marino Credit Hours: 3.0
  Department: Kahlert School of Computing Semester: Spring 2026
  Office: MEB 2178 Type: In Person
  Days / Times: TuTh / 03:40PM - 05:00PM Location: WEB L110
   
  Communication & Office Hours  Review the "Communication" section below for more information.
   

Course Description

Overview

Course CS 6960 Multimodal LLM Agents
Department Kahlert School of Computing
Credit Hours 3.0
Semester Spring 2026
Days / Times TuTh / 03:40PM - 05:00PM
Location: WEB L110
Description

This course will explore the rapidly developing area of Multimodal Large Language Models in embodied settings. Students will learn the foundations of Reinforcement Learning and Large Language Models to understand how large-scale models can be deployed to multimodal environments. Topics will include control flow and scaffolding for agents such as ReAct, Tool use LLMs, coding agents, game-playing agents, computer use and robotics. As part of the course, students will complete programming assignments, read recent papers in the field and complete a significant project. While there are no formal prerequisites, students are expected to be familiar with Machine Learning and the basics of Natural Language Processing (CS 5340) before taking the course.

Goals & Objectives

At the end of this course, students will be able to:

  • Understand the underlying basis of VLM agents: Reinforcement Learning, LLM, VLMs and agent frameworks
  • Have a broad understanding of different topics and applications of agents including: prompting frameworks, evaluation, tool use, RAG, coding agents, computer use, robotics and more.
  • Be able to find and develop interesting new areas of research to explore in agents.
  • Complete a novel project in agents using the skills and knowledge gained from the class.

Materials

There is no official textbook for the class. Selected readings and coding assignments will be used from the HuggingFace AI Agents Course: https://huggingface.co/learn/agents-course/en/unit0/introduction

Communication

Instructor

Kenneth Marino
Office: MEB 2178
Email: kenneth.marino@utah.edu
Office Hours: Mon 3pm-4pm 

TAs

Syeda Mishra Saiara
Email: Syeda.Saiara@utah.edu
Office Hours: Wed 2-3pm, MEB 3105

 

Mahnoor Saad
Email: mahnoor.saad@utah.edu
Office Hours: Thur 10am-11am, MEB 3105

 

Preferred Contact Methods

For questions about homework, post first on piazza (or see if your question has been asked already) and/or come to office hours.

For questions about grades on homework/quizzes/exams, please TAs during office hours.

Evaluation

Your performance in this course will be evaluated by:

  • 30% Class Presentations
  • 10% Homework Assignments
  • 5% Participation
  • 20% Intermediate Project Reports
  • 35% Final Project Report and Presentation

Schedule

Week Date Topic Homework/Project Recommended Readings
1
 Tue, Jan 6 Course Overview / Logistics
Thur, Jan 8 RL Basics Sutton Barto Ch 1, 3
2
 Tue, Jan 13 LLM Basics
Thur, Jan 15 VLM Basics HW0 Due
3
 Tue, Jan 20 Agent Frameworks
Thur, Jan 22
4
 Tue, Jan 27 RAG
Thur, Jan 29 HW1 - LLM Frameworks Due
5
 Tue, Feb 3 Tool Use
Thur, Feb 5
6
 Tue, Feb 10

Coding Agents
Thur, Feb 12 HW2 - RAG agent Due
7
 Tue, Feb 17 Coding Agents (Cont)
Thur, Feb 19 (Optional) HW3 - Assistant Agent
8 Tue, Feb 24 Evaluation
Thur, Feb 26 Project Proposals Due
9 Tue, March 3 Assistant Agents
Thur, March 5
10
 Tue, March 9 Spring Break
Thur, March 12
11
 Tue, March 17 Game Agents
Thur, March 19 Project Report 1
12
 Tue, March 24 Computer Use
Thur, March 26
13
 Tue, March 31 Computer Use (Cont.)
Thur, April 2 Project Report 2
14
 Tue, April 7 Robotics
Thur, April 9
15
 Tue, April 14 Robotics (Cont.)
Thur, April 16
16
 Tue, April 21 Final Project Presentations Final Report Due
Thur April 23 (3:30-5:30) Final Project Presentations

Recommended Readings / Links

RL:

Sutton Barto Textbook: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf 

Course Policies

Grade Disputes

Feedback on graded material will be posted on Canvas in as timely a manner as possible. Once feedback for a graded assignment is posted, you will have 1 week from the posting date to dispute a grade. No re-grade requests will be honored after 1 week from posting feedback. Grade changes will not be made at the end of the semester.

Collaboration Policy

You are encouraged to discuss course materials and reading assignments, and homework assignments with each other in small groups (two to three people). You must list all discussants in your homework write-up. Discussion about homework assignments may include brainstorming and verbally discussing possible solution approaches, but must not go as far as one person telling others how to solve a problem. In addition, you must write-up your solutions by yourself, and you may not look at another student’s homework write-up/solutions (whether partial or complete). The course project is expected to be done as a group. Groups will be formed and have time to meet and discuss during class.

AI Policy

AI use is allowed and encouraged as a study tool: generating practice problems, asking to explain concepts. However, it is not allowed to be used to complete homework assignments. Any use of AI to generate homework solutions will results in a 0 for the assignment.
AI use is allowed for projects, as long as any use is clearly documented in project reports. Avoid use of LLMs for writing reports, but they may be used for helping find spelling/grammar/clarity issues.

Grading

Grades will be determined based on correctness and relevance to the assignments and questions. Pay close attention to the instructions and rubrics provided for each assignment/task.

Extra credit opportunities may be available throughout the course. These will be clearly communicated and typically involve additional research, attending relevant events, or completing supplementary assignments. Extra credit can contribute up to 5% additional points to the final grade.

Grading scale

Letter Scoring
A 100% - 93%
A- 92.9% - 90%
B+ 89.9% - 87%
B 86.9% - 83%
B- 82.9% - 80%
C+ 79.9% - 77%
C 76.9% - 73%
C- 72.9% - 70%
D 69.9% - 60%
E 59.9% - 0%

Accommodations

Disclaimer

Accommodations will be considered on an individual basis and may require documentation.

Please contact your instructor as soon as possible (preferably shortly before the semester begins) to request accommodations of any kind.

Content Warnings

Please be aware that some materials and discussions within this course may contain challenging content. Your instructor may choose to notify students of potentially difficult content (e.g. explicit language, graphic images, violent themes, etc.) throughout the course.

If there are specific subjects that you need advanced notice for, please contact your instructor at the beginning of the semester.

Extreme personal circumstances

Please contact your instructor as soon as possible if an extreme personal circumstance
(hospitalization, death of a close relative, natural disaster, etc.) is interfering with your ability to
complete your work.

Religious Practice

To request an accommodation for religious practices, contact your instructor at the beginning of the semester.

Active Duty Military

If you are a student on active duty with the military and experience issues that prevent you from participating in the course because of deployment or service responsibilities, contact your instructor as soon as possible to discuss appropriate accommodations.

Disability Access

All written information in this course can be made available in an alternative format with prior notification to the Center for Disability Services (CDS). CDS will work with you and the instructor to make arrangements for accommodations. Prior notice is appreciated. To read the full accommodations policy for the University of Utah, please see Section Q of the Instruction & Evaluation regulations.

If you will need accommodations in this class, contact:

Center for Disability Services
801-581-5020
disability.utah.edu
162 Union Building
    200 S. Central Campus Dr.
     Salt Lake City, UT 84112

Changes to the Syllabus

This syllabus is not a contract. It is meant to serve as an outline and guide for your course. Please note that your instructor may modify it to accommodate the needs of your class.

You will be notified of any changes to the Syllabus.

University Policies

Americans with Disabilities Act (ADA)

The University of Utah seeks to provide equal access to its programs, services, and activities for people with disabilities.

All written information in this course can be made available in an alternative format with prior notification to the Center for Disability & Access (CDA). CDA will work with you and the instructor to make arrangements for accommodations. Prior notice is appreciated. To read the full accommodations policy for the University of Utah, please see Section Q of the Instruction & Evaluation regulations.

In compliance with ADA requirements, some students may need to record course content. Any recordings of course content are for personal use only, should not be shared, and should never be made publicly available. In addition, recordings must be destroyed at the conclusion of the course.

If you will need accommodations in this class, or for more information about what support they provide, contact:

Center for Disability & Access

  801-581-5020
  disability.utah.eduLinks to an external site.
 Third Floor, Room 350
 Student Services Building
 201 S 1460 E
 Salt Lake City, UT 84112

Safety at the U

The University of Utah values the safety of all campus community members. You will receive important emergency alerts and safety messages regarding campus safety via text message. For more safety information and to view available training resources, including helpful videos, visit safeu.utah.edu.

To report suspicious activity or to request a courtesy escort, contact:

Campus Police & Department of Public Safety

  801-585-COPS (801-585-2677)
  dps.utah.edu
  1735 E. S. Campus Dr.
  Salt Lake City, UT 84112

Addressing Sexual Misconduct

Title IX makes it clear that violence and harassment based on sex and gender (which includes sexual orientation and gender identity/expression) is a civil rights offense subject to the same kinds of accountability and the same kinds of support applied to offenses against other protected categories such as race, national origin, color, religion, age, status as a person with a disability, veteran’s status, or genetic information.

If you or someone you know has been harassed or assaulted, you are encouraged to report it to university officials: 

Title IX Coordinator & Office of Equal Opportunity and Affirmative Action

  801-581-8365
  oeo.utah.edu
  135 Park Building
  201 Presidents' Cir.
  Salt Lake City, UT 84112

Office of the Dean of Students

  801-581-7066
  deanofstudents.utah.edu
  270 Union Building
  200 S. Central Campus Dr.
  Salt Lake City, UT 84112

To file a police report, contact:

Campus Police & Department of Public Safety

  801-585-COPS (801-585-2677)
  dps.utah.edu
  1735 E. S. Campus Dr.
  Salt Lake City, UT 84112

If you do not feel comfortable reporting to authorities, the U's Victim-Survivor Advocates provide free, confidential, and trauma-informed support services to students, faculty, and staff who have experienced interpersonal violence.

To privately explore options and resources available to you with an advocate, contact:

Center for Student Wellness

  801-581-7776
  wellness.utah.edu
 350 Student Services Building
 201 S. 1460 E.
 Salt Lake City, UT 84112

Academic Misconduct

It is expected that students comply with University of Utah policies regarding academic honesty, including but not limited to refraining from cheating, plagiarizing, misrepresenting one’s work, and/or inappropriately collaborating. This includes the use of generative artificial intelligence (AI) tools without citation, documentation, or authorization. Students are expected to adhere to the prescribed professional and ethical standards of the profession/discipline for which they are preparing. Any student who engages in academic dishonesty or who violates the professional and ethical standards for their profession/discipline may be subject to academic sanctions as per the University of Utah’s Student Code: Policy 6-410: Student Academic Performance, Academic Conduct, and Professional and Ethical Conduct.

Plagiarism and cheating are serious offenses and may be punished by failure on an individual assignment, and/or failure in the course. Academic misconduct, according to the University of Utah Student Code:

“...Includes, but is not limited to, cheating, misrepresenting one’s work, inappropriately collaborating, plagiarism, and fabrication or falsification of information…It also includes facilitating academic misconduct by intentionally helping or attempting to help another to commit an act of academic misconduct.”

For details on plagiarism and other important course conduct issues, see the U's Code of Student Rights and Responsibilities.