Course Schedule

This is a tentative schedule. Specific readings are subject to change as new material is published.

Date	Topic
08/28	Introduction and Course Overview Lecture slides: Google Slides
09/02	Labor Day
09/04	Overview: Making of LLMs Lecture slides: Introduction to Language Modeling.pptx, LLama-3 Paper Submit your review before 09/03/2024 11:59 PM. The Llama 3 Herd of Models Language Models are Few-Shot Learners (GPT-3)
09/09	Inference System - Memory Submit your review before 09/07/2024 11:59 PM. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) SGLang: Efficient Execution of Structured Language Model Programs
09/11	Inference System - Scheduling Submit your review before 09/09/2024 11:59 PM. Orca: A Distributed Serving System for Transformer-Based Generative Models Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
09/16	Speculative Decoding Submit your review before 09/14/2024 11:59 PM. Accelerating Large Language Model Decoding with Speculative Sampling Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
09/18	Benchmark and Evaluation Submit your review before 09/16/2024 11:59 PM. Measuring Massive Multitask Language Understanding (MMLU) Proving Test Set Contamination in Black Box Language Models
09/23	Retrieval Submit your review before 09/21/2024 11:59 PM. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (HNSW)
09/25	Agents Submit your review before 09/23/2024 11:59 PM. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines MemGPT: Towards LLMs as Operating Systems
09/30	Project Proposal
10/02	Project Proposal
10/07	Quantization Submit your review before 10/05/2024 11:59 PM. FP8 Quantization: The Power of the Exponent AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration Mixed Precision Training GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
10/09	DL Compiler and Hardware Submit your review before 10/07/2024 11:59 PM. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
10/14	Data Submit your review before 10/12/2024 11:59 PM. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
10/16	Optimization and Scaling Submit your review before 10/14/2024 11:59 PM. Scaling Laws for Neural Language Models Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (μP)
10/21	Training Kernels Submit your review before 10/19/2024 11:59 PM. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Ring Attention with Blockwise Transformers for Near-Infinite Context
10/23	Training Systems - Model Parallelism Submit your review before 10/21/2024 11:59 PM. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
10/28	Training System - Optimizers Submit your review before 10/26/2024 11:59 PM. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models PipeDream: Generalized Pipeline Parallelism for DNN Training
10/30	Inference-time Compute Submit your review before 10/28/2024 11:59 PM. Let’s Verify Step by Step Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
11/04	Alignment Submit your review before 11/02/2024 11:59 PM. Training language models to follow instructions with human feedback (OpenAI RLHF) Direct Preference Optimization: Your Language Model is Secretly a Reward Model
11/06	Parameter-efficient Finetuning Submit your review before 11/04/2024 11:59 PM. LoRA: Low-Rank Adaptation of Large Language Models S-LoRA: Serving Thousands of Concurrent LoRA Adapters
11/11	Veterans Day
11/13	Knowledge Distillation Submit your review before 11/11/2024 11:59 PM. Textbooks Are All You Need Gemma 2: Improving Open Language Models at a Practical Size
11/18	New Architectures - MoE Submit your review before 11/16/2024 11:59 PM. ST-MoE: Designing Stable and Transferable Sparse Expert Models MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
11/20	New Architectures - Linear Time Architecture Submit your review before 11/18/2024 11:59 PM. Efficiently Modeling Long Sequences with Structured State Spaces (S4) RWKV: Reinventing RNNs for the Transformer Era
11/25	New Architectures - Sparse Attention Submit your review before 11/23/2024 11:59 PM. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
11/27	Thanksgiving Break
12/02	Final Presentation
12/04	Final Presentation
12/09	RRR Week
12/11	RRR Week

Course Schedule

Introduction and Course Overview

Labor Day

Overview: Making of LLMs

Inference System - Memory

Inference System - Scheduling

Speculative Decoding

Benchmark and Evaluation

Retrieval

Agents

Project Proposal

Project Proposal

Quantization

DL Compiler and Hardware

Data

Optimization and Scaling

Training Kernels

Training Systems - Model Parallelism

Training System - Optimizers

Inference-time Compute

Alignment

Parameter-efficient Finetuning

Veterans Day

Knowledge Distillation

New Architectures - MoE

New Architectures - Linear Time Architecture

New Architectures - Sparse Attention

Thanksgiving Break

Final Presentation

Final Presentation

RRR Week

RRR Week