Entropy, data compression, and channel capacity — from Shannon's 1948 framework to its role in modern AI. 26 lectures, Mon & Wed, 대학 C동 110호.
C = maxp(x) I(X;Y)
Same routine as 2025: 14 lectures before the midterm, 12 after. Lecture notes and full videos on the Heung-No Lee Lab YouTube channel. Dates follow the official GIST 2026 academic calendar: semester Aug 31 – Dec 18, midterm week Oct 19–23, final week Dec 14–18. Chuseok (Sep 24–26, Thu–Sat) does not affect Mon/Wed classes; Oct 5 (Mon) is the substitute holiday for National Foundation Day — study the assigned videos that week.
Course Materials — Lecture Slides, Lecture Book, and all shared class files are in the class Google Drive. Open Drive →| Wk | Monday | Wednesday | Assignments due Wed 23:59 |
|---|---|---|---|
| 1 | 8/31 Introduction to class | 9/2 Probability & Random Variables | |
| 2 | 9/7 LLM · Next-Token Prediction | 9/9 Information / Horse Race | HW0 not graded |
| 3 | 9/14 Entropy | 9/16 Mutual Information | HW1 |
| 4 | 9/21 Jensen's Inequality · Relative Entropy | 9/23 DPI · Markov Chains · 2nd Law 추석연휴 9/24–26 (목–토) | HW2 |
| 5 | 9/28 Sufficient Statistics · Fano · Transformers | 9/30 Quiz #1 · Convergences · LLN | Quiz 1 |
| 6 | 10/5 대체휴일 (개천절) — no class. Videos: AEP · Typical Sets | 10/7 Entropy Rate · Cesàro Mean | HW3 |
| 7 | 10/12 Data Compression · Kraft Inequality | 10/14 Huffman · Lempel-Ziv · Midterm Review | HW4 |
| 8 | 10/19 – 10/23 Midterm Exam (GIST midterm week) | ||
| 9 | 10/26 Communications over Channels | 10/28 Hamming Codes · Erasure Correction | |
| 10 | 11/2 Channel Capacity · Noisy Typewriter | 11/4 BSC · BEC · Symmetric Channels | HW5 |
| 11 | 11/9 Channel Coding Theorem: Setup & Key Ideas | 11/11 Forward Proof · Information BER | HW6 |
| 12 | 11/16 Converse Proof (Fano's Inequality) | 11/18 Quiz #2 · Differential Entropy | Quiz 2 |
| 13 | 11/23 Gaussian Channel Motivation | 11/25 AWGN Channel Capacity | |
| 14 | 11/30 Gaussian Coding Theorem · Sphere Packing | 12/2 Parallel Gaussian Channels · Water-filling | HW7 |
| 15 | 12/7 Multiple Access · Slepian-Wolf | 12/9 Final Preparation | HW8 |
| 16 | 12/14 – 12/18 Final Exam (GIST final week) | ||
Homework is due 23:59 KST on Wednesday — the end of Wednesday, not its beginning. Grading collects your repository at that moment, so anything pushed later is not graded. You register your GitHub username once and use one private repository all semester; the AI grader commits corrections there and opens an issue that mentions you.
it2026-<your-username>.problems/ in your repository.submission/hwN.pdf — exactly that name, N being the
assignment number. A file named anything else is not found, and counts as missing.feedback/hwN-feedback.md.HW#N graded: score/100 is opened and mentions you.Two channels. Quick questions — definitions, theorem statements, course logistics — get instant answers right here in this chat. For rigorous full proofs, deep dives, or homework disputes, use the GitHub Q&A (open an issue and mention @claude): a top-tier model answers there, and every Q&A stays public so you can search past answers. The TA gives hints, not homework solutions.
Attendance: no-show without prior notice, or late entry into (Zoom) class, is considered not polite.