2024 Faculty Courses School of Computing Undergraduate major in Mathematical and Computing Science
Information Theory
- Academic unit or major
- Undergraduate major in Mathematical and Computing Science
- Instructor(s)
- Satoshi Takabe
- Class Format
- Lecture (Face-to-face)
- Media-enhanced courses
- -
- Day of week/Period
(Classrooms) - 1-2 Tue / 1-2 Fri
- Class
- -
- Course Code
- MCS.T333
- Number of credits
- 200
- Course offered
- 2024
- Offered quarter
- 3Q
- Syllabus updated
- Mar 14, 2025
- Language
- Japanese
Syllabus
Course overview and goals
“Information” that we treat occasionally is not usually an entity. However, information theory can treat information mathematically by quantifying information. Moreover, information theory can show the mathematical achievable bounds for data compression and data transmission by using the notions of entropy and mutual information. These findings have been had large influences on recent information processing and transmission technologies.
The aim of this lecture is to understand the relation of information theory to recent information processing and transmission technologies by learning three basic points of information theory: (1) basic concepts such as entropy, (2) source coding for data compression, and (3) channel coding for data transmission.
Course description and aims
The student will treat information mathematically by learning the following:
(1) basic concepts such as entropy and mutual information
(2) source coding for data compression
(3) channel coding for data transmission
Keywords
self-information, entropy, mutual information, source coding, channel coding
Competencies
- Specialist skills
- Intercultural skills
- Communication skills
- Critical thinking skills
- Practical and/or problem-solving skills
Class flow
The student will learn each idea, theorem, and its proof according to slides for the course. Some practice for solving problems will be also performed.
Course schedule/Objectives
| Course schedule | Objectives | |
|---|---|---|
| Class 1 | What is information theory? | Learn the basics of information theory and communication model. |
| Class 2 | Entropy and its properties | Understand entropy and its various relationships. |
| Class 3 | Mutual information | Learn mutual information and KL divergence. |
| Class 4 | Markov chain and entropy rate | Learn Markov chain and related entropy rate. |
| Class 5 | Data compression/source coding | Learn basic concepts of source coding for data compression. |
| Class 6 | Source coding theorem | Understand source coding theorem as a mathematical achievable bound of data compression. |
| Class 7 | Huffman coding | Learn Huffman coding as an example of source coding. |
| Class 8 | Channel and capacity | Learn channel as a model of data transmission and its capacity. |
| Class 9 | Channel coding theorem | Understand the concept of channel coding theorem as a mathematical achievable bound of data transmission. |
| Class 10 | Typical sequences and proof of channel coding theorem | Understand proof of channel coding theorem based on the concept of typical sequences |
| Class 11 | Linear codes | Learn linear codes as basic examples of error-correcting codes. |
| Class 12 | Maximum likelihood decoding | Learn maximum likelihood decoding for linear codes. |
| Class 13 | Information theory for continuous variables | Learn information theory for continuous random variables. |
| Class 14 | Summary and outlook | Summarize learned concepts in this lecture and learn some concepts of advanced information theory. |
Study advice (preparation and review)
Students are encouraged to spend approximately 100 minutes preparing for class and another 100 minutes reviewing class content afterwards (including assignments) for each class by referring to textbooks and other course material.
Textbook(s)
Slides for lectures will be distributed.
Reference books, course materials, etc.
Jun-ichi Inoue, Beginners Guide: Information Theory, Pleiades publishing (Japanese), ISBN: 978-4-903814-17-9
Thomas M. Cover and Joy A. Thomas, Elements of information theory (2nd Edition), John Wiley & Sons, Inc ISBN: 978-0-471-24195-9
Evaluation methods and criteria
Students' knowledge of information quantities, skills for handling them, and understanding of their application such as data compression and channel coding will be assessed based on report assignments.
Related courses
- MCS.T212 ： Fundamentals of Probability
- MCS.T223 ： Mathematical Statistics
- MCS.T312 ： Markov Analysis
- MCS.T332 ： Data Analysis
Prerequisites
Students must have successfully completed both Fundamentals of Probability I (MCS.T212) and Mathematical Statistics (MCS.T223) or have equivalent knowledge.