# Courses  /  DescriptionsEECS 428: Information Theory

None ;

### Description

Information measures and their properties: entropy, divergence, mutual information, channel capacity. Shannon's fundamental theorems for data compression and coding for noisy channels. Applications in communications, statistical inference, probability, physics.

• Prerequisites by course: EECS 302 Probabilistic Systems and Random Signals or equivalent.
• Prerequisites by topic: Good understanding of basic probability.  (A review of probability theory will be given in Week 1.)

REQUIRED TEXT: Cover & Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.

REFERENCE TEXTS:

• R. G. Gallager, Information Theory and Reliable Communication, Wiley, 1968.
• D. J. MacKay, Information Theory, Inference and Learning Algorithms, Cambridge, 2004

COURSE DIRECTORProf. Dongning Guo

Problem Sets: There will be weekly problem sets. Problem sets are required to be handed in by the end of the class (usually on Friday) in which they are due. Each student is allowed to have one exception to turn in his/her homework within 72 hours past due time during the quarter. Late problem sets will not be accepted otherwise. Each of you is encouraged to work on the problem sets on your own and by consulting the textbook. Working together in small groups on the problem sets is encouraged whenever it helps to better learn the material. However, each person must write up his/her own solution to hand in.

COURSE GOALS:

DETAILED COURSE TOPICS:

1. Overview of information theory and its applications.
2. Review of probability theory.
3. Information measures (entropy, divergence, mutual information) and basic properties.
4. Typical sets and the Asymptotic Equipartition Property.
5. Data compression/lossless source coding.
6. Entropy rates for stochastic processes, Markov chains.
7. Huﬀman coding, Lempel-Ziv compression.
8. Channel coding, channel capacity.
9. The channel coding theorem for discrete memoryless channels.
10. Converse to the channel coding theorem, joint source channel coding.
11. Discrete and continuous-time Gaussian channels, band-limited channels.
12. Source-channel separation.
13. Rate distortion