A First Course in Information Theory

1. The Science of Information.- Information Measures.- 2.1 Independence and Markov Chains.- 2.2 Shannon's Information Measures.- 2.3 Continuity of Shannon's Information Measures.- 2.4 Chain Rules.- 2.5 Informational Divergence.- 2.6 The Basic Inequalities.- 2.7 Some Useful Information Inequalities.- 2.8 Fano's Inequality.- 2.9 Entropy Rate of Stationary Source.- Problems.- Historical Notes.- 3. Zero-Error Data Compression.- 3.1 The Entropy Bound.- 3.2 Prefix Codes.- 3.3 Redundancy of Prefix Codes.- Problems.- Historical Notes.- 4. Weak Typicality.- 4.1 The Weak.- 4.2 The Source Coding Theorem.- 4.3 Efficient Source Coding.- 4.4 The Shannon-McMillan-Breiman Theorem.- Problems.- Historical Notes.- 5. Strong Typicality.- 5.1 Strong.- 5.2 Strong Typicality Versus Weak Typicality.- 5.3 Joint Typicality.- 5.4 An Interpretation of the Basic Inequalities.- Problems.- Historical Notes.- The I-measure.- 6.1 Preliminaries.- 6.2 The I-Measure for Two Random Variables.- 6.3 Construction of the I-Measure ?*.- 6.4 ?* Can be Negative.- 6.5 Information Diagrams.- 6.6 Examples of Applications.- Appendix 6.A: A Variation of the Inclusion-Exclusion Formula.- Problems.- Historical Notes.- 7. Markov Structures.- 7.1 Conditional Mutual Independence.- 7.2 Full Conditional Mutual Independence.- 7.3 Markov Random Field.- 7.4 Markov Chain.- Problems.- Historical Notes.- 8. Channel Capacity.- 8.1 Discrete Memoryless Channels.- 8.2 The Channel Coding Theorem.- 8.3 The Converse.- 8.4 Achievability of the Channel Capacity.- 8.5 A Discussion.- 8.6 Feedback Capacity.- 8.7 Separation of Source and Channel Coding.- Problems.- Historical Notes.- 9. Rate-Distortion Theory.- 9.1 Single-Letter Distortion Measures.- 9.2 The Rate-Distortion Function R(D).- 9.3 The Rate-Distortion Theorem.- 9.4 The Converse.- 9.5 Achievability of RI(D).- Problems.- Historical Notes.- The Blahut-Arimoto Algorithms.- 10.1 Alternating Optimization.- 10.2 The Algorithms.- 10.3 Convergence.- Problems.- Historical Notes.- 11.