Information Theory

Module Identifier: MA35810
Module Title: Information Theory
Academic Year: 2025/2026
Semester: Semester 1
Co-ordinator: Dr Rolf Gohm
Reading List: View on Aspire
Pre-Requisite: MA26010
Exclusive (Any Acad Year): MAM5820
Other Staff: Dr Rolf Gohm

Assessment

Assessment Type	Assessment details	Proportion
Semester Exam	Exam: 2 Hours	100%
Supplementary Exam	Exam: 2 Hours	100%

Learning Outcomes

On successful completion of this module students should be able to:

State various concepts of information and entropy and explain the relationships betweenthem.
Achieve efficient data compression by coding procedures, guided by theoretical limits.
Explain the notion of a channel as a model of information transmission.
State Shannon’s main theorems about channel capacity and coding.
Reproduce the main assumptions and arguments leading to these theorems
Apply the theoretical results to construct and to analyse a variety of important channels.

Brief description

C. Shannon’s seminal paper ‘A mathematical theory of communication’ (1948) created information theory as a part of mathematics. It provides the tools for a rigorous understanding of information processing and communication. In this module we carefully develop the main concepts like entropy, data compression and coding, channels and their capacity. We explain the main theorems and results and apply them to various classes of examples.

Content

• Introduction, history, what is information, examples
• Entropy, joint and conditional entropy, relative entropy and mutual information, rules and inequalities
• Asymptotic equipartition property, typical sets and source coding
• Data compression, Kraft inequality and optimal codes, Huffman codes
• Channels and channel capacity, examples
• Shannon’s channel coding theorem, zero error codes, Hamming codes
• Source-channel coding theorem, binary case
• Information transmission guided by theory, detailed discussion of examples

Module skills

Skills type	Skills details
Adaptability and resilience	Good understanding of the contentsrequires considerable intellectual effortover an extended period of time.
Co-ordinating with others	Discussing the theory and solvingproblems together during the module isencouraged.
Creative Problem Solving	Problem sessions based on problemsheets to be solved independently. Thisis crucial to prepare for the problems inthe exam.
Critical and analytical thinking	Theory is developed rigorously and compared with real world situations
Digital capability	Insights are provided into themathematical principles of digitalinformation processing.
Professional communication	Discussing the theory and solvingproblems together during the module isencouraged.
Real world sense	Theory is compared with real worldapplications.
Reflection	Intuitive ideas need to be translated intomathematical reasoning.
Subject Specific Skills	The ability to solve concrete problems ininformation theory is checked in theexam.

Notes

This module is at CQFW Level 6