# Module Information

Module Identifier
MA35810
Module Title
Information Theory
2021/2022
Co-ordinator
Semester
Semester 1
Pre-Requisite
Exclusive (Any Acad Year)
Other Staff

#### Assessment

Due to Covid-19 students should refer to the module Blackboard pages for assessment details

Assessment Type Assessment length / details Proportion
Semester Exam 2 Hours   Exam  100%
Supplementary Exam 2 Hours   Exam  100%

### Learning Outcomes

On successful completion of this module students should be able to:

State various concepts of information and entropy and explain the relationships between
them.

Achieve efficient data compression by coding procedures, guided by theoretical limits.

Explain the notion of a channel as a model of information transmission.

State Shannon’s main theorems about channel capacity and coding.

Reproduce the main assumptions and arguments leading to these theorems

Apply the theoretical results to construct and to analyse a variety of important channels.

### Brief description

Shannon’s seminal paper ‘A mathematical theory of communication’ (1948) created information theory as a part of mathematics.. It which It provides the tools for a rigorous understanding of information processing and communication. In this module we carefully develop the main concepts like entropy,
data compression and coding, channels and their capacity. We explain the main theorems and results and apply them to various classes of examples.

### Content

• Introduction, history, what is information, examples
• Entropy, joint and conditional entropy, relative entropy and mutual information, rules and inequalities
• Asymptotic equipartition property, typical sets and source coding
• Data compression, Kraft inequality and optimal codes, Huffman codes
• Channels and channel capacity, examples
• Shannon’s channel coding theorem, zero error codes, Hamming codes
• Source-channel coding theorem, binary case
• Information transmission guided by theory, detailed discussion of examples

### Module Skills

Skills Type Skills details
Good understanding of the contents requires considerable intellectual effort over an extended period of time.
Theory is developed rigorously and compared with real world situations
Discussing the theory and solving problems together during the module is encouraged.
Problem sessions based on problem sheets to be solved independently. This is crucial to prepare for the problems in the exam.
The ability to solve concrete problems in information theory is checked in the exam.
Discussing the theory and solving problems together during the module is encouraged.
Intuitive ideas need to be translated into mathematical reasoning.
Theory is compared with real world applications.
Insights are provided into the mathematical principles of digital information processing.

### Notes

This module is at CQFW Level 6