Statistical NLP: Lecture 5
Entropy
Joint Entropy and Conditional Entropy
Mutual Information
The Noisy Channel Model
Relative Entropy or Kullback-Leibler Divergence
The Relation to Language: Cross-Entropy
The Entropy of English
Perplexity
Email: nat@cs.dal.ca
Home Page: http://borg.cs.dal.ca/~nat