LOG.OS
1.0.0
main1.0.0
1.0.0
main1.0.0
  • HOME
  • LOGOS
    • LOGOS
      • 2021
        • 4
        • 5
  • THEORY
    • Math Basics
      • Chapter 2 Groups, Rings & Fields
        • 2.1 Groups, Subgroups, Cosets
    • Information Theory
      • Chapter 2 Entropy, Relative Entropy & Mutual Information
    • Formal Languages
      • Regular Expressions
    • Language Processing
      • Chapter 2 RE, Text Norm, Edit dist.
        • 2.2 Words
        • 2.3 Corpora
        • 2.4 Text Normalization
        • 2.5 Minimum Edit Distance
      • Chapter 3 N-gram Language Models
        • 3.1 N-Grams
        • 3.2 Evaluating Language Models
        • 3.3 Generalization and Zeros
        • 3.4 Smoothing
        • 3.5 Kneser-Ney Smoothing
      • Chapter 4 Naive Bayes and Sentiment Classification
        • 4.1 Naive Bayes Classifiers
        • 4.6 Naive Bayes as a Language Model
        • 4.7 Evaluation: Precision, Recall, F-measure
      • Chapter 5 Logistic Regression
        • 5.3 The Cross-entropy Loss Function
      • Chapter 6 Vector Semantics and Embeddings
        • 6.1 Lexical Semantics
      • Miscellaneous
    • Algorithms
      • LeetCode Notebook
      • Miscellaneous
    • HCI Toolbox
      • Miscellaneous
  • Programming Languages
    • Python
      • File System & I/O
        • Serialization
      • String
      • Containers
        • Emulating Containers
      • Function
      • Class & Object
      • Type & Reflection
      • Module & Package
      • Python Doc
      • Iterator & Generator
      • Iterables
      • Numerical
      • Miscellaneous
    • C
      • String
      • Miscellaneous
    • C#
      • I/O
      • Miscellaneous
    • C51
      • Basics
      • I/O
      • Start Up Files
      • Listing File / LST
      • Compiler Options
      • Assembler Options
      • MAP file / M51
      • Linker Options
      • Miscellaneous
    • C++
      • DLL
      • Miscellaneous
    • Java
      • Introduction
      • Data Types
      • Control Flow
      • String
      • I&O
      • Classes & Data Structures
      • Miscellaneous
    • JavaScript
      • Control Flow
      • String
      • Array
      • Objects
      • Functions
      • Class
      • ArrayBuffer & TypedArray
      • Iterator & Generator
      • import & export
      • Promise
      • Miscellaneous
    • Bash
      • 3.2 Shell Commands
        • 3.2.5 Compound Commands
      • 6.2 Bash Startup Files
      • Miscellaneous
    • CMD & PowerShell
      • Miscellaneous
    • Rust
      • Basics
      • IO & String
      • OOP
      • Miscellaneous
  • TOOLS - META
    • Bash Tools
      • find
      • tar
      • tmux
      • scp
      • rsync
      • Miscellaneous
    • pip
      • Miscellaneous
    • Python Tools
      • Virtual Environment
      • Dependency Parsing
    • Vim
      • Miscellaneous
    • Git
    • CMake
    • Node.js & npm
    • adb
      • Basics
    • SSH
      • Introduction
      • Login to Remote with SSH Keys
      • Port Forwarding
    • Jupyter Notebook
  • TOOLS - GRAPHICAL
    • X11
    • VNCServer
    • scrcpy
    • OpenCV
      • Installation
      • Hello World
      • Mat
    • OpenNI 2
      • Installation
      • Untitled
    • OpenGL
      • Hello OpenGL
      • Pipeline Basics
    • DirectX
    • NVIDIA HairWorks
      • Interface Mapping in Unity Wrapper
    • Azure Kinect DK
      • Introduction
      • First Application
      • Initialization
      • Data Retrieval
        • Image Format Note
      • Image Manipulation
        • Example Practice
  • TOOLS - NUMERICAL
    • MATLAB
      • Basics
      • Linear Algebra
      • Plotting
      • Numerical Analysis
      • Control Theory
      • Statistics
    • Anaconda
      • Installation
      • Virtual Environment
      • Miscellaneous
    • NumPy
      • Array Basics
      • Miscellaneous
    • TensorFlow
      • Source Code
        • Basics
        • Framework
      • Miscellaneous
    • PyTorch
      • Introduction
      • Tensor
      • Dataset & DataLoader
      • Model Building
        • torch.nn
      • Training
      • torchtext
      • Miscellaneous
    • Auxiliary
      • nltk
      • transformers
        • tokenizer
  • TOOLS - VIRTUALIZED
    • LLVM
    • Docker
      • Installation
      • Basics
    • VirtualBox
  • TOOLS - NETWORKING
    • Flask
  • IDE
    • Visual Studio 2017
      • Miscellaneous
    • Unity
      • Lifecycle
      • Geometry
      • Assets and Scenes
      • Layers
      • Camera
      • User Interaction
      • Advanced Graphics
      • Environment & Platform
      • Serialization
      • Networking
      • Concurrency
      • Unity on Android
      • Miscellaneous
    • Keil μvision4
    • Simplicity Studio
    • WeChat Mini Program DK
      • Project Structure
      • WXML
      • Storage
      • Miscellaneous
  • ARCHITECTURE
    • C++ Optimization
    • USB Protocol
      • Miscellaneous
    • BLE Protocol
      • Introduction
    • AVR
    • Arduino
      • Miscellaneous
    • C8051Fxxx
      • Miscellaneous
    • Android
      • Bluetooth
    • WSL
    • Hololens 2
      • Build & Debug
      • Face Tracking
      • Locatable Camera
    • CI/CD Server from start
      • Initialization
Powered by GitBook
On this page

Was this helpful?

  1. THEORY

Information Theory

Reference

Thomas M. Cover, Joy A. Thomas, Elements of information theory 2nd ed.

Previous2.1 Groups, Subgroups, CosetsNextChapter 2 Entropy, Relative Entropy & Mutual Information

Last updated 3 years ago

Was this helpful?