Practical Introduction to Information Theory

Unlock Insights with Information Theory: A Comprehensive Course

Section 1: Foundations of Information Theory

Understanding Information Theory: This course explores information theory, also known as the "method of maximum entropy," a powerful branch of applied mathematics rooted in probability. We prioritize deriving plausible conclusions from available data, making it relevant to any problem framed as a probability challenge.

Key Skills: You will develop two core skills:

1. Problem Formulation: Learn to translate real-world scenarios into solvable probability problems.
2. Solution with Information Theory: Master techniques to apply information theory to find solutions.

Course Overview: This section establishes the probabilistic foundation of information theory.

• Lecture 1: Course Scope: An introduction to the course's focus, outlining the probabilistic basis and future sections covering thermodynamic perspectives, mineral processing applications, and other diverse examples. We'll touch on the connection to machine learning.
• Lecture 2: Combinatorics & Probability: A review of fundamental probability notation and formulas, building a strong understanding of functions like factorials and combinatorial principles.
• Lecture 3: Probability Theory Basics: Delve into basic probability theory, with a focus on the binomial distribution. We'll learn how to estimate probabilities of multiple events starting from single-event probabilities.
• Lecture 4: Non-Equitable Probability: Explore scenarios where single-event probabilities are unequal, extending your problem-solving skills with more complex probability problems.
• Lecture 5: Introduction to Information Theory: Define the mathematical basis of information theory, focusing on the concept of entropy and its relationship to probability.
• Lecture 6: Relative Entropy: Understand relative entropy, which allows for the use of "prior probabilities" – incorporating existing knowledge for more accurate solutions. We'll compare it to the standard entropy function.
• Lecture 7: Bayes' Theorem: Explore Bayes' theorem, a key concept in probabilistic inverse modeling – determining the initial scenario from observed outcomes.
• Lecture 8: Entropy in Action: Dice Throwing: Apply information theory to a practical problem – calculating the probability of a biased dice roll using Excel Solver and Lagrange multipliers.
• Lecture 9: Statistical Concepts & Normal Distribution: Compare and contrast the binomial distribution with the normal distribution (Gaussian distribution), exploring their relationship and applications in entropy calculations.
• Lecture 10: Maximum Likelihood vs. Entropy: Differentiate between maximum likelihood theory and the information theory approach, understanding when each is most suitable.

Section 2: Thermodynamic Perspective of Entropy

Entropy in Thermodynamics: This section explores the historical connection between information theory and thermodynamics, clarifying the distinction between the two while highlighting their shared principles.

• Lecture 11: Historical Basis of Entropy: Trace the origins of entropy in thermodynamics and its link to the development of statistical mechanics.
• Lecture 12: Molecular Energy Distribution: Learn the mathematical foundations of molecular energy distributions.
• Lecture 13: Thermodynamics with Excel: Calculate molecular energy distributions using Excel's Goal Seek functionality.
• Lecture 14: Reconciling Thermodynamics & Information Theory: Summarize the thermodynamic perspective, emphasizing that information theory offers a probabilistic approach to problems that also appear in thermodynamics.

Section 3: Mineral Processing Applications

Information Theory in Mineral Processing: Leveraging the lecturer's 20 years of experience, this section demonstrates how information theory can solve complex challenges in mineral processing.

• Lecture 15: Overview of Mineral Processing Applications: An overview of various applications, including thermodynamics, mineral processing, elections, and games/puzzles.
• Lecture 16: Comminution & Particle Size Distribution: Apply information theory to predict particle size distributions in comminution processes.
• Lecture 17: Comminution Exercise: Hands-on exercise using Excel to analyze feed and product size distributions, introducing the concept of prior distributions and relative entropy.
• Lecture 18: Conservation of Assays & Mass Balancing: Solve mass balance problems in mineral processing using information theory and relative entropy.
• Lecture 19: Creating Entropy Add-ins: Learn to create custom Excel functions for complex entropy calculations using VBA.
• Lecture 20: Mass Balancing with Excel: Solve mass balance problems using Excel Solver, exploring its limitations and how to overcome them.
• Lecture 21: Logit Transform: Understand and apply the logit transform to convert probability values for easier solution using Excel Solver.
• Lecture 22: Logit Transform Exercise: Revisit the mass balance problem using the logit transform and Excel