Information Entropy

Information entropy is a measure of uncertainty or unpredictability in a set of data or messages. Think of it as quantifying how much "surprise" there is in the information you receive. For example, if you're flipping a fair coin, there's high entropy because each flip is unpredictable. Conversely, if a coin always lands on heads, entropy is low since the outcome is certain. In essence, entropy helps us understand the average amount of information needed to describe a random process or message, highlighting the degree of unpredictability inherent in the data.