Make Entry -- Become an Editor -- Most Popular: (10, 25, 50, 75, 100)

   Help

Information Theory

Like the physically primitive notions of mass and energy of a particle, the information content, I, of an arbitrary measurement or message composed of particular symbol sequence, is itself a primitive concept. While the roots of information theory extend back to the definition of the classical entropy of a physical system introduced by Clausius in 1864 and Boltzman's probabilistic re-interpretation of classical entropy in 1896, the mathematical formalism for measuring I is due largely to a seminal 1948 paper by Claude E. Shannon. Within the context of sending and receiving messages in a communication system, Shannon was interested in finding a measure of the information content of a received message. Shannon's approach was to obtain a measure of the reduction of uncertainty given some a-priori knowledge of the symbols being sent. Suppose we are given N different and a-priori equally likely possible outcomes. A measure of the information gain, I, is obtained by required that I be additive for independent events. That is to say, if there are two independent sets of outcomes N1 nd N2, so that the total number of outcomes is N = N1 + N2, it is required that I(N1 * N2) = I(N1) + I(N2). This requirement is uniquely satisfied by the function I= c log(N), where "c" is an arbitrary constant.

Web Resources On Information Theory

http://www.cna.org/isaac/Glossb.htm

Book Resources On Information Theory

Information Theory, Evolution and the Origin of Life by Hubert Yockey

Site Maps: Most Recent | Clusters | Browse
New: Graduate Student Job Opportunity