The mathematical theory of how information is measured, encoded and transmitted. In 1948 Claude Shannon of Bell Labs measured information in bits and proved that messages can cross a noisy channel with arbitrarily few errors.
The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning ... These semantic aspects of communication are irrelevant to the engineering problem.
—— Claude E. Shannon, "A Mathematical Theory of Communication", Bell System Technical Journal 27 (July 1948), introduction, second paragraph
History
Information theory studies how messages are measured, encoded and sent through noisy channels. It grew directly out of telegraph and telephone engineering: Harry Nyquist analysed telegraph speed in 1924, and Ralph Hartley in 1928 proposed measuring information by the logarithm of the number of possible messages, both in the Bell System Technical Journal. Claude Shannon's 1937 master's thesis at MIT applied Boolean algebra to relay switching circuits; during the war he worked at Bell Labs on cryptography and fire control. In July and October 1948 he published "A Mathematical Theory of Communication" in the same journal. It broke any communication system into source, transmitter, channel, noise source, receiver and destination; measured a source's information by H = −Σ p log p, noting that the form is that of entropy in statistical mechanics; and, taking logarithms to base 2, called the unit the bit, "a word suggested by J. W. Tukey". He defined the capacity of a channel and proved that at any rate below it there exist codes that make the frequency of errors arbitrarily small. Norbert Wiener's Cybernetics appeared the same year. In 1949 the paper was reissued as a book with an introduction by Warren Weaver, its title now beginning "The" rather than "A". The story that von Neumann suggested the name "entropy" rests on a later recollection by Myron Tribus and has no independent confirmation.
Connections
Causes1
- The TelegraphenablesNyquist's and Hartley's work on telegraph transmission led directly to Shannon
Echoes2
- The Electronic ComputerechoesThe bit: two kinds of digitisation in the same decade
- ThermodynamicsechoesEntropy, from heat engines to information
Sources
- Claude E. Shannon, A Mathematical Theory of Communication (Bell System Technical Journal 27) (1948)
- Claude E. Shannon, Warren Weaver, The Mathematical Theory of Communication (1949)
- James Gleick, The Information: A History, a Theory, a Flood (2011)
- Ronald R. Kline, The Cybernetics Moment: Or Why We Call Our Age the Information Age (2015)
Open questionswell attested
- The story that von Neumann suggested the name "entropy" rests solely on Myron Tribus's account.
- The word "bit" was coined by John Tukey, but accounts differ on exactly when.
- Whether Shannon entropy and thermodynamic entropy are the same thing is debated.
Why it matters
The decisive step was to separate information from meaning. Shannon said outright that semantics was irrelevant to the engineering problem: the information in a message depends only on how probable its selection was among all possible messages, so the same mathematics serves telegraph, telephone, television and, later, digital storage. The abstraction had a price. Weaver's 1949 introduction distinguished technical, semantic and effectiveness levels and conceded that Shannon had answered only the first; as "information" was borrowed wholesale by biology, psychology and the social sciences, the levels were often run together. On the relation to thermodynamics opinion divides: some hold that Shannon's entropy shares only a formula with the physicist's, others, citing Maxwell's demon and its successors, that acquiring and erasing information carries a physical cost. Arriving in the same decade as the electronic computer, the bit became the common unit of computation and communication, and digitisation acquired a single measure.