But mathematical formulation of a problem can only follow upon the exploratory and descriptive studies by which we first become familiar with the phenomena, that call for reduction in more rigorous analysis. The factor problem for example, is to ascertain whether there is underlying order among attributes whereby we may be able to comprehend all n attributes in terms of smaller number of more fundamental or basic attributes. This will be, in general, the approach taken by this paper which will examine various facets of certain phenomena, then attempt to relate them in terms of information.
I was motivated to investigate this area by the rather familiar idea that, somehow, what a person says, and how he says it, is frequently a function of a basic attitude. Psychologists have tried to exploit this phenomenon through the use of projective tests. I wondered whether such expressions as "loaded words", "richness of meaning", and "packed with information" were indicative of something of a quantifiable nature with which perhaps the information theorists were familiar. If it were possible to measure meaning then we might have a tool which would glve us cardinal values in information terms of attitudes which could be used for interpersonal comparisons.We might also use this tool to measure spatial variation in attitude, because it is well known that expressions and meanings change from area to area.
The results of my research have been rather negative in any operational sense, which cannot convey much information to you, insofar as it may not be wholly unexpected. But while at present beyond the pale of current major efforts toward the quantification of intangibles, the burgeoning field of information theory shows some promise of contributing substantially in this area.
Social science is undergoing a rapid process of reintegration. The common core of theory is drawn from sociology and social psychology. An important cause is the recognition that all major events are aggregated from the interrelated behaviour of human beings. It is information which enables this interrelation. Wiener, one of the founders of information theory, has argued that the servomechanism model (or applied information theory) may be a useful model for describing physiological, psychological and sociological adaptive systems. Adaptation is a rational process in which movement toward a "better" is central. "Better" might be interpreted as "greater orderliness", and psychiatrists contend that many human emotional peculiarities are attempts to compensate for real or imagined threats to ego-order. Such compensations appear as attitudes, such as power-lust, prestige, rectitude, etc., and it is these that we wish to measure.
But first we should consider: What is information? How is it measured?
Since the founding of information theory in 1948 by Wiener and Shannon, "two distinct attempts have been made to deal with the notion of information, one in Europe, and one in America; these have been complementary rather than competitive. Both theories seem to have arisen from much the same class of applied problems; viz. communication involving electrical signals. The European school, in which the names Cherry, Gabor and MacKay are the most important, has been concerned with the problem of the information contained in a representation of a physical situation. ... In America, because of work by Wiener and Shannon, a theory of information transmission has been developed in which the dominant concepts are selection, statistical possibilities and noise".(l)
It is the European approach to the measurement of semantic information which will chiefly concern us here. The people most active in this specific area are MacKay, Carnap and Bar-Hillel.
The American approach, however, or as it is called, selective information theory, seems more fundamental and a more detailed description of it may provide a better introduction to the general field. It should be pointed out early that selective information theory ignores all questions of meaning. It may be useful to introduce at this point three common-sense observations which may form the basis for this presentation of the theory:
"1. A person communicating over a noisy telephone line can get less "across" in a given period of time than he can over a perfectly clear line. 2. Not every letter, nor indeed every word of a message in any natural language is as important as every other one in getting the sense of the message. For example, the missing letter in "q-iet" or the missing word in "many happy ------- of the day" can be filled in, with a high probability of being correct, by anyone knowing English, and therefore in the above context they do not carry much important information. 3. Every person seems to have a limited capacity to assimilate information, and if it is presented to him too rapidly and without adequate repetition, this capacity will be exceeded and communication will break down.
"As they stand it is not immediately obvious that these statements are not concerned with semantics, or, for that matter, that the whole problem of information transmission is not almost wholly semantic. One major contribution of selective information theory is in showing that much of what is implied or suggested in these examples and others like them can be given a precise and useful meaning by a purely statistical treatment".(2)
In order "to make precise and measurable some features of the transmissions of information, it is necessary to introduce a unit in terms of which amounts of information transmitted may be measured. The central observation which is needed before one can arrive at an appropriate unit is that a message conveys information in the sense of reducing uncertainty only by its relation to all other messages that might have been received. Suppose a person is asked whether he smokes. If we have no prior information other than population statistics on smoking, then all we know is the probability that he, as a random selection from the population, will answer "yes" or "no". When he selects one of these alternatives and transmits it, some information has been conveyed. But if it is known a priori that he does smoke, e.g. from previous conversation or from seeing him smoke, then with probability 1 the answer will be "yes" and the receipt of "yes" from him cannot convey any (new) information. In effect, our prior knowledge reduced the set of possible messages to a single element, and so far as we are concerned there was no choice to be made. Thus, no information could be transmitted.
"The minimum condition, therefore, under which information can be transmitted is that of a choice between two alternatives. The maximum uncertainty in such a choice exists when the two alternatives are equally probable. Hence the maximum information is conveyed by a choice between two alternatives when they are equally likely. We take such a choice to be one unit of information. That is, whenever a choice is made between two a priori equally likely alternatives (no matter what they are) we shall say that one unit or information has been transmitted by the choice".(3) It was proposed that this unit be called a bit -- a shorthand form of binary digit -- and that term is commonly used. All statements about information transmission, therefore, will be given in this unit.
We turn now to a derivation of the formula of information measurement. Let a particular sign be i, requiring say Ki successive binary subdivisions to identify it. Its probability is Pi; consequently the final subdivision, which identifies it, divided a range 2Pi into equal parts; the subdivision before that divided the range 22Pi; the one before that 23Pi; and so on until we arrive at the initial division of the whole alphabet, having probability
The average, or expected value of Ki, taken over the whole alphabet a,b,c, ... ,N (in the general case) is then:
.gif)
bits/sign (4)
The English alphabet consists of 26 letters which with punctuation marks comes to about 32 = 25 symbols or signs. Were we to suppose them to be chosen independently and with equal probabilities (both patently false assumptions) then each letter of a message would yield 5 bits of information. Clearly, this is not an accurate estimate of the bits per letter in English prose. However, it does stand as an upper bound to this number. Actually, it is somewhere between 1 and 2 bits per letter.
This latter fact is the result of interdependence of letters in the English language. To take this into account the formula for H above would have to be modified. Such modification gives rise to the concept of redundancy of a language. In information theory, redundancy is a measure of the interdependence of the signals or signs.
To illustrate redundancy (and concomitantly to provide a "feel" for the quantification of language) let us look at Shannon's example from the Mathematical Theory of Communication.
Suppose we put all the letters of the English alphabet into a hat in equal amounts and pull them out one by one "at random". What would they spell?
Here is a sample of such a "language":
(1) XFOML RXXKHRJFFJUJ ZLPWCFWKCYJ FFJEYVKCQHYD QPAAMKBZAACIB
In anyone's estimation this sample does not "make sense". Now suppose that instead of putting the letters into the hat in equal numbers, we put them in proportionally to the frequency with which they actually occur in English and again pull them out at random. The resulting sample now looks like this:
(2) OCRO HLI RGWR NMIELWIS EU LL NBNESEBYA TH EEI ALHENHTTPA OOBTTVA NAH BRL
This still does not make "sense". But there is no doubt that it makes somewhat more "sense" than before. It looks more like English. It does not bristle quite so much with J's and Z's. Somehow we feel that a gradation of sense can be established even among random samples of letters. The feeling is strengthened when we perform the next experiment. We now put into our hat not single letters but pairs, taking care to keep their numbers proportional to their actual occurrence in English. Now we get the following sample:
(3) ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN D ILONASIVE TUCOOWE AT TEASONARE FUSO TIZIN ANY TOBE SEACE CTISBE
Now there is no doubt that we are approaching English. The sample contains two or three real English words, and several "near-words" like "deamy" and "teasonare". A sample of triples looks even better:
(4) IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTURES OF THE REPTAGIN IS REGOACTION OF CRE
Perhaps this sample reminds us of Jabberwocky. It should, because Jabberwocky, too, is an "approximation" to English, a very good approximation that almost makes real sense.
What can be done with letters can be done with words. Compare, for example, the sample of randomly selected words:
(5) REPRESENTING AND SPEEDILY IS AN GOOD APT OR DOME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRhY COME TO FURNISHES THE LINE HAD BE THESE
with a sample of randomly selected pairs of words:
(6) THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED
and see how more "sense" there is in the second, although it still does not "mean" anything.
These "approximations" to English are examples of how the intuitive feeling that one piece of gibberish is somehow closer to the English language than another is a reflection of a precisely and quantitative defined situation. The situation has to do with the characteristic linkages used in English. The extent of these linkages is also a measure of the redundancy of the English language. Redundancy can also be taken as a measure of the fraction of letters which can be randomly deleted from a reasonably long message without making the message unintelligible.
(7) FR EXMPLE WENTYIVE PRCET OF HE LTTERS I TIS ENTNCE HVEBEN DLETED AT RANM.
The redundancy of English is said to be over 50% of the letters transmitted.
Redundancy is both a linguistic and a mathematical term. The more redundancy there is in a source, the more tolerance there is for noise and other imperfections of transmission without serious interference with intelligibility. The importance of the redundancy concept in cryptography is likewise apparent. The more redundant the source of messages, the easier to break the code. All these linguistic matters are contiguous to the field of interest of semanticists. A manner of expression full of clichés is, of course, high in redundancy. It turns out in the mathematical theory of information that messages from a cliché-ridden source (such as the repertoire of a run-of-the-mill politician) are also poor in information. This is some thing semanticists have known all along, but it is gratifying to have this knowledge formulated precisely.(5)
That concludes our discussion of the selective theory of information. In order to effect a transition to the semantic theory, we might first explore the rather unorthodox and controversial approach of some scientists to the role of information in the organism, the machine and the organization. We shall then have a broader base from which to examine the meaning of information and its relation to Man.
In order to penetrate the mysteries of Man's mind, it would appear reasonable to first investigate his origins. Information involves choice and complexity. The earliest organic molecules faced an unspoken choice: to unite or not to unite. Unification increased complexity and hence information content.
Schroedinger in his What is Life? (6) asks why must bodies be so large compared with an atom. He answers that if this were not so, if we were organisms such that a single atom, or even a few atoms, could make a perceptible impression on our senses -- what would life be like? An organism of that kind would not be capable of developing orderly thought. Therefore one may conclude that organisms grew to such a size that permitted a certain equilibrium. Freed of this elemental struggle for survival, the organism continued to develop its information-content in order to cope with the other vagaries of life. Much of this fundamental information is ingrained as the basic instincts. Unlike the unthinking animal, Man has a brain which not only enables him to make optimal use of information, but also to create it. Man can therefore almost literally create his own world.
But this brain of Man itself must feed upon information. And the meat which provides its sustenance may be Lasswell and Kaplan's basic values: power, respect, rectitude, affection, well-being, wealth, skill and enlightenment.
These are quite unlike the tangible needs of Man (such as food, clothing and shelter), and appear to be social phenomena. Thus Man is shown to be not only the product of physical information, but also of social information.
We can measure Man's consumption of physical information. What are the prospects for measuring the intake of social information? How much of the basic values must a man absorb in order to survive? How do we determine what are optimal quantities? And do such quantities vary spatially?
Let us now briefly consider some similarities between machines and organisms in the hope of gaining still more insights into our quantification problem Machines in their evolution undergo "mutations" of considerable magnitudes. Where it takes eons for a new biological species to develop, a new technological "phylum" has on occasion come into being within a generation.
A technological phylum is something similar to a biological phylum. If the latter is defined by a very general plan of organization in a wide class of living things, the former is defined in terms of a principle of operation. We can distinguish four technological "phyla":(7)
(1) Tools: These are functionally an extension of our limbs. They
serve to transmit forces originating in our own muscles.
(2) Clockworks: These actually store their own mechanical energy.
(3) Heat Engines: Now energy is stored in a fuel. The analogy to
living things is becoming stronger.
(4) Information Machines: These operate on the principle of storing and transmitting something called information. These machines are
primarily concerned with systematizing operations in which utilization
of energy is involved.
Information seems to bear a similar relation to energy as organization to effort. Consider traffic flowing through intersections at which are located traffic lights. An outsider might think that the flows were related to the power of the car engines.
But suppose the traffic lights failed. Traffic would soon slow down in traffic jams. The outsider, thinking in terms of energy, might assume that the engines had failed. He would be wrong, of course, because energy has little to do with the traffic problem under consideration. The key concept is not that of energy, but of directions for the utilization of energy (commands "stop" and "go" properly patterned), i.e. a matter of information.
We know that both organisms and machines receive, transmit, store and utilize information. The question of how information is "utilized" is particularly interesting. We now know what food is used for: 3 things:
(1) Source of heat
(2) Source of locomotive and chemical energy
(3) Source of materials for growth and restoring worn-out tissues
All these elements are being constantly dissipated by the organism; heat by conduction and radiation, energy by motion, materials through break down and excretion.
Can it be that besides energy in the form of food and sunlight, organisms also feed on something called "information", which serves to rcstore the order, which is constantly being dissipated in accordance with the second law of thermodynamics?
The formal mathematical equivalence between entropy (the measure of disorder in a physical system) and information (as defined mathematically) was commented on by Shannon, Wiener, MacKay and others. Can it be that this is no mere formal mathematical equivalence, such as obtains between an oscillating mechanical system and the analogous electrical one, but a more fundamental equivalence such as that between heat and energy, or between energy and matter? (8)
We now turn to a consideration of information, organization and values, the penultimate step in our advance to the problem of quantifying human values. Organization appears to be a sort of meta-information insofar as it brings to bear information in order to increase the efficiency of information flows. It epitomizes the systematic approach.
Our values, laws, customs, codes and taboos are means for organizing our behaviors. Art, too, is not only communication, but organization. All art exhibits "pattern", "structure", etc. Religion organizes Man and his universe. The essence of a new theory is to organize experience to a greater degree than before.
The theory of games comes under the concept of organizing an ensemble so as to maximize the value of some function defined over it. One could say that the strategy tells the player what to do, i.e. lowers the entropy of the set of alternatives associated with each move. Or, possession of a strategy organizes his playing. This is so because it is an organization with a function -- to win. More precisely, the function is to couple the ensemble of plays of the game (input) with a desired subset of the set of final positions, maximizing values (output).
But now let us get back to more formal theory, and turn to a discussion of some aspects of semantic information theory. It is in this area where, perhaps, we may see some contribution toward the measurement of intangibles with meaning.
You will recall that the mathematical theory of communication, which was termed selective information theory, measures symbols but is not interested in the symbols it measures. These symbols have nothing to do with what these symbols symbolize. In a study on semantic information, Carnap and Bar-Hillel write that it often turns out that "impatient scientists in various fields applied the terminology and the theorems of communication theory to fields in which the term 'information' was used in a semantic sense". In the opinion of two Dutch semanticists, Mannoury and Vuysje, this tendency may point to a natural and sound development, the more so as scientifically founded theories of signs deal with part of the problems with which theories of information are concerned.
It does seem that the semantic concept of information will serve as a better approximation for some future explication of a psychological concept of information than the concept dealt with in selective information theory. The theory of semantic information presented by Carnap and Bar-Hillel bears on rather restricted language systems, and it is questionable whether and how far it will hold for a full-fledged language. It might be useful, however, for experimental purposes at least, to be restricted to language systems of rigidly defined structures, so that any proposition expressible in one of these languages is indeed specifiable out of an ensemble or sample space of preconceived possibilities.
Carnap and Bar-Hillel have defined the fundamental concepts of their theory on the basis of the theory of inductive probability developed by Carnap. Their language system contains a finite number of individual constants which stand for individuals (things, events or propositions) and a finite number of primitive one-place predicates which designate primitive properties of the individuals. In an atomic statement a primitive property is asserted to hold for an individual. Statements formed out of one or more of the atomic statements with the help of the usual connectives (negation, disjunction, implication, etc.) are molecular statements. With the help of these tools numerical statements can be formed and absolute frequencies (cardinal numbers of classes or properties) and relative frequencies can be expressed in them. Any sentence is either logically true, or logically false or factual (logically indeterminate). The information carried by a sentence is treated as a class of something, and as synonomous with the content of this sentence, and the concept of amount of semantic information is developed by various measures of this content all based on logical probability functions ranging over the contents.
But this does not deal with what Weaver considered the semantic problem, which is concerned with the identity, or close approximation of the interpretation of meaning by the receiver, as compared with the intended meaning of the sender. It does not answer the questions: How effectively does the received meaning affect conduct in the desired way? How precisely do the transmitted symbols convey the desired meaning? How much information is necessary for the level of meaning required to precipitate a given amount of response?
MacKay has tried to come to grips with questions like these in his studies on the operational aspects of human communication. He observes that while the connection between statistical and semantic features of information is indirect, these are features of one and the same central concept, which admits of a single universal operational definition. He approaches this problem by seeking an objective description of what goes on when a man receives information.
The receipt of information seems to imply that in some circumstance or other the receiver's expectations will be different. He is now ready to react differently. Potential reactions may be internal or external. Information then seems to be that which alters the total state of adaptive readiness. By this line of reasoning MacKay arrfves at a definition of the "meaning of a message" as the selective function of the message on an ensemble of possible states of the receiver's conditional probability matrix of reaction, or his transition probability matrix (t.p.m.). The selective information content then, for the receiver as defined in communication theory is a logarithmic measure of the unexpectedness of a particular selective operation.
But what would happen to unexpectedness in a world of perfect knowledge on the part of speakers and receivers? Perhaps presentation of those regularities of its experience to which it has successfully responded. Where does new information come from if, like present computers, what a person knows is a function of what his senses absorb? Can a person cease to be an information source?
We might consider human processes of thought. Perhaps human originality may depend in part on random physiological processes disciplined by way of the t.p.m. of experience. Thus neural processes could occasionally be subject to significant perturbation by random fluctuations in metabolism. When input data do not completely determine output, as a result of metabolic perturbation, then the result will in general be both intelligible and novel, and the human is acting as a primary information source, in the technical sense of the term, i.e. as a source of surprise.
It would appear from this survey that there may be much to be gained by close co-operation between information theorists and psychologists. A key feature of MacKay's theory is the t.p.m. -- an essentially psychological notion. The psychologist has many projective techniques such as TAT, Rorschach and MMPI by which set, or state or readiness, may be established. Experiments may establish correlations between certain amounts of information and changes in set, at least on a probabilistic, if not absolute, basis. At present, scaling techniques are used to order different levels of set. Information correlations may provide the basis for a cardinal scale which would admit of interpersonal comparisons. For example, suppose we have a "satisfactory behavior" scale. A subject, on the basis of projective tests, is classed as "Better". He is then made the target of an information flow of exhortatory
But let us suppose that information theory does have a valid role in the quantification of intangibles, how does one hope to cope with the many problems of language? And what about the information-content of inarticulate objects, such as "status symbols"?
One can characterize science as an attempt to construct a logical system isomorphic to the totality of human experience. That theory is preferred which gives the most efficient (elegant) coding of experience in symbolic form. The world is so complex and our information about it increasing so rapidly that improved methods of storing the information and making it available when needed are imperative.
What is apparently needed (according to Rothstein (9)) is a universal language based on symbolic logic or Boolean algebra. This universal language should be one that can be programmed on a machine. One might employ a binary symbolism. Rothstein says it is easy to show that any language can be encoded into sequences of choices from only two alternatives. A theory of Godel to the effect that any logical system can be mapped into the integers with a one-to-one correspondence between propositions and integers shows that any logical system can be represented in a binary system because the integers can be represented in a binary system.
Let us take a look at the Godel number which facilitates this mapping operation.
Godel (10) described a formalized calculus within which all the customary arithmetical notations can be expressed and familiar arithmetical relations established. The formulae of the calculus are constructed out of a class of elementary signs, which constitute the fundamental vocabulary. A set of primitive axioms are the underpinning, and the theorems of the calculus are formulae derivable from the axioms with the help of a carefully enumerated set of rules of inference.
Godel showed that it is possible to assign a unique number to each elementary sign, each formula (or sequence of signs), and each proof (or finite sequences of formulae). This number, which serves as a distinctive tag or label, is called the "Godel number" of the sign, formula or proof.
Consider the formula (
x) (x = sy) which reads, "there is an x such
that x is the immediate successor of y". The numbers associated with its ten
constituent elementary signs are:
(2) 28 x 34 x 511 x 79 x 118 x 1311 x 175 x 197 x 2313 x 299
On a strictly intuitive plane, it seems that the Godel numbering technique might play an important role in facilitating the manipulation of information symbols.
Godel numbers may be neatly decomposed into the original expression which they represent. This example will illustrate:
There may be other useful insights to be had from a study of Godel's ingenious application of the mathematical concept of mapping.
The problem of non-verbal communication is a very important gap in this paper. There can be no doubt but that much of our information comes from inarticulate sources. There is considerable doubt, at least in my mind, as to the extent to which what is said and written about such a source (e.g. a symbol, such as a ring) can be interpreted as some kind of imputed value in information terms. Certainly when the psychologist uses a scaling technique to measure response or attitude to some objects, he perforce employs language as his medium of expression.
Now since it appears that, in some sense, we can treat all or most phenomena in terms of their information-content, this approach may provide hope for a solution to our pricing problem in an input-output system incorporating both tangibles and intangibles. Information may form a common denominator of exchange. Thus, in considering the production and purchase of a car, we might reason as follows:
Each car produced is a sum of many bits of information. These include not only the physical components of the car, but also the message content of pride of craftsmanship of the workers, technicians and managers. Pride of craftsman ship (using this word in its most general sense) is probably a function of the craftsman's depth of knowledge about his craft. This, in turn, would be a function of the information-input of the craftsman.
Each car purchased requires the surrender of many bits of information in, say, gold. (Note here, in passing, that we may postulate that one bit of information has a single value regardless of its source. Thus all labor gets the same wage per bit, but their rates of information-output will differ considerably.) Part of this gold is in payment for the pride of craftsmanship (or prestige) that went into the car. It seems that, in general, prestige is associated with articles of high craftsmanship. However, pride in craftsmanship in itself, cannot produce a prestigious article. The ornate fretwork on Victorian homes is not now a prestige symbol. Thus, culturally-based demand calls forth new craft trends to replace the old. There is, therefore, some sort of decay-rate in information-content of status symbols.
Another unorthodox view of information is taken, appropriately enough, by the unorthodox field of parapsychology. This field is concerned with the transmission of information thrcugh channels other than the accepted five senses of Man. Thus they have coined such expressions as extrasensory perception and psychokinetics. ESP refers to the alleged ability of some people, termed paragnosts, to sense attitudes and events through being in contact wlth objects which have been in the presence of the attitudes and events in question. Psychokinetics (or PK), on the other hand, is the idea that mentally generated information can be projected telepathicaily in order to affect physical events, e.g. to predetermine the throw of dice. Perhaps, someday, the parapsychologist may contribute to a scientific understanding of the sense or way in which objects seem to take on additional meaning or informstion, through association with other objects, persons or events.
Such is the extent of my present thoughts on the relation of information theory to the quantification of intangibles.
R. McDaniel.
Philadelphia, Pa.
May 8, 1961.
References:
1 Adapted from Luce, Developments in Mathematical Psychology.
2 Luce, op. cit., pp. 7-8.
3 Luce, op.cit., pp. 13-14.
4 Cherry, On Human Communication, p. 179.
5 A. Rapoport, "What is Information?", Synthese, Vol. 9, 1954, pp. 157-173.
6 E.Schroedinger, What is Life? The Macmillan Co., N. Y., 1945.
7 A. Rapoport, "Technological Models of the Nervous System", Methodos, Vol. 7, 1955, p. 132 ff.
8 Rapoport, ibid.
9 J. Rothstein, Communication, Organization and Science, The Falcon's Wing Press, Indian Hill, Colo., 1958.
10 See E. Nagel and J.R. Newman, Godel's Proof, New York University Press N.Y., 1960.