Cognitive Justification of Space as a Compositional Element in
Contemporary Electro-Acoustic Music
Aya
Natalia Karpinska
written for a graduate course in Cognitive Science at
the State
University of New York at Buffalo
I.
| L’espace n’est pas le milieu (réel ou logique) dans lequel se disposent les choses, mais le moyen par lequel la position des choses devient possible.C’est-à-dire qu’au lieu de l’imaginer comme une sorte d’éther dans lequel baignent toutes les choses ou de le concevoir abstraitement comme un caractère qui leur soit commun, nous devons le penser comme la puissance universelle de leurs connexions. [1] |
| Our ability to attend to and interpret the sounds which permeate the environment is essential for survival. Determining the location of a charging mastodon or attempting to cross a busy street both depend on this ability. Our capacity to understand and appreciate music is not a direct consequence of ecological demand.[2] By way of explanation we might offer that humans listen to the real world in one way, and to music in another way; but this cannot be the case. An analogy may be taken from the visual sense. We are able to appreciate a painting of a landscape. The accuracy of the work’s representation rests on the artist’s deployment of various techniques in color and line which reconstruct on a two-dimensional surface the relationships that three dimensional objects bear to one another in the real world.[3] For the artist’s labor to be appreciated, the mental structures used in observing the real world cannot lapse from consciousness when viewing the canvas. A good example was the discovery of techniques which simulated perspectival vision in paintings. The artist’s composition is impressive due to the fidelity his representation bears to real space, utilizing directional lines that converge at a vanishing point or points. In the same way that visual arts inform us about how we see in the real world, music reveals something about the process of listening. |
| In this paper, I will briefly present one of the most comprehensive and well-established theories of music cognition, and show how it does not account for elements of music composition not based on pitch, such as space. The use of space as a parameter in composition of many contemporary electro-acoustic works is not merely a metaphor, but a critical property of musical structure. Spatial composition is directly related with research in sound localization, taking advantage of what is known about our ability to discriminate sound sources in the real world to create a spatial experience within the musical domain. Finally, I will present an attempt at a theory which describes musical space and suggest how its shortcomings would benefit from an improved integration of knowledge about the process of listening. |
II.
| Given that the musical faculty is not necessary to survival, it is curious that some have developed into musical geniuses while others remain tone-deaf. Yet even the least talented individuals are able to recognize variations in melodies, follow a rhythm, or discern a skilled performance from one that is unskilled.[4] The experience of music is essentially a dynamic information process, unfolding in real time and requiring the listener to constantly update and refine his or her mental representation of the composition. This is well illustrated in Beethoven’s Fifth Symphony. The opening phrase is retained in the listener’s memory. It is heard many times throughout the piece, with obvious or subtle differences – the dynamics change, the phrase is transposed an octave, or is played solely by the woodwinds. The attentive listener is aware of the repetition and modulation of this opening theme throughout the piece, and must update her mental representation of the work’s overall structure with each echo of the theme. There is far more to music than the raw, uninterpreted physical signal. Music is a mentally constructed entity, and it is the central task of any theory of music cognition to explain the mental imposition of organization onto physical sounds. |
| Ray Jackendoff proposes a model of music cognition based upon five hierarchical levels of structure. The first and simplest is the musical surface, which involves encoding of the relationships among pitch-events or their combinations, as well as discriminating durations of pitch-events in order to form some basic, coherent representation of a composition. The other four levels are derived from the musical surface. Within the level of grouping structure the musical surface is segmented into phrases; with the metrical structure the events of the piece are related to regular alterations of strong and weak beats. In combination, these two strata describe the "rhythmic articulation" of music. The organization of the piece in terms of melody is accessed through time-span reduction, in which pitches of a piece are given a hierarchical "structural importance" with respect to position in the grouping and metrical structures; and through prolongational reduction, in which hierarchical weight is attached to pitches that express harmonic and melodic tension and relaxation, continuity and progression.[5] In this model music is pure structure, not tied to meaning. |
| The conjecture that complex understanding and analysis of music must build on simpler processes is solid enough, but in attempting to generalize to non-Western music or even to some contemporary computer music certain shortcomings of the theory are revealed. Jackendoff’s hierarchical system does not always hold true within analyses of non-Western compositions. When trying to understand a piece such as an Indian Raga, analysis may rest on the accumulation of motifs rather than a hierarchical system of themes and variations. A Raga is neither a scale nor a mode, its melodic structure is dominated by particular mood. In Ian Mott’s electro-acoustic composition Music of the Sphere, a loud scraping sound begins the piece, and is heard several times throughout the work. Yet it cannot be described as a theme in the same way as the opening phrase in Beethoven’s Fifth Symphony is a theme. The correlation among elements in Music of the Sphere may indicate that the elements are similar, but do not necessarily bear more or less importance to one another.[6] In addition, many contemporary electro-acoustic compositions depart from rhythm completely. A piece that does not conform to any metrical structure or whose components lack a hierarchical structure cannot rightly be termed "music" under Jackendoff’s system. The model put forth by Jackendoff presumes to undertake an explanation of musical structure, however it is better at delineating certain mental processes which are employed to determine properties of musical structure. Electro-acoustic music is still often discussed in terms of Jackendoff’s work, as it is one of the clearest and most comprehensive models of music cognition. Presentations at conferences such as the International Computer Music Conference and the Society for Electro-Acoustic Music in the United States that reference Jackendoff’s theory are numerous. It is not enough, however, for models of music cognition to be confined to tonal music. Jackendoff’s hierarchical system does not account for major compositional elements in electro-acoustic music, such as space. Compositions which utilize space as a means of interpreting pitch-events, as an organizational parameter, or as a basis for algorithmic transformation of sonic entities do not yield well to hierarchical categorizations. |
III.
| Professor and composer of electro-acoustic music Denis Smalley writes "The impressions of space created through pitch - pitch space - are intrinsic to music: for example, how the pitch-space boundaries for vocal and instrumental practice are explored (depth, brightness, the high note, etc.) has always been part of music’s psychology. The height/depth occupancy and spread of pitch-space affects the impression of dimension."[7] The presence of a spatial phenomenon within music pervades even quite simple musical discourse - we speak of high- and low-pitched sounds. Voice teachers instruct their students to sing with their ‘head voice’ for higher notes and ‘chest voice’ for lower notes. It is quite common to have a spatial image in which upper direction corresponds to higher pitch, without explicit training, or reference to notation represented on a staff.[8] Composers of computer music whose works exist solely as recordings are able to take advantage of the recorded format to create spatial experiences. The perceptual experience of musical space depends upon our ability to localize sound. Music is not only a perceptual experience, but also a conceptual one. Sound localization, such as turning around to greet someone who calls your name from behind, occurs in real space. In contrast, pitch is encompassed in metaphorical space. ‘Pitch’ is not an intrinsic property of waveforms like frequency or amplitude, rather it is a quality imposed by perception of audible waveforms. Cognitive processes are not merely qualitative categorizations of percepts. New conceptual constructs can emerge from combination, modulation, and juxtaposition of existing constructs. Composing is largely knowledge-modeling, and includes the composer’s awareness as a listener.[9] The utilization of space as a compositional element in computer music takes advantage of the cognitive phenomena that allow us to identify and localize acoustic events, thus enriching the experience of the piece. Musical works can re-structure our sense of hearing, or bring into a new light, by re-contextualizing certain cognitive functions. |
IV.
| When crossing a street, we need to be able to determine from which side cars are coming from, and at what velocity. We look from side to side, effortlessly combining our senses into a cognitive model of the environment. We are also able to attend to events which are not crucial to the task at hand - birds singing, bits of conversation from passers-by. A critical factor in our appreciation and awareness of acoustic events in everyday experience is that sound is multi-directional, meaning simply that sounds come from all directions. An example is the effect of being able to selectively attend to any number of conversations at a party. We can hear conversations in far reaches of the room, behind us, or to the left or right with ease, though perhaps with unequal amounts of comprehension. In addition, the continuous flow of information is modified by our movement, dynamically updating our cognitive representation of the space. For example, it may be unclear whether a sound is coming from the front or from behind. This occurs when the sound arrives simultaneously at both ears. Moving the head slightly to the left or right disambiguates this perceptual confusion, since one ear will be closer to the signal, and causes the interaural time difference to be asymmetrical – the direction of the sound is determined by analyzing phase differences between the disparate signals from both ears. For a sound coming from the front of a listener, moving the head to the left causes the right ear to receive sound earlier and with greater intensity. Mapping this known sound source within a mental representation of the space aids in ascertaining the location of subsequent sounds. Transferring the concept of multi-directional sound into the musical domain pervades the works of contemporary electro-acoustic composers |
| Spatial composition requires a multi-disciplinary approach, drawing on research which deals with the study of audition. The cognitive and scientific foundation of 3-D sound is contained within the disciplines of physical acoustics, psychoacoustics, and auditory neuropsychology. Physical acoustics deals with sound waves and the acoustic events that determine their properties, psychoacoustics focuses on the relationship between sound waves and the perception of spatial imagery in the listener, and auditory neuropsychology is concerned with understanding the neurological structures which form the basis for our ability to experience sound. |
| A naturally occurring sound produces waves which propagate in all directions. The waves are reflected and diffracted by the various surfaces in the environment, the resultant combination and cancellation of waves is the vibrant acoustic texture the listener encounters. The listener is also a surface off of which waves reflect and diffract, all arriving at different times and from different angles. The sound which travels the shortest distance from the source to the listener’s ears and arrives first is the direct sound. By consequence of its lack of interaction with objects in the environment, direct sound offers less spatial information than indirect sound, which is mediated by the surfaces in the environment. A high-frequency energy sound wave that reaches the listener’s ears is reflected away, low-frequency energy waves is diffracted and bends around the listener. The frequency which approximately divides high- and low-energy waves is around 1500 Hz, whose length corresponds to the diameter of the head. The time-delay which differentiates the arrival of the acoustic signal at each ear plays a critical role, determining the intensity of the sound. This effect is closely related to head size. The lateral location of the source of an acoustic signal arriving at the listener’s eardrums may be determined through an analysis of interaural time differences and interaural intensity differences. Localization of frequencies corresponding to lower pitches, which have longer wavelengths, generally relies on phase information based on interaural time differences resulting from head movements. Localization of frequencies corresponding to higher pitches, which have shorter wavelengths, tends to result from interaural intensity differences resulting from head movements. Directionally dependent information received at each ear can be illustrated in terms of a frequency response, called a head-related transfer function (HRTF).[10] An HRTF describes how the shape of the torso, head, pinnae (outer ears), and ear canals affect the properties of the sound wave. Depending on the application, HRTFs are measured at the entrance to the ear canal, within the ear canal, and/or at the ear drum. Analysis of HRTFs comparing frequency and decibel levels among different individuals reveals that although there is variance in the details of the profiles, their overall shapes are quite similar. Composer and professor of music technology and theory Gary Kendall writes in his 3-D sound primer that this similarity "…suggests that while individuals possess heads of different sizes and pinnae of different shapes, the acoustic processes that forge the individual HRTFs are the same." [11] |
| The listener’s decision about the direction and trajectory of an acoustic signal is influenced by the precedence effect, in which the judgment is biased towards focusing on the first, shortest, most direct path to reach the ears. Each ear is spectrally modified by HRTFs, each transforms the sound differently due to its shape; this changes in real time as the head and/or the source moves. The auditory system must combine these two disparate packets of information into a single perceptual image of the signal in space, extract the directional information, and then reconstruct the original source spectrum. The dynamic interaural cues actually override HRTFs in judging the direction of a sound event. The combined information from both ears, also referred to as binaural cues, performs best in resolving sound sources in the horizontal plane of the head, which is level with the listener’s ears. The minimum audible angle in front of the listener is two degrees, in back of the head it is six degrees. At the sides of the head this angle rises to ten degrees, as the ear opposite the signal can contribute little to localization. |
| Although the pinnae are designed to aid in localization of sound events, there is little evidence in favor of specialization for directional audition in the peripheral neurological system. Time-difference and frequency cues seem to be the primary perceptual data utilized by neural mechanisms, which are combined to yield a spatial referent map of auditory space.[12] The ear is able to discriminate between direct sound and reflected sound, which contributes to the construction of a spatial map of the environment. The reflected sound is usually lower in amplitude, delayed, and lower in pitch due to the fact that higher frequencies are absorbed more easily. A naturally occurring acoustical phenomenon called reverberation is caused by reflections of sound off of surfaces in a space, producing myriads of closely spaced echoes - we are able to hear it in large churches, concert halls, or spaces characterized by high ceilings and reflective surfaces such as marble or glass. Reverberation effects in electro-acoustic music strongly imply a sense of space to the listener. Manfred Schroeder of Bell Telephone Laboratories was the first to execute an artificial reverberation algorithm on a computer in 1961, which simulated the reverberant pattern of an actual room.[13] Contemporary reverberation techniques are not constrained to modeling existing rooms or concert halls, but may also develop fictional spatial effects that are not intended to be realistic by manipulating parameters such as the density of the echoes, the ratio of input sound to reverberated sound upon output, and the total time of the reverberation. Such effects contribute to spatial organization of pitch-events over time and structural relations within a composition. |
V.
| Space as an element in composition is not without precedent; Adrian Willaert composed works in the sixteenth century for two or more choirs distinguished by a systematic and absolute spatial division of the performing groups;[14] Mozart, Berlioz, and Mahler wrote music for multiple and/or spatially separated orchestras. These examples, however, remained exceptions until the rise of the electronic age.[15] The invention of the loudspeaker permitted simultaneous multi-channel playback; thus sound sources could be explicitly positioned within the listening space. Performance space has long influenced the composition of music – who could imagine a Gregorian chant in an open field, or a Balinese gamelan piece within a cavernous church? However, in those situations space was employed as an acoustic element and not as an element of internal structure. The two examples above rely solely on the acoustics of the performative space for emotional or timbral effect. Spatial composition is characterized by careful placement of sound-events - sounds themselves may be thought of as objects. Comprehension of spatial compositions rests primarily on the listener’s ability to localize sound. In a Gregorian chant, the acoustical and social associations of the performative space are the primary factors underscoring an understanding of the work. What was essential to the development of spatial composition was the movement from conceiving of space as a given to recognizing it as a material that could be assembled, created.[16] |
| I have not yet come across a theory of music cognition rivaling Jackendoff’s that considers spatial composition, nevertheless Denis Smalley has perhaps outlined an interesting alternative. Smalley identifies spatial experience within music as the interaction of different types of spaces: composed space, listening space, superimposed space, and diffused space. Composed space manifests integration of spatial imaging into the work. Spatial imaging is employed as a means of representing structural relations among sounds or sound groups as well as the dynamic spectral structure of pitch-events as they change over time; this is the primary cue by which spatial change is perceived by the listener. Smalley describes spatial imaging as follows: "Simply stated, a musical gesture can be more vividly dramatized through spatial displacement, just as a texture can be made more ‘environmental’ through spatial distribution."[17] An excellent illustration of composed space is Karlheinz Stockhausen’s Gesang der Jünglinge. His composition explicitly positions sounds within space through the use of several loudspeakers: |
| Von
welcher Seite, mit wievielen Lautspechern zugleich, ob mit Links- oder Rechtsdrehung, teilweise starr und teilweise beweglich die Klänge und Klanggruppen in den Raum gestrahlt werden: das alles ist für das Verständnis dieses Werkes massgeblich. (From which side, with how many loudspeakers, whether with rotation to left or right, whether motion- less or moving, how the sounds and sound groups should be projected into space: all this is decisive for the understanding of the work.)[18] |
| In contrast to the determined musical material of composition, listening space may vary from listening to listening. For example, the listening space of Stockhausen’s piece would differ dramatically if I experienced it as a live performance or on a record in my apartment, through headphones. The immersion of composed space within listening space gives rise to superimposed space. The acoustical changes inherent in the superimposition process weaken the clarity of the composer’s musical expression, most notably in a public listening space where loudspeakers are distant from the listeners. Aside from this negative effect, strategic multiple speaker set-ups in public listening spaces may enhance the perception of musical content and structure. In the previous example, a solitary experience would perhaps be less distracting than a live performance. If I am listening to a record, I may hear the piece again and again, seeking new relationships among sounds each time. In diffused space, successful manipulation of loudspeaker placement draws attention to the various spatial dimensions and perspectives of the composition, taking advantage of small differences of tone and intensity among the speakers. The timbral effects of speaker placement are analyzed by professionals in the field of sound diffusion. Speaker positions may vary as a function of room size, audience size, quality of the speakers, or the requirements of a composition (Stockhausen wrote Gesang der Jünglinge for five loudspeakers). Sensitivity to the effect of superimposed space on listening space can only be achieved experientially, listening to many electro-acoustic compositions in different spatial contexts. |
| Smalley also argues that electro-acoustic music may articulate the topological material in composition. He references the terms and mental structures we use in perceiving the topographical features of the external world to elucidate the spatial experience in music: |
|
…the spectral
makeup and shaping of sounds in themselves may suggest a |
| Smalley’s analysis delineates crucial elements of spatial composition, however it may begin to falter when the topology analogy is stretched too far. Topological concepts such as boundary or connection are difficult to transfer to the compositional realm. Electro-acoustic compositions which utilize space as a structural or interpretive parameter take advantage of our ability to perceive multi-directional sound, and in doing so illuminate the extent of this cognitive faculty. Kendall affirms that the development of 3-D sound technology is compelled more by composers using space as an artistic parameter than by a drive for physical modeling of existing instruments or performance spaces. |
VI.
| It is unlikely that our understanding of musical structure arises from specific ‘musical’ faculties; rather the perception of music draws on a unique combination of existing cognitive constructs such as the capacity to detect time differences and subtle variations in frequency. As a result of research in the science of audition, electro-acoustic composers are better able to incorporate elements of our cognitive representations and perceptual structure into the compositional process. Spatial experience in particular has entered into contemporary practices of the creation of music, introducing an innovative approach for understanding musical structure. |
| Smalley’s categorizations of musical space and introduction of topological terms for analyzing muscal structure is an admirable endeavor towards formulating a theory of music cognition not based entirely on pitch. Jackendoff’s system of music cognition does very well with Western tonal music, since it accounts for the fundamental components of music composition prior to the twentieth century - pitch and rhythm. Electro-acoustic music has introduced another dimension to composition, that of space. Not only are progression and duration important properties of pitch, but also location. |
| The large body of literature that explains what is known about how we understand space should be expanded to encompass the musical domain. For example, how would our visual-spatial map of the environment transfer to a musical work? To what extent is a spatial image appropriate for understanding musical structure? Psychologist Robert Walker has demonstrated many examples of a cross-modal relationship between auditory stimuli and employment of visual space in experiments done with children. Some of the children in the study had cross-modal relationships between pitch and position along vertical and/or horizontal ordinates, some connected perceived movements in auditory space with horizontal movements in the visual domain, some younger children even connected pitch movements with differences in size.[20] Smalley’s examination of space in electro-acoustic music needs a stronger cognitive basis, which will perhaps emerge from a field that is not as clearly hierarchic as Jackendoff’s linguistic analysis, such as the visual or proprioceptive fields. |
Comments, suggestions, corrections warmly appreciated... aya at technekai dotcom