Investigating non-tactile MIDI controllers
for severely disabled children

 

Cynthia Jacobs (TempleTap)
Master of Design (Digital Media) candidate 1997

Abstract

Conventional musical instruments require the player to grasp, blow, pluck or slap.

They usually also require the player to adapt to the instrument rather than stretching the parameters of the instrument to make it adaptable to the needs or preferences of the musician.

Severely disabled people, such as those with head movement only, cannot participate in musical experiences of their own free will - they must be physically assisted. The assistance may be supporting the person physically, or holding an instrument. This can leave the severely disabled musician's creative expression unresolved.

Devices available to disabled children to access various equipment and technologies such as computers are limited to oversize switches of various shapes.

Devices for sensing the motion or position of performers and turning that motion into MIDI control signals are being developed. The original instrument of this type is the theremin - an instrument requiring body movements unrepeatable by most severely disabled performers.

This paper will investigate different types of human movement tracking systems and MIDI controllers,

• and their appropriateness as non-tactile musical instruments for severely disabled children;

• limitations of current and other technologies will be discussed.

 

Background

Kurrambee School at Werrington NSW currently has over 70 students ranging in age from 4 to 18 years. They all have some intellectual disability with most having additional physical, sensory and emotional disabilities.

My volunteer work there in 1996 involved making videos at teacher's/therapist's request for record or diagnostic purposes. While videoing computer use and music therapy it was observed that some students can only participate with the intervention of a helper. These students are not physically able to hold a drum stick or use a keyboard or mouse. Their computer use is restricted to pressing a single button to trigger an animated sequence on the screen. Many students cannot even participate to this extent being either partially blind or unable to use their hands/legs or both. (Jacobs 1997).

 

Users Of The Technology

see appendix one

Kurrambee Class 3/5 Gold has eleven students ranging in age from 4 to 11. Six of the students have moderate, physical and intellectual disabilities. They are independently mobile, can feed themselves and can participate in traditional music therapy.

The other five children all have severe, multiple handicaps and high support needs. Bobby, Cain, Hayley and Luke have cerebral palsy affecting their entire bodies, are cortically blind, and have a severe level of intellectual disability as well. They are all trying to learn to say 'hi'. Terri Lee's mother was pushing her in a pram, across a pedestrian crossing, with Terri's older brother walking alongside. All three were struck by a speeding car. Terri's brother was killed, her mum was severely injured and Terri's life was changed forever. Terri is severely brain damaged and cortically blind from an innocent shopping trip. Terri is now able to crawl and stand, and she can take some steps if an adult helps her. (Kurrambee 1996) These children spend much of their time in their customised wheelchairs, and various other physio therapeutic devices.

Russell (1997) pointed out that "Students attending music classes in special education settings have been particularly subject to a subtle form of coercion, with many "music making" sessions consisting of groups of children banging away on a variety of coarsely made percussion 'instruments', the sounds of which do little to inspire the child or adult listener. Children with little voluntary movement are routinely helped "co-actively", that is to say, have their arms and hands worked from behind like a puppet...There is little scope for expressive music making by children under these conditions."

People with devastating neuromuscular handicaps do not have much opportunity to influence the world. Kurrambee kids need some empowerment in their lives, need to learn basic cause and effect, the ability to make their own choices, to express their feelings, to learn to take turns, to participate in music making and games with others.

These observations prompted this research for an assistive device that could be used to create/control music in a variety of situations by severely disabled children. On/off switches are available in a variety of types, shapes, sizes and colours, but these provide a most limited type of interaction. The children currently use head mounted mercury switches to play with specially modified toys. I think using the mercury switches will assist the children in making the connection between body movement and music making.

 

Criteria For Suitable Devices

For device rating see appendix two

sensitivity

The device should give the child the opportunity to participate &endash; if &endash; and when they choose (be sensitive to minute movements of any body part);

range

The device should have a range that accommodates a variety of users;

operation

Not require specialist instruction/knowledge to operate properly;

robust

enough to withstand forceful, (involuntary) movements;

on/off plus

The device should enable the child to create a 'flurry' of musical tones resulting from the child's full range of movement (more than an on/off switch)

Devices

The current digital technology of human-movement-tracking devices, biosensors, video-based devices, are mainly coming from the entertainment fields, dance and theatre.

 

The Dimension Beam by Interactive Light emits an invisible egg-shaped infra-red light field. Motion within the light field is translated into a MIDI signal which can operate effects, keyboards, samples, lights, computers.

 

The Body Harp is an octagonal array of Dimension Beam sensors that allows the entire body to create music. Music can be created with a wide selection of sounds like piano, organ, guitar, or drums. The Body Harp also comes with 24 built-in melodies. Interactive's Smart Beams have the ability to correct for electronic, temperature and other noise in the ambient environment. The company's high-speed filtering system enables powerful infra-red communications and distance sensing technologies. Smart Beam technology includes the ability to custom design the shape of a sensing zone for a particular application. Interactive has patented its ability to create planar shaped sensing spaces. Depending on the particular application, sensing spaces can be created with other shapes such as cones or squares.

comment The Dimension Beam should suit almost all the children, especially since the introduction of the Smart Beam technology. The Body Harp would not be used to its full capabilities by severely disabled users because of its fixed physical arrangement.

 

Axel Mulder's/Infusion Systems I-Cube allows control of a Midi set-up using almost any surface, space or movement. Some of the sensors available can capture body heat, distance, illumination, even G-Force and have that signal translated into Midi. The sensor control environment includes the Digitiser, a small, easily wearable box which is ready to accept the analog signals from sensing devices; the I-CubeX which allows the I-Cube to operate without a host computer and to be programmed using either a Mac or Windows95 compatible PC.

The I-Cube Reach sensor detects the proximity of an object as it moves closer to the center. Once the Reach is calibrated for the environment (eg. the table or wall it is placed on), the sensor will produce a high output value with no object present. As an object moves closer to the center of the Reach, the output value decreases and eventually reaches zero when the object hovers just above. The Reach sensor creates an oscillating electric field between the center electrode and the virtual ground of the outer ring. When a hand or object enters the field, it functions as an alternative virtual ground and attracts part of the electric field. Therefore the field between the center electrode and the outer ring is reduced, causing a smaller current to be detected.

comment The I-Cube system seems to be versatile enough to accommodate a wide range of users, however the programming required makes this device less than ideal.

 

MidiCreator is MMB Music's control centre that receives signals from many kinds of switches and translates them into music through a MIDI compatible sound source such as a keyboard or sound module. There are fourteen inputs for switches and eight preset programs.

MidiGesture is MMB Music's proportional ultrasound switch device that plays sound through the MidiCreator by detecting body movement from the simple wave of a hand to someone moving along or across the beam. Its three ranges cover between 1, 2, and 3 meters.

MidiSensor operates in a similar way to MidiGesture but is extremely sensitive and designed to detect even the slightest movement at a range of about six inches.

comment This group of products fulfils the criteria for an acceptable device, however, the difficulty in getting it set up and operational makes it unsuitable in this situation.

 

Optivideotone Based on the theremin, Professor Scott Hall of the Cogswell Polytechnic College has used light sensitivity in the creation of his Optivideotone, an assemblage of audio and video electronics combined to produce an object that is sculpture, musical instrument/composition tool, and projected video art exhibit in one.

comment This theremin is probably not available on a commercial basis, but could be used as the basis for the creation of a similar device.

 

SensorLab created at STEIM in Amsterdam, is a small, general purpose, analog to MIDI interface for the prototyping of musical instruments and interactive control systems. This box has thirty-two channels of analog to digital conversion, two ultrasound inputs for measuring distance between sensors, over one hundred switch inputs and more.

comment The utility and scope of the SensorLab requires specialist knowledge to operate properly and is therefore unsuitable.

 

The EMS Soundbeam is a no-contact movement-into-sound device which enables users, regardless of physical or cognitive ability, to experiment and communicate with sound by moving within an invisible elastic keyboard in space. Soundbeam works by sending out an ultrasonic ray which can be varied in length from a few inches to several yards, adjusted to the mobility level of the individual user. It converts movement and distance information into MIDI code which is used to send instructions to electronic instruments. Ten selectable MIDI programs are supplied.

comment The sensitivity and range of this device makes it very suitable for a wide range of severely disabled in a classroom setting. The continuously variable range, and the transpose facility of this device lets you adjust the instrument to exactly accommodate the range of movement of the user. These features are not available on other devices.

 

The Synth A Beam System by Interactive Entertainment consists of a MIDI Interface, and photo sensor strips. Almost any directional lighting fixtures can be focused on the sensors. When beams of light are interrupted, MIDI notes are generated. The system can be used to control any equipment that responds to MIDI commands. The Synth A beam sensors are NPN photo transistors that only react to directional light sources and are not affected by ambient lighting.

comment The photo sensor strip restricts the active areas to a preset arrangement that is not suitable for most severely disabled users. The light beams as triggers would probably make the association between movement and sound more apparent for sighted users. I think this device would be excellent for the older, more mobile disabled users.

 

Theremin The theremin is played by waving one's hands near two metal antennas: one for pitch and the other for volume. The antennas vary the frequency of two oscillators. To create the sound, a fixed oscillator is mixed with the variable pitch oscillator and their difference (or beat frequency) is amplified. The Theremin is still popular today in experimental music circles, having been invented in 1919 by Leon Theremin (1896-1993).This unique instrument is easily discernible in the pop song "Good Vibrations" by the Beach Boys, sounding like a demented violin.

Ethervox is a prototype theremin with complete MIDI implementation that Robert Moog demonstrated at the First International Theremin Festival on 21 June 1997, in Portland Maine, USA. It is interesting that Ethervox was the original name for the Theremin.

Multi-voiced Theremin John Rigg's multi-voiced theremin is capable of producing up to 12 tones at once. This theremin can be used by either left or right-handed players. Each of the two brass antennas can control either pitch or volume. Two rows of knobs below the antennas vary the tone produced by the unit. Each side of the instrument can produce sinusoidal or square wave tones in the fundamental, one octave higher, one octave lower, a third, and a fifth. By turning up the volume on different knobs, the instrument can play major chords.

comment The theremin is usually the first instrument to come to mind when discussing non-tactile musical instruments. Although readily available, the arm and body movements required to play the instrument are beyond the severely disabled.

 

Biosensor-based Systems

The BioMuse by BioControl Systems is a bioelectric signal controller which allows users to control computer functions directly from muscle, eye movement, or brainwave signals, bypassing the standard input hardware such as keyboard or mouse.

 

Eagle Eyes developed at Boston College is a new technology that allows a person to control a computer simply by moving the eyes or head. The signals can be used to control the cursor on the screen, making this device suitable for hands-free drawing. The company claims that as the child learns to control the cursor on the computer screen, the child's capabilities and knowledge can be assessed.

 

Interactive Brainwave Visual Analyzer (IBVA) The IBVA provides easy real time analysis and intricate interactive biofeedback control of brainwave conditions. The IBVA reads brainwaves in real time and allows you to use them to trigger images, sounds, other software or almost any electronically addressable device through its MIDI, serial and expansion pak features. With the network and modem features of the IBVA, brainwaves can be analysed and control equipment from anywhere in the world. The user doesn't have to be confined to a few feet of freedom. The IBVA comes standard with a lightweight wireless transmitter that works up to thirty feet away. This allows you to do just about anything while monitoring your mind state.

comment Biosensor systems will probably suit these children in later years when they have experienced more technology of this biofeedback type. At this time the cause and effect reaction of the equipment would make it difficult for the children to associate their bioelectric signals or brainwaves with any reaction.

 

Video based human movement tracking devices

Mandala

The Mandala VR system by the Vivid Group uses a video camera to put the user into virtual worlds. When the user moves the virtual world responds to the user's presence. One of the first uses of the Mandala System was demonstrating how people could create music with their whole body using the system. The company these days is turning their Mandala Systems into marketable games, but they are still in touch with their musical roots, with a recent module dedicated to drumming.

 

Very Nervous System is the third generation of interactive sound installations created by David Rokeby. In these systems, video cameras, image processors, computers, synthesisers and a sound system are used to create a space in which the movements of one's body create sound and/or music. It can sense motion in a space and where that motion occurs. Output from the VNS is via a SCSI connection. Objects are provided that allow access from the MAX programming environment. It has been primarily presented as an installation in galleries but has also been installed in public outdoor spaces, and has been used in a number of performances.

 

comment These would be wonderful devices for an interactive performance piece for more able users. The amount of equipment, and programming requirements take them out of the realm of suitable devices.

 

Anticipated Outcomes

Profoundly handicapped children respond to sound therapy in different ways:

performing, listening, verbalising, composing with sound;

often showing 'aesthetic resonance' through most telling facial expressions;

actively involved for extended periods of time;

revealing an ability for concentration not apparent elsewhere;

beginning to discover, explore, give expression to and communicate their own feelings;

making significant physical responses - movements and gestures which hitherto have not been seen, or have not previously been made independently. (Ellis 1995)

 

Conclusion

To have a true interactive/creative experience the users need to be able to make the connection between their action and the musical reaction. Because many users have little language comprehension, they have to make this connection without instruction by drawing on their relevant life experiences. Some devices are inappropriate because of this. The biosensors rely on detecting brainwaves and other signals that the users have no experience in controlling. Similarly, eye movement tracking devices can be unsuitable for visually impaired users. Those devices that detect minute movements would be the easiest for users to make the action/reaction connection, since all the children have used headmounted tilt-switches to play with modified, battery controlled toys.

The rate at which highend technology is becoming available to the general public is heartening, and should provide more and more appropriate assistive technology devices for the severely disabled.

 

Cynthia Jacobs can be contacted on phone 61 2 45 755105 cyn email

 

REFERENCES

 

Kurrambee School Kaleidiscope Magazine Hydrotherapy leaflet, 1996.

Russell, Kathryn and Russell, Neil ' "See that? That's magic": New Sounds and Sights in Music Movement Improvisation - the Soundbeam Experience. ASME, Brisbane, July 1997.

Ellis, Phil: 'Incidental Music' video with booklet 'Sound Therapy, The Music of Sound', The Soundbeam Project, 1996.

Ellis, Phil: 'The Music of Sound: a new approach for children with severe and profound and multiple learning difficulties'. British Journal of Music Education, 1997.

Jacobs, Cynthia 'BeamZones Software for the Soundbeam', 1997.

 

 

URL REFERENCES (correct at November 1997)

* items are devices investigated

Ability Research Centre
http://www.abilitycorp.com.au/

Accessible Arts Australia Incorporated
http://hepworth.cfa.unsw.edu.au/intersections/aai.html

Adaptive/assistive technology
http://library.uwaterloo.ca/discipline/Disability_Issues/adaptive.html

Australian Commonwealth Office of Disability
http://www.health.gov.au/hfs/ood/c&t.htm

Australian Deaf Blind Council
http://connexus.apana.org.au/~dba/

Australian Disability Resources
http://www.iinet.com.au/~ilcwa/ilcozres.html

Australian Network for Art & Technology
http://www.anat.org.au/

Australian Rehabilitation & Assistive Technology Association
http://www.iinet.net.au/~sharono/arata/

Bill Buxton's references
http://www.dgp.toronto.edu/people/BillBuxton/billbuxton.html

Dance and Technology Zone
http://art.net/~dtz/

* EagleEyes - eye movement tracking
http://www.cs.bc.edu/~gips/EagleEyes/

Electronic Music Foundation Resources
http://www.emf.org/sites_resources.html

* IBVA brainwave controller
http://www.IBVA.com/

* Inclusive Technology - multisensory rooms
http://www.inclusive.co.uk/infosite/multisen.htm

* Infusion Systems (I-Cube)
http://www.infusionsystems.com/

* Interactive Light (Dimension Beam)
http://www.interactivelight.com

* Kurrambee Special School, Werrington NSW
http://www.zip.com.au/~rsterry/kurrambe.htm

Measuring Motion
http://www.science.unimelb.edu.au/SMTU/jon/Papers/motion/Measuring%20Motion.html

* MMB Music Inc (MidiCreator)
http://www.mmbmusic.com/mmbmusic/cat_topics.html

MIDI technology and music therapy
http://www.erols.com/leopold/webdoc3.htm

Massachusettes Institute of Technology vision and modelling group
http://vismod.www.media.mit.edu/vismod/demos/demos.html

* Axel Mulder's references (I-Cube)
http://fas.sfu.ca/cs/people/ResearchStaff/amulder/personal/webpages.html

David Rodger's references
http://farben.latrobe.edu.au/d_rodger/

* Soundbeam home page
http://www.soundbeam.co.uk

Special Education Resources on the Internet
http://www.hood.edu/seri/serihome.htm

Spectronics special needs technologies
http://www.spectronicsinoz.com/

STEIM home page
http://www.xs4all.nl/~steim/

Technology and creative expression
http://brainop.media.mit.edu/Archive/Hyperinstruments/index.html

* Theremin home page
http://www.nashville.net/~theremin/

* Very Nervous System
http://www.interlog.com/~drokeby/vns.html

 


Appendix 1

device

device type

Bobby

Caine

Hayley

Luke

Terri Lee

Dimension Beam

infrared

yes

yes

-

yes

yes

Body Harp

infrared

-

-

-

-

yes

I-Cube

sensor

yes

yes

-

-

yes

MidiCreator

sensor

yes

yes

-

-

yes

MidiGesture

ultrasound sensor

yes

yes

-

-

yes

MidiSensor

ultrasound sensor

yes

yes

yes

yes

yes

Optivideotone

light sensor

-

-

-

-

yes

SensorLab

sensor

-

-

-

-

yes

Soundbeam

ultrasound sensor

yes

yes

yes

yes

yes

Synth A Beam

light sensor

-

-

-

-

yes

Theremin - Ethervox

electric field sensor

yes

yes

-

-

yes

John Rigg's Theremin

electric field sensor

yes

yes

-

-

yes

BioMuse

biosensor

yes

yes

yes

yes

yes

Eagle Eyes

biosensor

yes

yes

yes

yes

yes

Interactive Brainwave Visual Analyzer

biosensor

yes

yes

-

-

-

Mandala

video recognition

-

-

-

-

-

Very Nervous System

video recognition

-

-

-

-

-


Appendix 2

 

device

device type

sensitivity

response range

easy to operate

robust

on off plus

Dimension Beam

infrared

yes

yes

yes

yes

yes

Body Harp

infrared

yes

-

-

yes

yes

I-Cube

sensor

yes

yes

-

-

yes

MidiCreator

sensor

yes

yes

-

yes

yes

MidiGesture

ultrasound sensor

yes

yes

yes

yes

yes

MidiSensor

ultrasound sensor

yes

yes

yes

yes

yes

Optivideotone

light sensor

-

-

-

yes

-

SensorLab

sensor

yes

yes

-

-

yes

Soundbeam

ultrasound sensor

yes

yes

yes

yes

yes

Synth A Beam

light sensor

yes

-

-

yes

yes

Theremin - Ethervox

electric field sensor

yes

-

-

yes

yes

John Rigg's Theremin

electric field sensor

-

-

-

yes

-

BioMuse

biosensor

yes

yes

-

-

yes

Eagle Eyes

biosensor

yes

yes

-

-

yes

Interactive Brainwave Visual Analyzer

biosensor

yes

yes

-

yes

yes

Mandala

video recognition

-

yes

-

yes

yes

Very Nervous System

video recognition

-

yes

-

yes

yes


 

****Soundbeaming in Australia****

Cynthia Jacobs and Bill Stern specialise in digital media and the application of non-tactile and unusual MIDI controllers for assistive technology, performance interactivity, and facilitating enjoyment and participation in music and creative expression with everyone.

 

Cynthia and Bill have been Soundbeaming with kids at Kurrambee Special School since 1996. They developed BeamZones software to extend the capabilities of the EMS Soundbeam. BeamZones has been used with children with high suport needs, for computer mediated games based on sound samples assigned to zones of the Soundbeam.

 

1999 activities include working with the MIDIcreator, EMS Soundbeam with BeamZones,

 

CYNTHIA JACOBS Master of Design (Digital Media)

 

ARATA member - Australian Rehabilitation and Assistive Technology Association

 

Cynthia Jacobs/Bill Stern
Ph: 4575 5105

 

EMAIL: cyn email 

"Investigating non-tactile MIDI controllers for severely disabled children"

http://www.templetap.com/ntmidi.html

 

BeamZones software for the Soundbeam, and beyond...

http://www.templetap.com/sbeaming/sbing.html

http://www.zip.com.au/~rsterry/sndbeam.htm#top

http://www.templetap.com/BeamZones.html

 

 

-


FastCounter by bCentral

since 6 Nov 00