Most of the time, the human brain works like a brilliant machine, seamlessly piecing together the disparate sensory impressions that bombard us from every angle during our waking hours. Our five main senses—sight, hearing, touch, smell and taste—take cues from our environment via specialized cells, sending a constant stream of signals to the brain. The brain usually does a good job of making sense of this cacophony, sometimes too good. When the brain comes across a “glitch” in the information it’s getting from its surroundings, it can work a bit too hard to make sense of it. This effort to make sense of unusual patterns leads to some peculiar illusions.

Ventriloquism, when we perceive sound coming from an object that is in fact silent, is one example. The classic example of this illusion is a performer on a stage with a puppet sitting on their knee. The performer talks while moving their lips as little as possible, while making much more visible movements with the puppet’s mouth. The result is that the inanimate puppet seems to be the one making the quips and jokes, not the ventriloquist.

Audio brought to you by curio.io

The common misconception is that this trick involves the performer somehow “throwing” their voice through a clever trick of the voice box. In fact, it’s the brains of the audience members that perform this feat.

“Imagine you hear a loud sound, and at exactly the same time, there is an abrupt appearance of something. Then, automatically—because of the coincidence in time—you would tend to associate these two events as originating from the same cause,” says Salvador Soto-Faraco, ICREA Research Professor at the Universitat Pompeu Fabra in Spain, who researches perception and ventriloquism. “That is the inference that happens in ventriloquist illusions.”

In ordinary life, the assumption that sounds and movements go together is a very reasonable one. A loud bang and a flash of light that happen at the same time usually are the consequence of a single event—like a fizzle and spark from a plug or an ambulance’s siren and flashing lights. Crucially, the two events, visual and auditory, have to happen at nearly the same time for the trick to work.

“The ventriloquist illusion can be made more or less powerful, depending on the timing between flashing light and sound,” says Soto-Faraco. “If there’s a longer gap, then all of a sudden the illusion is not so strong.”

But in the case of ventriloquism, the stumbling block that results in the illusion comes from the different ways that the senses transmit information. Our sense of hearing, for example, reports information about the location of a sound in a very different way than our sense of sight. For people with good vision, locating an object in space is extremely easy, but trying to pin down exactly where a sound originates from hearing alone—i.e., with eyes closed—is much, much harder. Think of how much more difficult it would be to play football with a blindfold on: You can hear, but you cannot actually see exactly where the ball is.

“Here we have two different sensory modalities giving information about spatial location, but in very different ways,” says Soto-Faraco. “We could think of different currencies. They could not be directly used together. They have to be converted.”

In this conversion calculation, the brain places much more weight on vision than hearing. This is because vision tends to be much more accurate. “In 99.9% of the situations in daily life, this is reliable and doesn’t lead to an illusory perception—or we’d be utterly confused the whole time,” says Soto-Faraco.

But this is where the ventriloquist illusion trips the brain up: The sensory information from vision is not reliable. The puppet’s mouth is moving, but there’s no sound coming out. The sense of hearing is overruled by vision. As a result, it genuinely seems to us that the puppeteer’s voice is coming straight out of the puppet’s mouth.

There is in fact a remarkably simple law for how the brain weighs the information it receives from different sensory sources. It appears to work by a model called “optimal integration theory.” This theory can be described in a mathematical formula for deciding how much reliability to place on any particular bit of sensory information at a given time. So, in other words, the ventriloquism illusion happens when the brain does the math right, but the answer comes out slightly wrong because of something deceptive in the environment—like a clever person pantomiming speech with a dummy and talking without moving their lips.

The quirks of the ventriloquism illusion don’t end there. Our brains find ventriloquism so convincing that the sensation of misdirected sound can persist up to half an hour after we stop seeing the trick. This is known as the ventriloquism aftereffect.

Consider the waterfall illusion. If you watch a waterfall for a little while with your eyes at a fixed point, then turn your head to look at something not moving –a rock, for example—you will see the rock moving upwards, in the opposite direction to the waterfall.

“This is explained by neural fatigue. After a while of the neurons for direction of motion being excited, they get tired,” says Soto-Faraco. “So they stop inhibiting the neurons that compete with them. When they do not receive any more input, because of the lowered inhibition of the competitors, you see the opposite motion.”

A similar thing happens with the ventriloquism effect. When someone is exposed to a sound and a flash that have different spatial origins for a while, the brain adapts to this difference. When the sound and flash are united again, the brain will still be compensating for the difference, making the sound appear to be coming from somewhere else.

Picture this: First, you see a person at the center of your gaze moving their lips, but the sound is actually coming from your right. Your brain compensates for this, and it seems like the person moving their lips is actually the one speaking. So far, so simple. But then the person in front of you actually starts to speak. Your brain will continue to compensate for the spatial discrepancy. So now, the person’s voice will appear to be “thrown” to somewhere off to your left.

This illusory aftereffect happens only very briefly, so you have to move fast to measure it in the lab, Soto-Faraco says. “It’s also not clear whether this happens because of neural fatigue, like in the waterfall illusion. So it’s still a bit of a mystery why these aftereffects happen, in my view.”

Stranger still, scientists have now found that ventriloquism works even when there’s no puppet (or other similar object) involved, a recent study in the journal Psychological Science has found. In the lab, researchers tend to use a slightly more banal set-up than a ventriloquist with a puppet, instead opting for the simpler cues of a tone and a flashing circle. In this latest study, participants were trained to associate a circle on a screen with the tone. Then researchers recreated the ventriloquism effect by shifting the sound to come from a source away from the circle. The researchers found that when you remove the circle (or puppet-replacement) entirely and simply told the participants to imagine a circle in front of them, the ventriloquism aftereffect still happened.

“We were surprised to find that the effects on participants’ perception of acoustic space were almost as strong for imagined stimuli as they were for real visual stimuli,” says study author Christopher Berger of the Karolinska Institute in Sweden in a 2018 press release. “That is, what we imagine seeing can affect our future perception of sound as much as what we actually see.”

As well as shedding light on how our brains process sensory information, researchers believe that studying ventriloquism could help in a number of practical fields. Understanding this quirk of perception could help to train brain-computer interfaces, and to help stroke patients regain their neural function, the Swedish researchers hope. And knowing how our audio-visual perception works could help to fine-tune video communication technologies, says Soto-Faraco.

Similar illusions in different senses are of particular interest in developing virtual reality technology. More closely linking the sensory experiences that define a virtual world is key to advancing the sense of how real the virtual world feels. But it’s a picture that quickly gets very complicated, as each combination of senses is given a different weight in the brain’s calculations of how reliable each sense impression is, and how much attention we should pay to it.

“It’s beautiful but very complex,” says Soto-Faraco. Far from a seemingly trivial party trick, the ventriloquism illusion actually says a lot about the fundamental ways our brains make sense of the sensory information we receive from the world around us. “Like all good illusions, it actually reveals things that we otherwise wouldn’t normally be aware of in our everyday lives,” Soto-Faraco says.

If you’re interested in more research from JSTOR about ventriloquism, try saving this text as a PDF and dropping it into our Text Analyzer tool on JSTOR.