"Seeing" voice with frequency plots?

About 10 years ago I proposed that a hand-held device could help hearing-impaired people "see" spoken sounds. I once read an article about a researcher who learned how to read time-frequency plots (on paper back then) such that he could tell what people were saying from the plots *alone*. It makes sense that such a skill could be learned by hearing-impaired people also. If such a person had a hand-held plotting device, they could carry it around and understand what people are saying. Here is an old sketch of such a device: http://www.geocities.com/tablizer/seehear.gif Has anybody heard of research on such? I know there are some specialized similar tools for vowel pronunciation training, but this is more of a general-purpose approach. Rather than being a custom-manufactured device, nowadays it could perhaps use existing devices such as Palm PDA's and other hand-held computers with a few minor additions.

I run a recording studio. I look at time frequency plots all day long. To me the thought of someone telling more than what frequencies are present at what time is laughable. Even the best voice responders used today still have a hard time distinguishing zero through nine or yes and no without being trained to the speakers unique voice and pronunciation. It will be a long time until machines can listen and give people the gist of what is going on even in a perfect acoustic environment with voices the unit may be trained to listen to. peace Phil. "topmind" <topmind@technologist.com> wrote in message news:1152042562.315968.8030@m79g2000cwm.googlegroups.com.. : About 10 years ago I proposed that a hand-held device could help : hearing-impaired people "see" spoken sounds. I once read an article : about a researcher who learned how to read time-frequency plots (on : paper back then) such that he could tell what people were saying from : the plots *alone*. : : It makes sense that such a skill could be learned by hearing-impaired : people also. If such a person had a hand-held plotting device, they : could carry it around and understand what people are saying. : : Here is an old sketch of such a device: : : http://www.geocities.com/tablizer/seehear.gif : : Has anybody heard of research on such? I know there are some : specialized similar tools for vowel pronunciation training, but this is : more of a general-purpose approach. : : Rather than being a custom-manufactured device, nowadays it could : perhaps use existing devices such as Palm PDA's and other hand-held : computers with a few minor additions. :

Interesting. I have IBM's ViaVoice and it can hardly detect and translate voice into text. It claims that if you work with it long enough, it will recognize your voice. But it's so sensitive, you have to be in the same listening environment everytime. That is to say, to use the same microphone each time you use it. Also if it hears outside noise like a refrigerator running, it can't understand your voice. It says with practice you have to build a voice model which I tried and tried and tried. Not much good. The downside too is that it will (even after building a voice model) can not distinguish somebody else's voice. I thought it was a real bummer. Wasted money. I tried to dictate letters for email and it does a fair job but still makes a mess. This is not related, but I saw on TV years ago where a man could identify a song of classical music by feeling the vibrations from a phonograph with the sound turned off. "topmind" <topmind@technologist.com> wrote in message news:1152042562.315968.8030@m79g2000cwm.googlegroups.com.. > About 10 years ago I proposed that a hand-held device could help > hearing-impaired people "see" spoken sounds. I once read an article > about a researcher who learned how to read time-frequency plots (on > paper back then) such that he could tell what people were saying from > the plots *alone*. > > It makes sense that such a skill could be learned by hearing-impaired > people also. If such a person had a hand-held plotting device, they > could carry it around and understand what people are saying. > > Here is an old sketch of such a device: > > http://www.geocities.com/tablizer/seehear.gif > > Has anybody heard of research on such? I know there are some > specialized similar tools for vowel pronunciation training, but this is > more of a general-purpose approach. > > Rather than being a custom-manufactured device, nowadays it could > perhaps use existing devices such as Palm PDA's and other hand-held > computers with a few minor additions. >

On 4 Jul 2006 12:49:22 -0700, "topmind" <topmind@technologist.com> wrote: >About 10 years ago I proposed that a hand-held device could help >hearing-impaired people "see" spoken sounds. I once read an article >about a researcher who learned how to read time-frequency plots (on >paper back then) such that he could tell what people were saying from >the plots *alone*. > >It makes sense that such a skill could be learned by hearing-impaired >people also. If such a person had a hand-held plotting device, they >could carry it around and understand what people are saying. > >Here is an old sketch of such a device: > >http://www.geocities.com/tablizer/seehear.gif > >Has anybody heard of research on such? I know there are some >specialized similar tools for vowel pronunciation training, but this is >more of a general-purpose approach. > >Rather than being a custom-manufactured device, nowadays it could >perhaps use existing devices such as Palm PDA's and other hand-held >computers with a few minor additions. Recognition of voice or music is very difficult. No program can draw the sheet music from a small dance band, never mind a symphony. However. The human brain is a marvel at recognition -- hear that bassoon in the Mozart symphony? Spot a deer in the forest? Listen to one voice in a cocktail party. No problem. So. It should be possible to convert speech (two-dimensional air pressure versus time) into a two-dimensional visible display that a trained person could recognize as words spoken. Challenge: has it been done? If not, do it. Mason C P.S. hint: frequency vs time probably won't do it, nor will a simple pressure vs time graph

I mentioned in another thread how the people developing cochlear technology were humbled by the ability of the brain to learn to interpret the odd noises generated by implants so that initially meaningless noise soon became speech or even music. Not that the cochlear software does not do wonders but, as I say, the brain solves problems which, to date, are way beyond the scope of software. I have no doubt that a graphic rendition of sounds would be helpful - not so much that you would learn to interpret words but rather you would soon be able to indentify the hisses, clicks etc which may be beyond the capacity of your ears and which make all the difference to comprehension. It probably comes as a surprise to many profoundly deaf people how much they depend on seeing the speaker's lips. Anyone who has been hearing impaired for a long time has some lip-reading skills. The graphic information would be a similar valuable supplement to defective hearing. One would be unlikely to use such a gadget with strangers but it would be an interesting addition to a telephone. One day a hand-held device which converts speech to text with a high degree of reliability will be developed but, as has often been pointed out in this group, there are many hurdles to be overcome. Rather sooner, someone will develop a device to convert speech to text which, even though offering a low level of reliability, will be a useful supplement to faulty hearing. I certainly hope someone will experiment with the graphical portrayal of speech frequency patterns.

Mason C wrote: > On 4 Jul 2006 12:49:22 -0700, "topmind" <topmind@technologist.com> wrote: > > > Recognition of voice or music is very difficult. No program can draw > the sheet music from a small dance band, never mind a symphony. > > However. The human brain is a marvel at recognition -- hear that > bassoon in the Mozart symphony? Spot a deer in the forest? > Listen to one voice in a cocktail party. No problem. > > So. It should be possible to convert speech (two-dimensional air > pressure versus time) into a two-dimensional visible display that > a trained person could recognize as words spoken. > > Challenge: has it been done? If not, do it. > > Mason C > > P.S. hint: frequency vs time probably won't do it, nor will a simple > pressure vs time graph http://en.wikipedia.org/wiki/Visible_Speech -- Patricia Burns (Just one s)