Hearing loss support - "Seeing" voice with frequency plots?

Author

"Seeing" voice with frequency plots?

topmind

2006-07-04, 4:24 pm

About 10 years ago I proposed that a hand-held device could help
hearing-impaired people "see" spoken sounds. I once read an article
about a researcher who learned how to read time-frequency plots (on
paper back then) such that he could tell what people were saying from
the plots *alone*.

It makes sense that such a skill could be learned by hearing-impaired
people also. If such a person had a hand-held plotting device, they
could carry it around and understand what people are saying.

Here is an old sketch of such a device:

http://www.geocities.com/tablizer/seehear.gif

Has anybody heard of research on such? I know there are some
specialized similar tools for vowel pronunciation training, but this is
more of a general-purpose approach.

Rather than being a custom-manufactured device, nowadays it could
perhaps use existing devices such as Palm PDA's and other hand-held
computers with a few minor additions.

jay

2006-07-04, 9:23 pm

Looks like the Etch-A-Sketch I had 40 years ago. Why not a speech to
text device? Via Voice and other software already does speech to text,
no?

Paul Revere

2006-07-04, 9:23 pm

I run a recording studio. I look at time frequency
plots all day long. To me the thought of someone
telling more than what frequencies are present at what
time is laughable.

Even the best voice responders used today still have a
hard time distinguishing zero through nine or yes and
no without being trained to the speakers unique voice
and pronunciation. It will be a long time until
machines can listen and give people the gist of what is
going on even in a perfect acoustic environment with
voices the unit may be trained to listen to.

peace
Phil.

"topmind" <topmind@technologist.com> wrote in message
news:1152042562.315968.8030@m79g2000cwm.googlegroups.com...
: About 10 years ago I proposed that a hand-held device
could help
: hearing-impaired people "see" spoken sounds. I once
read an article
: about a researcher who learned how to read
time-frequency plots (on
: paper back then) such that he could tell what people
were saying from
: the plots *alone*.
:
: It makes sense that such a skill could be learned by
hearing-impaired
: people also. If such a person had a hand-held
plotting device, they
: could carry it around and understand what people are
saying.
:
: Here is an old sketch of such a device:
:
: http://www.geocities.com/tablizer/seehear.gif
:
: Has anybody heard of research on such? I know there
are some
: specialized similar tools for vowel pronunciation
training, but this is
: more of a general-purpose approach.
:
: Rather than being a custom-manufactured device,
nowadays it could
: perhaps use existing devices such as Palm PDA's and
other hand-held
: computers with a few minor additions.
:

Don

2006-07-05, 2:30 am

Interesting.

I have IBM's ViaVoice and it can hardly detect and translate voice into
text. It claims that if you work with it long enough, it will recognize
your voice. But it's so sensitive, you have to be in the same listening
environment everytime. That is to say, to use the same microphone each time
you use it. Also if it hears outside noise like a refrigerator running, it
can't understand your voice. It says with practice you have to build a
voice model which I tried and tried and tried. Not much good. The downside
too is that it will (even after building a voice model) can not distinguish
somebody else's voice. I thought it was a real bummer. Wasted money. I
tried to dictate letters for email and it does a fair job but still makes a
mess.

This is not related, but I saw on TV years ago where a man could identify a
song of classical music by feeling the vibrations from a phonograph with the
sound turned off.

"topmind" <topmind@technologist.com> wrote in message
news:1152042562.315968.8030@m79g2000cwm.googlegroups.com...
> About 10 years ago I proposed that a hand-held device could help
> hearing-impaired people "see" spoken sounds. I once read an article
> about a researcher who learned how to read time-frequency plots (on
> paper back then) such that he could tell what people were saying from
> the plots *alone*.
>
> It makes sense that such a skill could be learned by hearing-impaired
> people also. If such a person had a hand-held plotting device, they
> could carry it around and understand what people are saying.
>
> Here is an old sketch of such a device:
>
> http://www.geocities.com/tablizer/seehear.gif
>
> Has anybody heard of research on such? I know there are some
> specialized similar tools for vowel pronunciation training, but this is
> more of a general-purpose approach.
>
> Rather than being a custom-manufactured device, nowadays it could
> perhaps use existing devices such as Palm PDA's and other hand-held
> computers with a few minor additions.
>

Mason C

2006-07-05, 4:23 pm

On 4 Jul 2006 12:49:22 -0700, "topmind" <topmind@technologist.com> wrote:

>About 10 years ago I proposed that a hand-held device could help
>hearing-impaired people "see" spoken sounds. I once read an article
>about a researcher who learned how to read time-frequency plots (on
>paper back then) such that he could tell what people were saying from
>the plots *alone*.
>
>It makes sense that such a skill could be learned by hearing-impaired
>people also. If such a person had a hand-held plotting device, they
>could carry it around and understand what people are saying.
>
>Here is an old sketch of such a device:
>
>http://www.geocities.com/tablizer/seehear.gif
>
>Has anybody heard of research on such? I know there are some
>specialized similar tools for vowel pronunciation training, but this is
>more of a general-purpose approach.
>
>Rather than being a custom-manufactured device, nowadays it could
>perhaps use existing devices such as Palm PDA's and other hand-held
>computers with a few minor additions.

Recognition of voice or music is very difficult. No program can draw
the sheet music from a small dance band, never mind a symphony.

However. The human brain is a marvel at recognition -- hear that
bassoon in the Mozart symphony? Spot a deer in the forest?
Listen to one voice in a cocktail party. No problem.

So. It should be possible to convert speech (two-dimensional air
pressure versus time) into a two-dimensional visible display that
a trained person could recognize as words spoken.

Challenge: has it been done? If not, do it.

Mason C

P.S. hint: frequency vs time probably won't do it, nor will a simple
pressure vs time graph

Ken

2006-07-06, 8:26 am

I mentioned in another thread how the people developing cochlear
technology were humbled by the ability of the brain to learn to
interpret the odd noises generated by implants so that initially
meaningless noise soon became speech or even music. Not that the
cochlear software does not do wonders but, as I say, the brain solves
problems which, to date, are way beyond the scope of software.

I have no doubt that a graphic rendition of sounds would be helpful -
not so much that you would learn to interpret words but rather you
would soon be able to indentify the hisses, clicks etc which may be
beyond the capacity of your ears and which make all the difference to
comprehension. It probably comes as a surprise to many profoundly deaf
people how much they depend on seeing the speaker's lips. Anyone who
has been hearing impaired for a long time has some lip-reading skills.
The graphic information would be a similar valuable supplement to
defective hearing.

One would be unlikely to use such a gadget with strangers but it would
be an interesting addition to a telephone.

One day a hand-held device which converts speech to text with a high
degree of reliability will be developed but, as has often been pointed
out in this group, there are many hurdles to be overcome.

Rather sooner, someone will develop a device to convert speech to text
which, even though offering a low level of reliability, will be a
useful supplement to faulty hearing.

I certainly hope someone will experiment with the graphical portrayal
of speech frequency patterns.

PATRICIA BURNS

2006-07-06, 4:25 pm

Mason C wrote:

> On 4 Jul 2006 12:49:22 -0700, "topmind" <topmind@technologist.com> wrote:
>
>
> Recognition of voice or music is very difficult. No program can draw
> the sheet music from a small dance band, never mind a symphony.
>
> However. The human brain is a marvel at recognition -- hear that
> bassoon in the Mozart symphony? Spot a deer in the forest?
> Listen to one voice in a cocktail party. No problem.
>
> So. It should be possible to convert speech (two-dimensional air
> pressure versus time) into a two-dimensional visible display that
> a trained person could recognize as words spoken.
>
> Challenge: has it been done? If not, do it.
>
> Mason C
>
> P.S. hint: frequency vs time probably won't do it, nor will a simple
> pressure vs time graph

http://en.wikipedia.org/wiki/Visible_Speech

--
Patricia Burns
(Just one s)

topmind

2006-07-12, 2:24 am

Paul Revere wrote:
> I run a recording studio. I look at time frequency
> plots all day long. To me the thought of someone
> telling more than what frequencies are present at what
> time is laughable.

It is basically the same info that the ear uses. It is just "presented"
differently that our brain is used to. You could take a picture of the
screen and convert it right back into the orginal sound (although it
would probably be a bit noisier than the original due to analog
translation back and forth.)

Plus, like I said, a professor/researcher trained himself to read paper
versions of such charts. So if the article was correct, then it has
already been proven possible.

I agree that it would take a very persistent and patient person, but
persistent people exist.

>
> Even the best voice responders used today still have a
> hard time distinguishing zero through nine or yes and
> no without being trained to the speakers unique voice
> and pronunciation.

That is because the computers are doing the pattern recognition instead
of people's brain. Brains are better for such. I don't propose having
the computer interpret the sounds, only plot the frequencies using
half-century-old algorithms. Humans do the pattern recognition.

> It will be a long time until
> machines can listen and give people the gist of what is
> going on even in a perfect acoustic environment with
> voices the unit may be trained to listen to.
>
> peace
> Phil.

This machine would allow one to see the peace :-)

>
>
> "topmind" <topmind@technologist.com> wrote in message
> news:1152042562.315968.8030@m79g2000cwm.googlegroups.com...
> : About 10 years ago I proposed that a hand-held device
> could help
> : hearing-impaired people "see" spoken sounds. I once
> read an article
> : about a researcher who learned how to read
> time-frequency plots (on
> : paper back then) such that he could tell what people
> were saying from
> : the plots *alone*.
> :
> : It makes sense that such a skill could be learned by
> hearing-impaired
> : people also. If such a person had a hand-held
> plotting device, they
> : could carry it around and understand what people are
> saying.
> :
> : Here is an old sketch of such a device:
> :
> : http://www.geocities.com/tablizer/seehear.gif
> :
> : Has anybody heard of research on such? I know there
> are some
> : specialized similar tools for vowel pronunciation
> training, but this is
> : more of a general-purpose approach.
> :
> : Rather than being a custom-manufactured device,
> nowadays it could
> : perhaps use existing devices such as Palm PDA's and
> other hand-held
> : computers with a few minor additions.
> :

Dr. Michael W. Ridenhour

2006-07-12, 8:24 am

Is there documentation from another source that says this
professor/researcher was able to accomplish this unlikely feat, or is it
just she herself who says so. As an actual professor/researcher I have
learned to beware the statements one reads in the press, even the science
news press.
I doubt the report, based on 35 years of having worked in the field. The
visual and auditory cortex are not particularly amenable to cross switching,
although it "sounds" like they should be. I would love to read the article
you cite. Would you please post the url?

Michael

"topmind" <topmind@technologist.com> wrote in message
news:1152677983.950013.54660@35g2000cwc.googlegroups.com...
> Paul Revere wrote:
>
> It is basically the same info that the ear uses. It is just "presented"
> differently that our brain is used to. You could take a picture of the
> screen and convert it right back into the orginal sound (although it
> would probably be a bit noisier than the original due to analog
> translation back and forth.)
>
> Plus, like I said, a professor/researcher trained himself to read paper
> versions of such charts. So if the article was correct, then it has
> already been proven possible.
>
> I agree that it would take a very persistent and patient person, but
> persistent people exist.
>
>
>
> That is because the computers are doing the pattern recognition instead
> of people's brain. Brains are better for such. I don't propose having
> the computer interpret the sounds, only plot the frequencies using
> half-century-old algorithms. Humans do the pattern recognition.
>
>
>
> This machine would allow one to see the peace :-)
>
>

Paul Revere

2006-07-12, 8:24 am

"Dr. Michael W. Ridenhour" <ocie@medbush.net> wrote in
message news:VT4tg.9895$PO.7167@dukeread03...
: Is there documentation from another source that says
this
: professor/researcher was able to accomplish this
unlikely feat, or is it
: just she herself who says so. As an actual
professor/researcher I have
: learned to beware the statements one reads in the
press, even the science
: news press.
: I doubt the report, based on 35 years of having
worked in the field. The
: visual and auditory cortex are not particularly
amenable to cross switching,
: although it "sounds" like they should be. I would
love to read the article
: you cite. Would you please post the url?
:
: Michael

Thanks Michael:

I would not be surprised if it turned up in Scopes as
urban legend. I met a kid 35 years ago who told me he
could sight read an oscilloscope display. I believed
him only to be embarrassed at his complete inability to
do anything but make up sh*t.

Maybe some guy can tell the difference between the two
words yes and antidislesestablishmantarianism but
reading a time amplitude display to "read" any one's
voice saying just about anything, even if it were
stored and easily displayed at any resolution like
modern digital audio workstations, is IMHO impossible.

thanks for playing

peace
Paul Revere

Mikey

2006-07-12, 4:25 pm

"topmind" <topmind@technologist.com> wrote in message
news:1152677983.950013.54660@35g2000cwc.googlegroups.com...[vbcol=seagreen]
> Paul Revere wrote:
>
> It is basically the same info that the ear uses. It is just "presented"
> differently that our brain is used to. You could take a picture of the
> screen and convert it right back into the orginal sound (although it
> would probably be a bit noisier than the original due to analog
> translation back and forth.)
>
> Plus, like I said, a professor/researcher trained himself to read paper
> versions of such charts. So if the article was correct, then it has
> already been proven possible.
>
> I agree that it would take a very persistent and patient person, but
> persistent people exist.
>
>
>
> That is because the computers are doing the pattern recognition instead
> of people's brain. Brains are better for such. I don't propose having
> the computer interpret the sounds, only plot the frequencies using
> half-century-old algorithms. Humans do the pattern recognition.
>
>
>
> This machine would allow one to see the peace :-)
>

Do you know if this time-frequency plots were called a sound spectrograph?
If so, one word could be a foot long on the machine, which breaks sound into
phonemes. I used to analyze the same graphs. I got so that I could guess
fairly accurately what one particular phoneme was. Doing it rapidly with
thousands of phonemes in several seconds is just not possible. Even using a
palm pilot, there would be a requisite time delay to differentiate the
differing phonemes. It requires analyzing the minute differences visually.
Who has several hours to decipher, "Hello, how are you?" If it were
possible, which I doubt, it certainly isn't practical. Persistence isn't
helpful, any more than being persistent at trying to jump 10 feet straight
up into the air from level ground would be (excluding Superman, of course).

mikey

Paul Revere

2006-07-12, 4:25 pm

"Mikey" <occam@medscape.com> wrote in message
news:iZ6tg.18838$f76.11857@dukeread06...
:
: "topmind" <topmind@technologist.com> wrote in message
:
news:1152677983.950013.54660@35g2000cwc.googlegroups.com...
: > Paul Revere wrote:
: >> I run a recording studio. I look at time frequency
: >> plots all day long. To me the thought of someone
: >> telling more than what frequencies are present at
what
: >> time is laughable.
: >
: > It is basically the same info that the ear uses. It
is just "presented"
: > differently that our brain is used to. You could
take a picture of the
: > screen and convert it right back into the orginal
sound (although it
: > would probably be a bit noisier than the original
due to analog
: > translation back and forth.)
: >
: > Plus, like I said, a professor/researcher trained
himself to read paper
: > versions of such charts. So if the article was
correct, then it has
: > already been proven possible.
: >
: > I agree that it would take a very persistent and
patient person, but
: > persistent people exist.
: >
: >>
: >> Even the best voice responders used today still
have a
: >> hard time distinguishing zero through nine or yes
and
: >> no without being trained to the speakers unique
voice
: >> and pronunciation.
: >
: >
: > That is because the computers are doing the pattern
recognition instead
: > of people's brain. Brains are better for such. I
don't propose having
: > the computer interpret the sounds, only plot the
frequencies using
: > half-century-old algorithms. Humans do the pattern
recognition.
: >
: >
: >> It will be a long time until
: >> machines can listen and give people the gist of
what is
: >> going on even in a perfect acoustic environment
with
: >> voices the unit may be trained to listen to.
: >>
: >> peace
: >> Phil.
: >
: > This machine would allow one to see the peace :-)
: >
: >>
: >>
: >> "topmind" <topmind@technologist.com> wrote in
message
: >>
news:1152042562.315968.8030@m79g2000cwm.googlegroups.com...
: >> : About 10 years ago I proposed that a hand-held
device
: >> could help
: >> : hearing-impaired people "see" spoken sounds. I
once
: >> read an article
: >> : about a researcher who learned how to read
: >> time-frequency plots (on
: >> : paper back then) such that he could tell what
people
: >> were saying from
: >> : the plots *alone*.
: >> :
:
: Do you know if this time-frequency plots were called
a sound spectrograph?

Frequency is synonomous with spectrum. A lot of things
could be spetcrographs. If you are talking about the
etch a scketch picture someone linked, that is
frequency vs amplitude but it does not appear from the
pucture to have frequency vs amplitued plotted along a
time line so one could see how the spectrograph varies
over time.

The etch a scketch is a picture of a continous sound ,
most likely averaged over time to get a reasonable
approximation . If the time is sliced to quickly one
can not plot the low frequencies as the time slice
could be shorter than the wave that needs to be
measured, kind of like looking at a newspaper picture
with a microscope.

Spectragraphs may be usefull to tell what made the
sound but now what sound is being made. You could
probably tell the difference between you and me talking
with one but not a thing about what we are talking
about.

: If so, one word could be a foot long on the machine,
which breaks sound into
: phonemes. I used to analyze the same graphs. I got so
that I could guess
: fairly accurately what one particular phoneme was.
Doing it rapidly with
: thousands of phonemes in several seconds is just not
possible. Even using a
: palm pilot, there would be a requisite time delay to
differentiate the
: differing phonemes. It requires analyzing the minute
differences visually.
: Who has several hours to decipher, "Hello, how are
you?" If it were
: possible, which I doubt, it certainly isn't
practical. Persistence isn't
: helpful, any more than being persistent at trying to
jump 10 feet straight
: up into the air from level ground would be (excluding
Superman, of course).
:
: mikey
:
:

jim

2006-07-12, 4:25 pm

topmind wrote:
> About 10 years ago I proposed that a hand-held device could help
> hearing-impaired people "see" spoken sounds. I once read an article
> about a researcher who learned how to read time-frequency plots (on
> paper back then) such that he could tell what people were saying from
> the plots *alone*.
>
> It makes sense that such a skill could be learned by hearing-impaired
> people also. If such a person had a hand-held plotting device, they
> could carry it around and understand what people are saying.
>
> Here is an old sketch of such a device:
>
> http://www.geocities.com/tablizer/seehear.gif
>
> Has anybody heard of research on such? I know there are some
> specialized similar tools for vowel pronunciation training, but this is
> more of a general-purpose approach.
>

That display is similar to what is called a "Wavelet Transform." Before
he died, my brother was playing around with this. He made a computer
routine called the "Sound Scalpel" that let you see a sound and
graphically remove unwanted parts of it (like noise). So you could
recognize certain sounds graphically.

He made a computer demonstration showing Wavelet plots of familiar
sounds (like the "Tada" chord) mixed with other sounds. Here is his
description of a display of it on a copy of his web site:

"The absolute value of the wavelet transform of the Tada Chord is shown
for the frequency range 400 - 1300 Hz.. This display is similar to a
musical score, with the frequency plotted on a log scale on the y-axis
and time plotted on the x-axis.

A Police Siren (of shorter duration) is shown for the same range of
frequencies.

The Tada Chord plus Police Siren was obtained by zero-padding the
Police Siren at both ends to make it the same length as the Tada Chord
and then adding the two sound signals together. The Tada Chord appears
darker in this display than it does by itself because, for display
purposes, the images are normalized.

For this experiment, the Tada Chord will be considered to be the signal
and the Police Siren will be considered noise. The goal is to remove
the Police Siren and preserve the Tada Chord. The part of the Police
Siren in the frequency range 400-1300 Hz was removed in an editing
session. The Police Siren also has overtones in the range 1300 - 3500
Hz. These overtones were edited out in a similar fashion (not shown).

The De-Noised Tada Chord was then computed by applying the
pseudo-inverse wavelet transform."

I could post these demo pages of his if anyone was interested.

Paul Revere

2006-07-12, 4:25 pm

google Waves x-noise.

I have used it to get the turntable noise out of a
recording. It works great when you can train it and if
that training is for something simple and repetitive.
x-hiss, x crackle, x-hum and other x's exist for their
unique ability to recognize common determinate sounds.
I will take notice when they have x reverb! now we
would be talking recognition.

Also get familiar with latency. It is what technology
is conquering. Humankind is working on programming the
algorithms, but we may not be able to benefit in our
lifetimes.

peace
Paul Revere

"jim" <jwcrim@gmail.com> wrote in message
news:1152723896.386665.115410@75g2000cwc.googlegroups.com...
: topmind wrote:
: > About 10 years ago I proposed that a hand-held
device could help
: > hearing-impaired people "see" spoken sounds. I once
read an article
: > about a researcher who learned how to read
time-frequency plots (on
: > paper back then) such that he could tell what
people were saying from
: > the plots *alone*.
: >
: > It makes sense that such a skill could be learned
by hearing-impaired
: > people also. If such a person had a hand-held
plotting device, they
: > could carry it around and understand what people
are saying.
: >
: > Here is an old sketch of such a device:
: >
: > http://www.geocities.com/tablizer/seehear.gif
: >
: > Has anybody heard of research on such? I know there
are some
: > specialized similar tools for vowel pronunciation
training, but this is
: > more of a general-purpose approach.
: >
:
: That display is similar to what is called a "Wavelet
Transform." Before
: he died, my brother was playing around with this. He
made a computer
: routine called the "Sound Scalpel" that let you see a
sound and
: graphically remove unwanted parts of it (like noise).
So you could
: recognize certain sounds graphically.
:
: He made a computer demonstration showing Wavelet
plots of familiar
: sounds (like the "Tada" chord) mixed with other
sounds. Here is his
: description of a display of it on a copy of his web
site:
:
: "The absolute value of the wavelet transform of the
Tada Chord is shown
: for the frequency range 400 - 1300 Hz.. This display
is similar to a
: musical score, with the frequency plotted on a log
scale on the y-axis
: and time plotted on the x-axis.
:
: A Police Siren (of shorter duration) is shown for the
same range of
: frequencies.
:
: The Tada Chord plus Police Siren was obtained by
zero-padding the
: Police Siren at both ends to make it the same length
as the Tada Chord
: and then adding the two sound signals together. The
Tada Chord appears
: darker in this display than it does by itself
because, for display
: purposes, the images are normalized.
:
: For this experiment, the Tada Chord will be
considered to be the signal
: and the Police Siren will be considered noise. The
goal is to remove
: the Police Siren and preserve the Tada Chord. The
part of the Police
: Siren in the frequency range 400-1300 Hz was removed
in an editing
: session. The Police Siren also has overtones in the
range 1300 - 3500
: Hz. These overtones were edited out in a similar
fashion (not shown).
:
: The De-Noised Tada Chord was then computed by
applying the
: pseudo-inverse wavelet transform."
:
: I could post these demo pages of his if anyone was
interested.
:

topmind

2006-07-23, 4:26 pm

Dr. Michael W. Ridenhour wrote:
> Is there documentation from another source that says this
> professor/researcher was able to accomplish this unlikely feat, or is it
> just she herself who says so. As an actual professor/researcher I have
> learned to beware the statements one reads in the press, even the science
> news press.
> I doubt the report, based on 35 years of having worked in the field. The
> visual and auditory cortex are not particularly amenable to cross switching,

What kind of studies were done to determine this?

> although it "sounds" like they should be. I would love to read the article
> you cite. Would you please post the url?
>

I believe it was Popular Science magazine, and it was so long ago that
it is hard for me to narrow it down. It was an old copy even when I
read it IIRC. I would roughly estimate between 1983-1990. I apologize
for not having more detail.

I have played with PC-based sound spectragraphs myself, and did start
to see patterns with words, even with cheap software and mic. I agree
it would take a *lot* of practice to do fast identification, but it did
not seem out of reach for somebody dedicated enough.

> Michael
>

-T-

topmind

2006-07-23, 4:26 pm

Mikey wrote:

> Do you know if this time-frequency plots were called a sound spectrograph?

Could be.

> If so, one word could be a foot long on the machine, which breaks sound into
> phonemes.

The length depends on how wide one wants to stretch it. I don't know
what the optimum is, and it may depend on the person. If you have a
good memory, then you may chose improved fidelity instead by letting it
stretch.

Such a divice should have a "compression" knob/button to control how
the time factor is multiplied, and the ability to scroll back and
review recent sounds.

I am not sure what you mean by "breaks sound into phonemes". I am not
proposing that the machine try to do any word or phonemes
interpretation itself, but just spit out the spectragraph as faithfully
as possible. Perhaps your machine was slow because it *did* try to do
phoneme interpretation in addition to frequency plots.

> I used to analyze the same graphs. I got so that I could guess
> fairly accurately what one particular phoneme was. Doing it rapidly with
> thousands of phonemes in several seconds is just not possible.

What was the limiting factor? Somebody doing it day in and day out for
years may get pretty fast at it.

> Even using a
> palm pilot, there would be a requisite time delay to differentiate the
> differing phonemes. It requires analyzing the minute differences visually.
> Who has several hours to decipher, "Hello, how are you?" If it were
> possible, which I doubt, it certainly isn't practical. Persistence isn't
> helpful, any more than being persistent at trying to jump 10 feet straight
> up into the air from level ground would be (excluding Superman, of course).

You never know. Maybe some hearing-impaired person is a "graph-reading
savant", and years of practice may make them quite quick. I can't do
(book) speed-reading, but some do and do it quite fast.

It would make an interesting experiment even if it does not produce
real-time reading ability.

>
> mikey

-T-

Michael Ridenhour

2006-07-23, 4:26 pm

> What was the limiting factor? Somebody doing it day in and day out for
> years may get pretty fast at it.
>
>
> You never know. Maybe some hearing-impaired person is a "graph-reading
> savant", and years of practice may make them quite quick. I can't do
> (book) speed-reading, but some do and do it quite fast.
>
> It would make an interesting experiment even if it does not produce
> real-time reading ability.
>
>
> -T-
>

Years ago I interpreted sign language for a man who had entered a community
college and needed a two week interpreter. I had some sign, but was very
slow. After the two weeks, however, I was amazed that even fingerspelling
(it was a college level class, so lots of fingerspelling) became words, not
particles of words. What I mean is, I saw the word, not the fingerspelled
letters after a few weeks. Lost it very quickly when I stopped, though.
Maybe your plan would work, though now I'm sure I'm too old.

Michael

topmind

2006-07-23, 4:26 pm

Here is a related and interesting link from researcher Tim Carmell:

http://cslu.cse.ogi.edu/tutordemos/...pectrogram.html

Quote:

"An experienced spectrogram reader has no trouble identifying the word
"compute" from the visually salient patterns in the image above. To
give one example, the vertical burst of energy followed by a red area
at the bottom and lesser energy above at the extreme right of the
spectrogram is a typical pattern for the sound 't' at the end of a
syllable or word. The other speech sounds, or phonemes, in the word
"compute", are equally distinct in their shapes; the initial unstressed
syllable /kh ^ m/, the silence and bilabial burst of /pc ph/, and the
stressed vowel /ju/ which represents the passage from a high front
vowel to a high back vowel by the falling F2, and the proximity of the
alveolar plosive by a subsequent rise in F2 toward the alveolar locus
of 1800 Hz."

Here is a gray-scale rendition of the same example:

http://cslu.cse.ogi.edu/tutordemos/.../grayscale.html

-----------------------------------------
topmind wrote:
> About 10 years ago I proposed that a hand-held device could help
> hearing-impaired people "see" spoken sounds. I once read an article
> about a researcher who learned how to read time-frequency plots (on
> paper back then) such that he could tell what people were saying from
> the plots *alone*.
>
> It makes sense that such a skill could be learned by hearing-impaired
> people also. If such a person had a hand-held plotting device, they
> could carry it around and understand what people are saying.
>
> Here is an old sketch of such a device:
>
> http://www.geocities.com/tablizer/seehear.gif
>
> Has anybody heard of research on such? I know there are some
> specialized similar tools for vowel pronunciation training, but this is
> more of a general-purpose approach.
>
> Rather than being a custom-manufactured device, nowadays it could
> perhaps use existing devices such as Palm PDA's and other hand-held
> computers with a few minor additions.

topmind

2006-07-23, 4:26 pm

Mason C

2006-07-23, 4:26 pm

Just to toss a pebble into this water:

Certainly an *imperfect* phoneme-recognition program could be
used to create a visible representation much more readable
than variant on spectograms?

Mason C

Paul Revere

2006-07-24, 8:26 am

So you sat in a room with a person signing ( presumably
in the same room where the person speaking was speaking
then this signing person converts that to sign
language.......go on) then YOU read the sign language
and yelled it into the deaf guys ear. What the heck are
you talking about? Why would anyone need an
interpreter to interpret sign language?

"Michael Ridenhour" <ocie@medbush.net> wrote in message
news:yfPwg.103637$IZ2.43179@dukeread07...
:
:

:
: Years ago I interpreted sign language for a man who
had entered a community
: college and needed a two week interpreter. I had some
sign, but was very
: slow. After the two weeks, however, I was amazed that
even fingerspelling
: (it was a college level class, so lots of
fingerspelling) became words, not
: particles of words. What I mean is, I saw the word,
not the fingerspelled
: letters after a few weeks. Lost it very quickly when
I stopped, though.
: Maybe your plan would work, though now I'm sure I'm
too old.
:
: Michael
:
:

Paul Revere

2006-07-24, 8:26 am

If you can get hold of a program called "cool edit" you
can see this in color on whatever audio you feed in,
even real time audio. Still useless IMHO.

btw the link is dead.

peace
Paul Revere

"Mason C" <masonc2@earthlink.net> wrote in message
news:san7c2hq2afemr8h6joktoji3jf2on699g@4ax.com...
: Just to toss a pebble into this water:
:
: Certainly an *imperfect* phoneme-recognition
program could be
: used to create a visible representation much more
readable
: than variant on spectograms?
:
: Mason C

Michael Ridenhour

2006-07-24, 9:25 pm

I interpreted into sign language normal college classroom lectures and
activities. Sorry if my wording was imprecise. I didn't mean to upset you.

Michael

"Paul Revere" <PaulRrev@impeach_the_efffin_chimp.gov> wrote in message
news:QX2xg.4156$157.2727@newsread3.news.pas.earthlink.net...
> So you sat in a room with a person signing ( presumably
> in the same room where the person speaking was speaking
> then this signing person converts that to sign
> language.......go on) then YOU read the sign language
> and yelled it into the deaf guys ear. What the heck are
> you talking about? Why would anyone need an
> interpreter to interpret sign language?
>
> "Michael Ridenhour" <ocie@medbush.net> wrote in message
> news:yfPwg.103637$IZ2.43179@dukeread07...
> :
> :
>
> :
> : Years ago I interpreted sign language for a man who
> had entered a community
> : college and needed a two week interpreter. I had some
> sign, but was very
> : slow. After the two weeks, however, I was amazed that
> even fingerspelling
> : (it was a college level class, so lots of
> fingerspelling) became words, not
> : particles of words. What I mean is, I saw the word,
> not the fingerspelled
> : letters after a few weeks. Lost it very quickly when
> I stopped, though.
> : Maybe your plan would work, though now I'm sure I'm
> too old.
> :
> : Michael
> :
> :
>
>

Paul Revere

2006-07-25, 4:26 pm

Mike:

I am not upset. What would make you think your post
would upset me? You confused me.

I interpret what whomever was asking about as what one
person, who could not hear well, needs to do to hear
better. IE. READ the spectrograph (lips, sign language,
a teleprompter). Your statement was more like how to
SPEAK in sign not read it.

I take it your short experience in performing at
signing is like what any person who types, plays music,
or plays sports finds after getting mildly proficient
at it, you can easily sequence small movements into
larger tasks without focusing on the minutia of the
individual parts.

If you can get cool edit and hook it to the TV and
watch the spectrogram. I suppose you feel that after a
long enough exposure to the spectrograph running
synchronized to what you are hearing you will be able
to turn the sound off and just watch the spectrograph
to "hear" the dialog. Let us know how you fair on it.
Maybe you can be the one who finds a use for this
virtually useless display. I think the reason that
they put it there was to help mixers visualize timbre
and perhaps guide them in matching the timbre of
something they are attempting to punch in (re record a
small part of something while keeping the greater part
of it)

There are others displays for sound pressure that maybe
interesting to you. Google Sepstrum. It is another
convolved version of audio that can be produced but no
one I can find has found any practical use for it.

peace

Paul Revere

"Michael Ridenhour" <ocie@medbush.net> wrote in message
news:Myexg.103728$IZ2.47858@dukeread07...
: I interpreted into sign language normal college
classroom lectures and
: activities. Sorry if my wording was imprecise. I
didn't mean to upset you.
:
: Michael

:
: "Paul Revere" <PaulRrev@impeach_the_efffin_chimp.gov>
wrote in message
:
news:QX2xg.4156$157.2727@newsread3.news.pas.earthlink.net...
: > So you sat in a room with a person signing (
presumably
: > in the same room where the person speaking was
speaking
: > then this signing person converts that to sign
: > language.......go on) then YOU read the sign
language
: > and yelled it into the deaf guys ear. What the heck
are
: > you talking about? Why would anyone need an
: > interpreter to interpret sign language?
: >
: > "Michael Ridenhour" <ocie@medbush.net> wrote in
message
: > news:yfPwg.103637$IZ2.43179@dukeread07...
: > :
: > :
: >
: > :
: > : Years ago I interpreted sign language for a man
who
: > had entered a community
: > : college and needed a two week interpreter. I had
some
: > sign, but was very
: > : slow. After the two weeks, however, I was amazed
that
: > even fingerspelling
: > : (it was a college level class, so lots of
: > fingerspelling) became words, not
: > : particles of words. What I mean is, I saw the
word,
: > not the fingerspelled
: > : letters after a few weeks. Lost it very quickly
when
: > I stopped, though.
: > : Maybe your plan would work, though now I'm sure
I'm
: > too old.
: > :
: > : Michael
: > :
: > :
: >
: >
:
:

Paul Revere

2006-07-25, 4:26 pm

Come to think of it Mike, seeing voice with
spectrographs is kind of like this. Just because
someone can fart Dixie does not mean they can tell the
difference between southern fried chicken and chicken
cor-don-blu with their butt.

peace
Paul Revere

"Paul Revere" <PaulRrev@impeach_the_efffin_chimp.gov>
wrote in message
news:gArxg.4535$bP5.3811@newsread1.news.pas.earthlink.net...
: Mike:
:
:
:
: I am not upset. What would make you think your post
: would upset me? You confused me.
:
:
:
: I interpret what whomever was asking about as what
one
: person, who could not hear well, needs to do to hear
: better. IE. READ the spectrograph (lips, sign
language,
: a teleprompter). Your statement was more like how to
: SPEAK in sign not read it.
:
:
:
: I take it your short experience in performing at
: signing is like what any person who types, plays
music,
: or plays sports finds after getting mildly proficient
: at it, you can easily sequence small movements into
: larger tasks without focusing on the minutia of the
: individual parts.
:
:
:
: If you can get cool edit and hook it to the TV and
: watch the spectrogram. I suppose you feel that after
a
: long enough exposure to the spectrograph running
: synchronized to what you are hearing you will be able
: to turn the sound off and just watch the spectrograph
: to "hear" the dialog. Let us know how you fair on it.
: Maybe you can be the one who finds a use for this
: virtually useless display. I think the reason that
: they put it there was to help mixers visualize timbre
: and perhaps guide them in matching the timbre of
: something they are attempting to punch in (re record
a
: small part of something while keeping the greater
part
: of it)
:
:
:
: There are others displays for sound pressure that
maybe
: interesting to you. Google Sepstrum. It is another
: convolved version of audio that can be produced but
no
: one I can find has found any practical use for it.
:
:
:
: peace
:
: Paul Revere
:
:
:
:
:
:
:
: "Michael Ridenhour" <ocie@medbush.net> wrote in
message
: news:Myexg.103728$IZ2.47858@dukeread07...
: : I interpreted into sign language normal college
: classroom lectures and
: : activities. Sorry if my wording was imprecise. I
: didn't mean to upset you.
: :
: : Michael
:
:
: :
: : "Paul Revere"
<PaulRrev@impeach_the_efffin_chimp.gov>
: wrote in message
: :
:
news:QX2xg.4156$157.2727@newsread3.news.pas.earthlink.net...
: : > So you sat in a room with a person signing (
: presumably
: : > in the same room where the person speaking was
: speaking
: : > then this signing person converts that to sign
: : > language.......go on) then YOU read the sign
: language
: : > and yelled it into the deaf guys ear. What the
heck
: are
: : > you talking about? Why would anyone need an
: : > interpreter to interpret sign language?
: : >
: : > "Michael Ridenhour" <ocie@medbush.net> wrote in
: message
: : > news:yfPwg.103637$IZ2.43179@dukeread07...
: : > :
: : > :
: : >
: : > :
: : > : Years ago I interpreted sign language for a man
: who
: : > had entered a community
: : > : college and needed a two week interpreter. I
had
: some
: : > sign, but was very
: : > : slow. After the two weeks, however, I was
amazed
: that
: : > even fingerspelling
: : > : (it was a college level class, so lots of
: : > fingerspelling) became words, not
: : > : particles of words. What I mean is, I saw the
: word,
: : > not the fingerspelled
: : > : letters after a few weeks. Lost it very quickly
: when
: : > I stopped, though.
: : > : Maybe your plan would work, though now I'm sure
: I'm
: : > too old.
: : > :
: : > : Michael
: : > :
: : > :
: : >
: : >
: :
: :
:
:

artis

2006-07-25, 4:26 pm

I think, Paul, you are mistaking me for someone else. As I stated in a
previous post, I do not believe one can read a spectrogram and understand it
as language.

Mike

"Paul Revere" <PaulRrev@impeach_the_efffin_chimp.gov> wrote in message
news:nOrxg.4538$bP5.1013@newsread1.news.pas.earthlink.net...
> Come to think of it Mike, seeing voice with
> spectrographs is kind of like this. Just because
> someone can fart Dixie does not mean they can tell the
> difference between southern fried chicken and chicken
> cor-don-blu with their butt.
>
> peace
> Paul Revere
>
>
> "Paul Revere" <PaulRrev@impeach_the_efffin_chimp.gov>
> wrote in message
> news:gArxg.4535$bP5.3811@newsread1.news.pas.earthlink.net...
> : Mike:
> :
> :
> :
> : I am not upset. What would make you think your post
> : would upset me? You confused me.
> :
> :
> :
> : I interpret what whomever was asking about as what
> one
> : person, who could not hear well, needs to do to hear
> : better. IE. READ the spectrograph (lips, sign
> language,
> : a teleprompter). Your statement was more like how to
> : SPEAK in sign not read it.
> :
> :
> :
> : I take it your short experience in performing at
> : signing is like what any person who types, plays
> music,
> : or plays sports finds after getting mildly proficient
> : at it, you can easily sequence small movements into
> : larger tasks without focusing on the minutia of the
> : individual parts.
> :
> :
> :
> : If you can get cool edit and hook it to the TV and
> : watch the spectrogram. I suppose you feel that after
> a
> : long enough exposure to the spectrograph running
> : synchronized to what you are hearing you will be able
> : to turn the sound off and just watch the spectrograph
> : to "hear" the dialog. Let us know how you fair on it.
> : Maybe you can be the one who finds a use for this
> : virtually useless display. I think the reason that
> : they put it there was to help mixers visualize timbre
> : and perhaps guide them in matching the timbre of
> : something they are attempting to punch in (re record
> a
> : small part of something while keeping the greater
> part
> : of it)
> :
> :
> :
> : There are others displays for sound pressure that
> maybe
> : interesting to you. Google Sepstrum. It is another
> : convolved version of audio that can be produced but
> no
> : one I can find has found any practical use for it.
> :
> :
> :
> : peace
> :
> : Paul Revere
> :
> :
> :
> :
> :
> :
> :
> : "Michael Ridenhour" <ocie@medbush.net> wrote in
> message
> : news:Myexg.103728$IZ2.47858@dukeread07...
> : : I interpreted into sign language normal college
> : classroom lectures and
> : : activities. Sorry if my wording was imprecise. I
> : didn't mean to upset you.
> : :
> : : Michael
> :
> :
> : :
> : : "Paul Revere"
> <PaulRrev@impeach_the_efffin_chimp.gov>
> : wrote in message
> : :
> :
> news:QX2xg.4156$157.2727@newsread3.news.pas.earthlink.net...
> : : > So you sat in a room with a person signing (
> : presumably
> : : > in the same room where the person speaking was
> : speaking
> : : > then this signing person converts that to sign
> : : > language.......go on) then YOU read the sign
> : language
> : : > and yelled it into the deaf guys ear. What the
> heck
> : are
> : : > you talking about? Why would anyone need an
> : : > interpreter to interpret sign language?
> : : >
> : : > "Michael Ridenhour" <ocie@medbush.net> wrote in
> : message
> : : > news:yfPwg.103637$IZ2.43179@dukeread07...
> : : > :
> : : > :
> : : >
> : : > :
> : : > : Years ago I interpreted sign language for a man
> : who
> : : > had entered a community
> : : > : college and needed a two week interpreter. I
> had
> : some
> : : > sign, but was very
> : : > : slow. After the two weeks, however, I was
> amazed
> : that
> : : > even fingerspelling
> : : > : (it was a college level class, so lots of
> : : > fingerspelling) became words, not
> : : > : particles of words. What I mean is, I saw the
> : word,
> : : > not the fingerspelled
> : : > : letters after a few weeks. Lost it very quickly
> : when
> : : > I stopped, though.
> : : > : Maybe your plan would work, though now I'm sure
> : I'm
> : : > too old.
> : : > :
> : : > : Michael
> : : > :
> : : > :
> : : >
> : : >
> : :
> : :
> :
> :
>
>

Paul Revere

2006-07-25, 4:26 pm

sorry if I offended you then, but I think we are just
talking here anyhow.

peace

"artis" <artis5@farscape.com> wrote in message
news:p8uxg.2830$5H.2084@dukeread06...
: I think, Paul, you are mistaking me for someone else.
As I stated in a
: previous post, I do not believe one can read a
spectrogram and understand it
: as language.
:
:
: Mike
:
: "Paul Revere" <PaulRrev@impeach_the_efffin_chimp.gov>
wrote in message
:
news:nOrxg.4538$bP5.1013@newsread1.news.pas.earthlink.net...
: > Come to think of it Mike, seeing voice with
: > spectrographs is kind of like this. Just because
: > someone can fart Dixie does not mean they can tell
the
: > difference between southern fried chicken and
chicken
: > cor-don-blu with their butt.
: >
: > peace
: > Paul Revere
: >
: >
: > "Paul Revere"
<PaulRrev@impeach_the_efffin_chimp.gov>
: > wrote in message
: >
news:gArxg.4535$bP5.3811@newsread1.news.pas.earthlink.net...
: > : Mike:
: > :
: > :
: > :
: > : I am not upset. What would make you think your
post
: > : would upset me? You confused me.
: > :
: > :
: > :
: > : I interpret what whomever was asking about as
what
: > one
: > : person, who could not hear well, needs to do to
hear
: > : better. IE. READ the spectrograph (lips, sign
: > language,
: > : a teleprompter). Your statement was more like how
to
: > : SPEAK in sign not read it.
: > :
: > :
: > :
: > : I take it your short experience in performing at
: > : signing is like what any person who types, plays
: > music,
: > : or plays sports finds after getting mildly
proficient
: > : at it, you can easily sequence small movements
into
: > : larger tasks without focusing on the minutia of
the
: > : individual parts.
: > :
: > :
: > :
: > : If you can get cool edit and hook it to the TV
and
: > : watch the spectrogram. I suppose you feel that
after
: > a
: > : long enough exposure to the spectrograph running
: > : synchronized to what you are hearing you will be
able
: > : to turn the sound off and just watch the
spectrograph
: > : to "hear" the dialog. Let us know how you fair on
it.
: > : Maybe you can be the one who finds a use for this
: > : virtually useless display. I think the reason
that
: > : they put it there was to help mixers visualize
timbre
: > : and perhaps guide them in matching the timbre of
: > : something they are attempting to punch in (re
record
: > a
: > : small part of something while keeping the greater
: > part
: > : of it)
: > :
: > :
: > :
: > : There are others displays for sound pressure that
: > maybe
: > : interesting to you. Google Sepstrum. It is
another
: > : convolved version of audio that can be produced
but
: > no
: > : one I can find has found any practical use for
it.
: > :
: > :
: > :
: > : peace
: > :
: > : Paul Revere
: > :
: > :
: > :
: > :
: > :
: > :
: > :
: > : "Michael Ridenhour" <ocie@medbush.net> wrote in
: > message
: > : news:Myexg.103728$IZ2.47858@dukeread07...
: > : : I interpreted into sign language normal college
: > : classroom lectures and
: > : : activities. Sorry if my wording was imprecise.
I
: > : didn't mean to upset you.
: > : :
: > : : Michael
: > :
: > :
: > : :
: > : : "Paul Revere"
: > <PaulRrev@impeach_the_efffin_chimp.gov>
: > : wrote in message
: > : :
: > :
: >
news:QX2xg.4156$157.2727@newsread3.news.pas.earthlink.net...
: > : : > So you sat in a room with a person signing (
: > : presumably
: > : : > in the same room where the person speaking
was
: > : speaking
: > : : > then this signing person converts that to
sign
: > : : > language.......go on) then YOU read the sign
: > : language
: > : : > and yelled it into the deaf guys ear. What
the
: > heck
: > : are
: > : : > you talking about? Why would anyone need an
: > : : > interpreter to interpret sign language?
: > : : >
: > : : > "Michael Ridenhour" <ocie@medbush.net> wrote
in
: > : message
: > : : > news:yfPwg.103637$IZ2.43179@dukeread07...
: > : : > :
: > : : > :
: > : : >
: > : : > :
: > : : > : Years ago I interpreted sign language for a
man
: > : who
: > : : > had entered a community
: > : : > : college and needed a two week interpreter.
I
: > had
: > : some
: > : : > sign, but was very
: > : : > : slow. After the two weeks, however, I was
: > amazed
: > : that
: > : : > even fingerspelling
: > : : > : (it was a college level class, so lots of
: > : : > fingerspelling) became words, not
: > : : > : particles of words. What I mean is, I saw
the
: > : word,
: > : : > not the fingerspelled
: > : : > : letters after a few weeks. Lost it very
quickly
: > : when
: > : : > I stopped, though.
: > : : > : Maybe your plan would work, though now I'm
sure
: > : I'm
: > : : > too old.
: > : : > :
: > : : > : Michael
: > : : > :
: > : : > :
: > : : >
: > : : >
: > : :
: > : :
: > :
: > :
: >
: >
:
:

topmind

2006-08-07, 2:25 am

Paul Revere wrote:
> If you can get hold of a program called "cool edit" you
> can see this in color on whatever audio you feed in,
> even real time audio. Still useless IMHO.
>
> btw the link is dead.

It still seems fine to me. Maybe the URL is being scrambled by your
newsgroup browser. Try googling this phrase:

Spectrogram Reading Tim Carmell

Then click on the "Spectragrams" icon.

>
>
> peace
> Paul Revere
>
>

-T-

topmind

2006-08-07, 2:25 am

Mason C wrote:
> Just to toss a pebble into this water:
>
> Certainly an *imperfect* phoneme-recognition program could be
> used to create a visible representation much more readable
> than variant on spectograms?
>
> Mason C

It is possible to do *both* I suppose. Estimated phenomes could be
shown at the top or bottom of the spectrogram as they are computed. I
just wonder if the trainee would end up using the computer-estimated
phenomes as a crutch to avoid learning to read the spectrogram.

But this is all hypothetical and we won't know until it is actually
tried on multiple individuals. Hopefully somebody with a budget will
try it. (Although computing power is getting cheap enough for somebody
to try it on their own.)

Take care,

-T-

Paul Revere

2006-08-07, 8:23 am

I have added Carmell's link to my page
http://philsaudio.com/spectrographs.htm

I read Carmell's link and it appears to me that there
is NO claim in this paper that anyone can "read the
spectrogram" with any possibility of extracting the
meaning of the words being processed and displayed by
the spectrograph display. I see activity and ideas
popping from the pictures this technology generates,
but no useable results from a human using the tools.

In my mind this is nothing more than another solution
looking for application other than what it was designed
for. Spectrographs were developed to match one voice
(sound) sample to another more like a "voice
fingerprint" of that sounds source, not identification
of the intelligence contained in the sound being
produced by the source.

I believe the original application for the spectrogram
was to develop a tool that could identify the sound of
submarine screws to determine class and perhaps the
actual ship being tracked by another sub/ship.

I hope I am being too pessimistic.

peace
Phil.
"topmind" <topmind@technologist.com> wrote in message
news:1154935080.374660.197790@i42g2000cwa.googlegroups.com...
:
: Paul Revere wrote:
: > If you can get hold of a program called "cool edit"
you
: > can see this in color on whatever audio you feed
in,
: > even real time audio. Still useless IMHO.
: >
: > btw the link is dead.
:
: It still seems fine to me. Maybe the URL is being
scrambled by your
: newsgroup browser. Try googling this phrase:
:
: Spectrogram Reading Tim Carmell
:
: Then click on the "Spectragrams" icon.
:
: >
: >
: > peace
: > Paul Revere
: >
: >
:
: -T-
: