A device to help the blind, anybody think it will work?

[Balduran]

Imagine wearing a very wide headband*. On this headband, there is a grid-work of small pins, not sharp, but you would feel them if they were pressed into the skin. You wear these glasses which are actually a couple of small black and white cameras. The images from these cameras are then digitized and sent to the grid of pins. The digitized images are translated by the grid such that the darker the pixel of the image, the further the corresponding pin is pressed into your skin.

So essentially the images picked up from the cameras will be outlined on your skin.

Two questions arise:
1. Do you think your skin can be trained to be sensitive enough to be able to discerne a fairly fine grid, say 600x600 points?

I have run my fingers over braille before and I honestly can't tell the difference between 3 or 4 bumps let alone their pattern. Thus I concluded that your sense of touch can be sensitized to this with enough practice.

2. With enough training, will it be possible for you brain to make sense of this device so that the outlines actually form "images" in your mind? Eventually to become as second nature to you as seeing?

I have read stories about people blind from birth who have somehow gained sight later on in life. In the beginning, visual information is just a jumble, but eventually the brain starts to figure it out. If you think about the brain as an information processor, it shouldn't matter which input the information comes from.


Anybody think this can work, or suggest improvements?


*The pin-prick grid does not necessarily need to be worn on the forehead, maybe a different part of the body where there is more surface area to draw.

[Hari Seldon]

I have read about these devices and I see no reason they wouldn't work, but I also see no way of finding out without trying it. As you say, you can't "see" braille and neither can I, but obviously blind people do. And I have read that the part of the cerebral cortex connected to the braille reading finger expands enormously (taking over a large part of the visual cortex). I would try it on children born blind since their brains would be most plastic.

[NoGoodNamesLeft]

I'm not sure. The camera would have to take still images or be held absolutley still to avoid the 'image' output from being a constantly moving mass of pins. (Our brains stabalize these images for us, but go look through a pair of binoculars to see this effect when its taken away).
Also, I'm not sure there would be a way to represent 3-dimensional vision using cameras.

The general idea may be good for reading, though. What if a scanner worn on the hand could 'capture' text and output it against the skin using the pins.

[Mangetout]

There are parts of the body where the resolution of sensitivity would not be sufficient due to the sparse spacing of nerve endings - I remember testing this in biology class with two pointed sticks; on certain parts of the body (the outer thighs and buttocks, for example), it was not possible to differentiate between being poked with one or two sticks, even if they were several centimetres apart.

However, a device that fits like a glove and stimulates the skin on the last three joints of your middle three fingers might work (no real reason why the area of skin being stimulated need be contiguous)

[starman]

Why not do it direct? plant a ccd to directly stimulate optic nerve. Would this work? If I recall correctly, the humours of the eye are basically an immune free zone, so I shouldn't think you'd have problems with rejection.
-Oli

[Balduran]

Wow, you guys are restoring my faith in the Straight dope. Lots of good comments already.

Hari, you have heard of these devices before? Have they actually been manufactured?

NoGoodNamesLeft Very good points. I would think there must be some image stabilation software used for some video cameras, but it would certainly have to be quite good for this application.

I was thinking for 3-D, if you have the two cameras, you could possibly process it so that the closer the object is, the further the pin is pressed. Thus you couldn't distinguish so well between dark lines and lines that are close. But our sight also plays tricks on us. Another possibility I was thinking of was that each pin could send a minor jolt of electricity (just enough to feel a slight vibration). By adjusting the strength and frequencies of these vibrations, you could convey more information. Don't know how localised you could get that though.

Mangetout, exactly what I was thinking. I would imagine the brain could make sense of the information no matter where it was coming from on the body. Might was well find a place where it is the most sensitive.

starman, that would be excellent, and is the ultimate goal. But I don't have the expertise to know if that would be possible with today's technology. I thought this device may be workable in the meantime.

[Phage]

starman,
Good idea... already taken tho. I remember reading a couple years ago that someone had already tried that with something like a 12x12 grid. The man obviously lacked resolution, but he could detect changes in light level which could be fairly useful. With that as a proof-of-concept larger grids were being planned; maybe a 1600x1200 grid would be useful...

[Earl Snake-Hips Tucker]

There's an organization called the Dobelle Institute which is trying to develop artificial vision. Don't know whether it's on the up-and-up or just snake oil.

[flight]

There has been the statement that the use of the pin idea would not be able to detect distance. Why not? Just have two cameras positioned side by side (as in the human eyes) that use different sets of pins. If the concepet is valid you could have these two different grids at very different locations (though symmetry would be a good idea for constant resolution). If our brain can develop the ability to translate pinpricks into a visual cue, it should also be able to resolve the two "images".

[micco]

Regarding starman's post, I noticed this news item today:
http://www.wired.com/news/medtech/0,1286,59634,00.html

[homercles]

What youre talking about is a heavily researched area, namely that of computer vision. How do you represent the 7 dimensional space, in the lower-limit i should add, of vision as just 3 dimensions? In the 70s computer vision folks made some rather bold claims, suggesting they would have it figured out in 10 years. Here we are 30 years later, with hardly a shred of an efficient computer vision algorithm.

[Nanoda]

micco found a good article about the retinal implant. Of course, sometimes the problem is with the retina, and you have to go a bit furthur along.

For the OP: I've heard about similar devices (well, ditch the pins and use electrical stimulation, similar thing) that are worn on the chest (more area/sensitivity, more definition). According to this article of a couple of years ago, they've moved on to the tongue (seeing as how it's a very dense nerve center.

So it doesn't seem like you're gonna get a grant for this, but you have managed to duplicate some of the latest ideas used in this field. (No wait, more apropos: )

[micco]

Originally posted by homercles
Here we are 30 years later, with hardly a shred of an efficient computer vision algorithm.

We're talking about two very distinct fields with very different problems. The human visual system could be described as two parts: perception and cognition. In human vision, these two systems are tightly interwoven and if you tried to draw a boundary there would be a lot of argument about where that boundary is, but basically you have something which detects signals and something which interprets them.

The computer vision algorithms you mention are mostly on the interpretive side. It's not too much of a problem to outfit a machine with cameras to serve as the perceptual side, but it's much more difficult to take that data and produce 3D mappings and other information that a machine would need for decisions. That is, the software analogous to our cognitive systems are the hard problem.

On the other hand, the OP is talking about the perceptual side. In this case, we already have the "software" in the human brain to do the analysis of the data, and the problem is getting the data input when the perceptual elements fail. As Nanoda mentions, the point of failure may be a real issue because the cognitive processes begin in the optic nerve before the signals even get to the brain, so if the failure of the system is more than just the retina (the input device) then it's not purely a perceptual issue. In any case, input devices like the OP suggests are completely separate issues than the analysis software you seem to be referring to.

[Balduran]

Cool links folks.

That's okay Nanoda, I'm just happy to know research is being done in this area that may yield good results. I've got lot's of lame, useless ideas that I will use to make my fortune

[homercles]

Originally posted by micco
On the other hand, the OP is talking about the perceptual side. In this case, we already have the "software" in the human brain to do the analysis of the data, and the problem is getting the data input when the perceptual elements fail.

THat is exactly what im talking about. Your "perceptual" side. In my previous post i mentioned that there are 7 important dimensions that the visual system processes at the earliest levels of the cortex--V1. But, in actuality, its more like 9:
<space x, space y, space z, time, orientation, spatial frequency, highly reduced color (LM, S, Lum)>

Pins, electrodes, whathave you can only crudly encode 3 of these dimensions:
<space x, space y, intensity/Lum/space z/whatever>

Understand?

As Nanoda mentions, the point of failure may be a real issue because the cognitive processes begin in the optic nerve before the signals even get to the brain, [/b]

Source, please. This is just preposterous. Sure, there are a number of feedback projections from V1, V2, etc, but this is a far cry from claiming that "cognitive processes begin in the optic nerve..."

Also, from that link: “a very limited navigational aid, and a far cry from the visual experience that normal people enjoy.” From what i read, it seems that all they have been able to do is give Jerry soem VERY crude info about <space x, space y>.

[micco]

Originally posted by homercles
THat is exactly what im talking about. Your "perceptual" side. In my previous post i mentioned that there are 7 important dimensions that the visual system processes at the earliest levels of the cortex--V1. But, in actuality, its more like 9:
<space x, space y, space z, time, orientation, spatial frequency, highly reduced color (LM, S, Lum)>

Pins, electrodes, whathave you can only crudly encode 3 of these dimensions:
<space x, space y, intensity/Lum/space z/whatever>

Understand?

I do understand. When you talked about "computer vision algorithms" I assumed you were talking about the extensive work in the field of computer science and artificial intelligence which uses that term to describe the software processes of interpreting and using optical data. These are analogous to the cognitive processes in humans, not the perceptual ones. I understand your point that the perceptual system has to provide adequate inputs (the 7 or 9 dimensions you discuss), but that is separate from what I would call the "computer vision algorithms" processing those inputs into useful information. If we disagree on this, it is merely on semantics.

Source, please. This is just preposterous. Sure, there are a number of feedback projections from V1, V2, etc, but this is a far cry from claiming that "cognitive processes begin in the optic nerve..."

Sorry, I don't have a source handy. I will look for one but I thought this was pretty basic stuff. I am not an experimental psychologist, but I hang out with a bunch of them who work on visual processing, pattern recognition, etc. and I have heard them discuss many times the various signal processing that goes on before the visual data reaches the brain. I tend to translate what they say into machine analogies I can understand, and I translated their discussions to mean that there was significant processing in terms of frequency filtering and pattern recognition very early in the "hardware". I apologize in advance if I'm wrong, and I will ask them for a cite to back up my interpretation.

[homercles]

Originally posted by micco
I do understand. When you talked about "computer vision algorithms" I assumed you were talking about the extensive work in the field of computer science and artificial intelligence which uses that term to describe the software processes of interpreting and using optical data. These are analogous to the cognitive processes in humans, not the perceptual ones. I understand your point that the perceptual system has to provide adequate inputs (the 7 or 9 dimensions you discuss), but that is separate from what I would call the "computer vision algorithms" processing those inputs into useful information. If we disagree on this, it is merely on semantics.

No, actually, it was probably my overly generalized wording that caused confusion. Let me explain furhter (but i should mention that Im a vision scientist who works with and develops both computer and human vision algorithms. So, if somethings not clear just feel free to ask). The processing of the high dimensional space i described would be akin to "image segmentation" in the computer vision circles. The human visual system does this image segmentation with no "cognitive" input (we'll just ignore some of the obvious examples of top-down processing cog psy folks love to show in classes). Now, most image segmentation algorithms that i know of only rely upon a 3 dimensional input (usually a grey scale image). And some of them can do "ok," but only with certain types of images. The problem is that as dimensionality increases, so does computational cost and complexity. But its still the same issue that the human visual system deal with, "you gotta know that something is there before you try to determine what it is." Determining what it is, is the more cog side which is akin to object recognition or discrimination AI. Pretty standard computer vision stuff, IMHO.

and I translated their discussions to mean that there was significant processing in terms of frequency filtering and pattern recognition very early in the "hardware". I apologize in advance if I'm wrong, and I will ask them for a cite to back up my interpretation.

No, this is also standard stuff. In a simplisitic manner, there are basically two major stages:
1. receptors (tranduction of quanta)
2. "reorganization for data compression"

I will just address the second part. The receptors have a number of cells of various types that are primarily concerned with "reorganizing" (actually there is a hell of alot more going on than this, but lets keep it simple) the incoming singnals into representations that make it possible for the signals from 100500000 rods and cones to be sent down a meager 1500000 optic nerve fibers. [aside] one of my qualifying exam members actually developed an image compression algorithm based on what the retina does [/aside] This converged signal is later diverged at the cortex which has 1.00e+010 neurons. The 9 dimensional space im referring to is the representation seen in the very first cortical area (V1), not the retina->LGN stage. An area, that void of any "cognitive influences." So, yes, youre right there is a ton of "pre-processing" that goes on before info has reached the cortex, its just not "cognitive processing."

[Chronos]

I'm not sure about seven dimensions, there: I only count three dimensions, repeated six times. Each eye detects X position, Y position, and time, in three channels which give the colors. Put another way, you can reduce everything a person sees to two TV screens, each of which can be reduced to three monochrome screens. The other "dimensions" you mention, like spatial frequencies, are just non-independent reprocessings of those six three-dimensional images. Two devices like the one described in the OP would give all the same information as black-and-white human vision, and you could add "color" information easily enough, perhaps with flattened pins oriented in three different directions, or the like. The same brain which can deduce spatial frequencies from visual information should, in principle, be able to process that same information from tactile stimuli.

[micco]

Originally posted by homercles
there is a ton of "pre-processing" that goes on before info has reached the cortex, its just not "cognitive processing."

My bad. In translating the psych discussions to the metaphors I use, I'd equated perception to data acquisition and cognition to signal processing. This is obviously too simplistic. Sorry to have muddied the waters.

[homercles]

Originally posted by Chronos
I'm not sure about seven dimensions, there: I only count three dimensions, repeated six times. Each eye detects X position, Y position, and time, in three channels which give the colors. Put another way, you can reduce everything a person sees to two TV screens, each of which can be reduced to three monochrome screens. The other "dimensions" you mention, like spatial frequencies, are just non-independent reprocessings of those six three-dimensional images. Two devices like the one described in the OP would give all the same information as black-and-white human vision, and you could add "color" information easily enough, perhaps with flattened pins oriented in three different directions, or the like. The same brain which can deduce spatial frequencies from visual information should, in principle, be able to process that same information from tactile stimuli.

Well, im not sure what youre referencing without quoting me (hell, im not even sure if youre referencing my posts about cortical implants or retinal implants, which it appears you are confusing), but i will try. At the VERY FIRST level of the visual cortex there are at least 9 dimensions represented. If you seek to build a tactile equivalent, yes, you could throw out the color information and just talk about greyscale images in x and y. Now naturally, a third dimension--time--falls out this as the system is dynamic. But the sheer magnitude of the information across x, y, and t is simply too large to hope to create such a tactile equivalent. Hence, the reson for these terribly low res systems. Are you aware of the fact that between 33-45% of your entire cortex is devoted to processing visual information? Not to mention there is no way to represent depth. How would you know the difference between a big box far away or small one close? Furthermore, my suggestion that spatial frequency and orientation should be mentioned is that tactile receptors are nothing like visual receptors or subsequent visual processing. The two simply dont jibe. Its like saying that you should be able to build a system that would allow somone to smell by ear. You cant override millions of years of evolution with some simple engineering. And i havent even touched sensroy adaptation (both systems have very different mechanisms for adaptation). If these two systems did jibe all we would need for blind reading would be "raised letters." Have you ever tried to read by feeling raised letters? It simply doesnt work. Visual information is simply too massive to hope to represent it in another sensory system not accustomed to processing n-Dimensional spaces. If youre talking about retinal implants, the amount of information coming out of a single retina is effectively at least 7 dimensional:

<space x, space y, time, weak spatial frequency, 3 dimensions of color>

all multi-plexed onto a minimun of 3 cell types. The final two dimensions are created at the cortex.

Disclaimer: as i have said before i am a vision scientist and in the community prostheses are "the holy grail." I applaud those folks working on this stuff (i even have a close friend working on retinal implants at Wyane State), just dont expect to see any functioning prostheses in your children's lifetimes. Because as im saying, this is a profoundly complicated matter.

[KidCharlemagne]

I saw this experimental technology for the blind on the Discovery Channel that was something like what some of you are talking about but it's aural rather than tactile. A camera creates a low res picture of the field of view and assigns a major scale tone to varying shades of light and dark. A frame is then "played" from right to left. They had blind people using the device describe what was in front of them and they did a pretty damn good job of getting the rough idea.

[mnemosyne]

http://www.uoguelph.ca/mediarel/archives/001718.html

I haven't read this whole thread yet, but the subject reminded me of this article....sounds like what you might be interested in.

[even sven]

I don't think depth is that important. I am practically blind in one eye, and it no way affects me. I'm bad at playing baseball, but I have an otherwise normal understanding of space and vision. The mind has a lot of power to understand context.

[Chronos]

Except that depth and color aren't, mathematically speaking (which is what's relevant for the amount of information) dimensions in this argument. Each of our eyes can detect only two spatial dimensions. Depth, we inferr by comparing the information from the two eyes. But the total amount of information is still only two eyes worth of two dimensions of spatial information.

Likewise, our color perception does not contribute even one full dimension, much less three. For color to contribute even one dimension, we would need to have full Fourier resolution of wavelengths of light, like we do for sound with hearing. To get more than one dimension, we'd have to extend it to something (I know not what) which would be beyond anything we could remotely think of as color.

To elaborate: Suppose I had six computer screens, of high spatial resolution. On screen one, I show, in monochrome, the red component of what I see in my left eye. On screen 2, I similarly show the green component of my left eye, and on the third, I show the blue of my left eye. Screens 4, 5, and 6 show the red, green, and blue components of my right eye's vision. You agree, do you not, that these six screens represent the complete amount of information I'm receiving through my eyes? But counting time, those computer screens are only three dimensional. And just the fact that there are six computer screens doesn't change the fact that the total of all of them together is still only three dimensional. This is the same as the number of dimensions we could achieve with a pin array pushing on skin.

Now, I don't know what the "resolution", so to speak, of human skin is, and it's quite possible that you can't get enough information this way for a good image. And I likewise don't know if the brain could adapt to getting this information from a different source. But I can say, at least, that the skin has enough dimensions of information.

As for "smelling by ear", the ear picks up two dimensions of information (frequency and time), whereas I don't know how many dimensions of information smelling entails (I'm pretty sure it's at least two, but I don't know if it might be more).

[treis]

I am a bit confused as to how I see time.

[Balduran]

I like your link mnemosyne, especially since it is from one of my old schools. They also agree with something I was thinking about with these devices which help you navigate by sound, which is that it would be difficult because you are always trying to hear other things too. Ideally you would like to have an input system which does not conflict too badly with your other standard sensory inputs.

I'm guessing that while Chronos is correct about the strict mathematical definitions of the dimensionality of the system. I suspect homercles is using a colloquial definition which is more useful in his field.

The extra dimensions due to color are because the eye uses basically a RGB system right? I don't see any reason why you would have to build a system based on these if you are not actually using sight. Perhaps, if you are using electrical impulses on the skin, you could just use a simple linear range of frequency of the pulses to represent the color.

treis, the time dimensionality is just that objects may be moving (changing position in time). Thus, in the mathematical sense, time is a dimension.

Good point sven, context is very important. I use both eyes all the time, but I can recognize distances of objects when I close one eye.

[treis]

treis, the time dimensionality is just that objects may be moving (changing position in time). Thus, in the mathematical sense, time is a dimension.

I still don't understand how I see time. My eyes take in the visual data and my brain percieves the motion by time. The same way I don't feel time when someone runs their finger up my arm. I am recieving input from different nerves and my brain says its moving.

[treis]

Ugh that didn't make sense. My eyes take in the visual data and my brain percieves the motion by comparing the images.

[easy e]

Researchers at my alma mater want to do this same thing, but provide the input to the tongue.