I can’t say for sure what they were trying to demonstrate about quantum physics…if you wanted to know what it demonstrates in fluid mechanics, I’d have a better idea.
As for “how does it work,” I can tackle that for you.
The cylinders, as you describe, are made to rotate relative to one another (doesn’t matter which one is doing the moving, it’s all about their relative motion). For this example, lets assume the outer cylinder reamins stationary, and the inner cyliner is rotating. The glycerin filling the gap wants to stay attached to the surfaces of both cylinders, and it does. The fluid touching the outer cylinder stays with that surface, and remains still. The glycerin in contact with the inner cylinder says in contact with that wall, and rotates along with it. But the fluid in the middle has to help make the transition, and so moves at various speeds through its thickness. As you move through the glycerin from the outer, stationary wall, the fluid will be rotating faster and faster until it’s going as fast as the inner wall when you get there. This phenomenon, and the stress it creates in the fluid, is called “shear”.
Shear is intimately tied to viscosity. In fact, viscosity is a fluid’s inherent resistance to shear. If the fluid in the gap is air, the resistance to turning the inner drum is almost nothing. If water, it’s somewhat higher. And if it’s glycerin, it will be quite noticeable, because glycerin is a very viscous liquid.
But that high viscosity makes glycerin just the stuff to use for this experiment. Because of the glycerin’s high viscosity, the dimensionless quantity called “Reynolds number” that would apply to this experiment would be very, very low. All other things being equal, the Reynolds number for the same experiment using water would be 1200 times higher. Why is this important? Because the Reynolds number determines how the fluid behaves as is moves past itself. If it’s really low, there will be laminar flow, where the the different “layers” of fluid move over each other smoothly, resulting in the velocity distribution I described above. If the Reynolds number gets fairly high, the fluid motion will be turbulent, and things will get mixed up. Irreversibly mixed up…you can’t undo turbulence.
So turning the cylinder slowly with glycerin in the gap makes for a very low Reynolds number, maybe something like 50. This is called creeping flow, and is definitely laminar. As the cylinder turns, the ink line gets smeared out in this shear region, and it gets smeared out uniformly and predicably. It actually stays a continuous sheet of ink wrapped around the inner cylinder, but eventually becomes so thin as to be invisible. Because it stretches slowly and smoothly, reversing the cylinder can reverse the shear, and it all goes back into place.
Now if everything else stays the same, except the glycerin is replaced with water, the Reynolds number in the experiment may be something like 60000. That may likely induce turbulent flow as the water tries to slide past itself, but trips in the process. The ink line will get swirled and mixed into oblivion, and no amount of reversing can undo it.
Another key factor that makes this demonstration work is that glycerin has a very low mass diffusivity coefficient compared to something like water. Put a line of ink in motionless glycerin, and it will stay in tact for a long time. Put a line of ink in very still water, and it will diffuse to become evenly distributed in no time. With glycerin, you have more than a few millieconds to run the experiment, and you can go nice and slow to avoid turbulent mixing.