Setting up rotating machinery. The manual gives a spec for vibration of “0.20 V in/sec.” at speeds less than 3600 RPM.
How do I convert this into an acceleration reading (G, or in/s^2, or something like that) that I might measure directly with an accelerometer? 
A piezoelectric accelerometer alone is only going to give you a varying voltage output; if your accelerometer is standalone, rather than a component of a vibration analysis test set, you’ll have to find data for your particular accelerometer (IME, a graph from the accelerometer’s manufacturer) that plots voltage output with vibration intensity.
After further reading/inquiry, I’m now pretty sure that the “V” in my spec is an indication that this is a velocity spec (of 0.2 inches per second). In documents discussing vibration of rotating machinery, there are equations that relate displacement, velocity, acceleration, and frequency (RPM) of the vibration.
Correct. Incidentally, 0.20 IPS is quite common in the helicopter world as the upper limit on rotor system and drivetrain vibration.
What are you planning to use to measure the vibration on this machinery? A bare accelerometer, with no means of determining the clock angle of the vibration, will only tell you that a vibration exists.
Does the machinery manual give a procedure (or hint at suggestions) for correcting an out-of-balance situation that causes the vibration to exceed 0.20 IPS?
The guys @ work who do our vibration analysis utilize a proprietary analyzer supported by another proprietary software package. A fairly large piece of equipment will have quite a few test points with permanent accelerometers installed. To achieve a complete analysis, all the test points are logged for magnitude and frequency (and harmonics), then run through the software.
Other than determining a voltage min/max output from your accelerometer, without a proper analyzer I think you may be up the creek…
Any idea of the span of your accelerometer? With that, you may at least be able to calculate the force involved.