May I present the possibility that the overriding factor in this problem would be the law of inertia, not F=ma. With inertia, m1v1=m2v2. The implication of this law is that the mass of the ball-bat system and the mass weighted (no pun intended) velocity of said system would equal the mass of the ball times its final velocity.
Velocity of the ball off the bat is what determines distance, assuming that the ball’s path is not towards an object. Thus the heavier bat would seem to have the advantage, presuming that it can be swung at the same speed. Additionally, this simple equation doesn’t account for the fact that the batter (mechanical or otherwise) is applying a force to the bat, adding to the amount of kinetic energy that the ball receives.
That being said, if you had a machine that would supply a constant torque, the speed would be inversely proportional to the mass of the bat, and it would end up that (all other things being equal) both should hit the ball the same distance. This is what can be said about a human baseball player. They can only apply so much of a torque to a bat, and this is the limiting variable. This torque can only be applied for a short distance, causing a heavy bat not to get to as high a speed because of it’s initial inertia. To get the ideal bat weight, one would have to test the player with a bunch of bats. The best bat would be the one that the player could swing with the greatest inertia, that is, with the highest product of bat mass and bat speed. Now, if the machine could rotate the bat at a constant speed, regardless of weight, the heavier bat again would hit the ball further.
This reminds me of a quip I heard once. Michael Jordan tried out baseball and a scout remarked, “It’s called bat speed, and he hasn’t got it.” Good luck back at basketball, Mike.