Last night, my opponent was shooting over some balls up table and he moves a ball with his hand (not the object ball). The cueball goes around the table and passes through where the moved ball would have been, changing the final resting place of the cueball. We did not know if it was a foul. Also we didn't know if we should move the balls back because we don't know where the cueball would have ended up. So we just left them alone, and i think they favored him. Whats the right thing to do?