It would be a conflict only at the most insignificant level. Unless it was a million dollar buy-in fantasy league, I cannot see how a federal judge would put his livelihood at risk over a replacement level fantasy QB. Now, if the judge revealed that he had put down a substantial bet on the Pittsburgh v. New England game next week? Yeah - that’d be a conflict of interest.
It’s trivial, but on the other hand it would be very embarrassing for a judge to get written up in the press as betting on football in some form while he or she had an NFL case on the docket.