Are there TV RF modulators which output digital/HD on channel 3?

FWIW - here’s a list of common display resolutions:

You’ll see that NTSC is ~ 440x486 and 1080p/I is 1920x1080

So you get almost 10x as much pixels at 1080p/i - there is simply no way to force that into an NTSC device (using and NTSC signal) and get more than that.