I thought I had replied to this earlier, I guess the hamsters ate it.
I was considering doing what you describe but the pain in the ass of using several different methods of switching combined with the limited number of HD channels - none of which I use - made me decide to just run it all through the switch box.
The effects were a loss of sharpness. It was not huge but it was noticeable, and only because the HD signal is usually crystal clear. You only really see it where there is a very sharp edge on the screen. Didn’t notice and trouble in colors.