It’s a coax adapter. It just looks like a black plastic dohicky with a female coax port and underneath it are two screws. Attach your TV/Game box to the screws, plug the coax end into your male coax (where you plug in the cable) and you’re golden.
Oh, and for the record, the part you want should run you about $1.99 or so, so if you see something for $12.59 it’s not what you want. Seriously, you can go to about any place that sells even household electronic type things (large drugstore, hardware store, KMart, Walmart, etc) and find one. I probably have a half dozen of them lying in a box from various sources. Just look by the televisions or over where they have phone cords, 3-prong outlet adaptors and that sort of thing.
You should be able to use the 2600 and have your TV set for cable. The signal would show up on your TV’s “video” or “aux. input” setting instead of chs. 3 or 4. This is how my SNES works.