The milkman has yet to die. Its a hollow threat really.
Remember: net Once, stamp Twice. Three or Four times if its COMCAST. 
I can’t believe Wikipedia would say the damage is “minor or nonexistent”. Carpenter bee nests can extend for several yards from the entry hole and significantly weaken dimensional lumber.
To the OP, be sure to fill any holes they create with caulk or wood putty. The bees love to reuse existing holes and will burrow deeper and deeper.
They are more of a problem in residential structures than they used to be because of a change in the way that pressure-treated wood is produced. The old way used Arsenic and was better at repelling insects.
A trick for active nests is to spray WD-40 into the hole with the flexible straw. I was skeptical after reading it online but it works better than wasp spray.
This shit really works. I don’t have carpenter bees but I have used it for wasp nests and it is amazing.