My neighbors just moved in - Tyler and Erin, love 'em.
Tyler’s favorite hobby, one he indulges quite a bit, happens to be homebrewing his own beer. And he just brought over a sample pack for me to try.
I think I love him. This IPA is fit for a king.
Cool. I bet he appreciates that you like good beer. Odds are, most neighbors would be adjunct lager fans.
Score!
Save the bottles. I’m pretty sure that most home brewers reuse the bottles, and the bottle carriers, too. I think that the caps are disposable, but I’m not sure.
Man, I wish you the best.
Unfortunately, after a few experiences of soured relations with neighbors, I no longer want mine to be friends. I’d be perfectly happy if they were relatively quiet and maintained their property.
But like I said - good luck. Few things could beat a nice homebrew passed over the fence on a summer afternoon! 
Caps can be tossed. It’s extra-nice to rinse the yeast out of the bottom of the bottle with some plain water.
If you really want to be a good neighbor, ask him if he’d like you to save any brown bottles that had non-twist-off caps. 
(Last I looked into it, you can’t - or can’t easily - cap a twist-off bottle as a homebrewer.)
Reminds me, I have a batch to start.