So, if you aren’t traveling to Montreal to visit family, why are you visiting Montreal? Sure, there are lots of legitimate reasons to go to Montreal and back in a day. There are also lots of illegal ones. Their job is to stop the people committing illegal acts. If you are evasive in answering their questions, they are going to dig deeper.
As far as your concerned, it is not an issue, but to the boarder guards, it might be. It all depends on what your actual purposes are for going to Canada (or, more importantly, your purposes are for entering the United States).
Whether or not you did anything illegal while you were in Canada is of no concern to the US guards (provided, of course, you aren’t on the lam from the RCMP). The guards have to deal with 100’s of people every day. If you don’t give them a reason to spend more time with you than average, they aren’t going to want to. If you try to act as if they have no right to question you by virtue of the fact that you haven’t done anything wrong, well, they will probably think you are trying to hide something. Either that, or you’re just looking for trouble. If you’re looking for trouble, messing with federal agents is is good way to find it.