Depends on how “active” your server is. If it sends out email a few times a day a simple activity count won’t distinguish you from a normal email using person. If the ISP’s software is analyzing the headers of the messages, I’m sure they are not perfect and spammers know how to avoid detection.
It’s like with http service. You can use outgoing port 80 all you want to browse the web. It’s incoming on port 80 that ISP’s like to block.
That’s up to the specific software the person’s using. I’m sure there’s a lot of software which won’t allow anyone to go behind the curtains like that, but other software will.