IMHO, this answers your question. “Secular” means: of or pertaining to worldly things or to things that are not regarded as religious, spiritual, or sacred; not pertaining to or connected with religion. I donate to secular charities and take part in outreach programs that are done through non-religious community groups. To devote my time to an organized group, I would need to be motivated by something I care about, I don’t actually care if anyone else is agnostic or not, so as BoyoJim entioned upthread, a lack of belief really isn’t a good, sticky, rallying point. I don’t like tomatoes, but I’m not anti-tomato. I don’t particulalry care enough about it to join a group of people who don’t eat tomatoes. Religion has no relevance to me. Joining the No Relevance Bandwagon doesn’t appeal.
Atheist/agnostic/skeptic groups around here are mostly either political in nature, or they are discussion groups about various aspects of secular ethics, science, and/or philosophy. But looking up local groups, the Freethought Association of Canada has does community services, and the Centre for Inquiry does community voulnteering as well. There are also a ton of campus groups here that also do volunteering as a part of being good citizens, but mostly they are discussion groups.