The whole “brother” label has always been a toss for me. I have encountered a large number of extremely Christian southerners who simply call everyone brother/sister. This practice is not necessarily easy to detect, but a little time and attention reveals it quickly. Additionally, most of my close white friends have hippie parents who call everyone brother/sister as well. So if you referred to a group of black people as “brothers” and you either had a southern drawl or were wearing a dashiki, I would not likely notice the term or even take offense to it. Still, I would not advocate continuing this practice as many people might misinterpret your racial views.
I have also had people refer to me as “my brother” in a negative way to draw attention to something in my behavior that they found to be undesirable. For example “Was that you I saw conversatin wit all dem white womens over there, my brother?” Ignorant and close minded too, what a treat.