Indeed - I think the “black” component has well slipped from the standard usage.
As an amusing aside, and to show just how the word is now perceived, I asked my wife if she knew what swarthy meant. She said “It’s an adjective for a pirate, isn’t it?”