If you see mold, it generally means that there is mold where you can’t see, thanks to the fungus’ hyphae. This depends on how “squishy” the food is and I believe lychee applies. Will it kill you or at least make you sick if you cut it off? Probably not. But be aware that you aren’t removing much. I don’t know if I’ve ever seen “natural” lychee (unpeeled?), so I can’t say if there is an acceptable amount.