I remembered where I read about how you could be allergic to specific proteins from the white or the yolk. A link in this post:
Link:
Both egg yolks and egg whites contain proteins that can cause allergies, but allergy to egg whites is most common.
j