An ngrams search shows that illusory is used 20 times more than illusionary. Illusionary is much newer. Probably it has no more meaning than the pedantic effect of sticking in an extra syllable, a trend in current English.
The quotes from the books listed don’t show much difference in the use of the two words. I doubt that there is any useful distinction, unless some academic discipline has adopted illusionary as jargon.