It’s mostly a matter of material science and engineering not supporting the compression ratios needed for small diesel engines (or high compression gasoline engines, which did occassionally detonate in the catastrophic sense) until low-compression gasoline technology (and supply logistics) were already entrenched.
Since then it’s mostly a market problem. Nobody’d buy small diesel, so nobody researched small diesel, so small diesels sucked for a very very long time.
I have no cite, but I was once asked to try to obtain a model airplane diesel engine manufactured in Britain. This was back in the 1980s and I understood that any such devices would be old then. I wasn’t able to find any, but it gives you an idea how small they can be.
Model diesel engines still exist. Some are modern copies of old designs. They can be smaller than 1cc. However, they used to run on their own kind of compression-ignitable fuel - a typical mix was equal parts of ether, castor oil and kerosene. Ether was easy to ignite, kerosene gave added fuel value, castor stopped everything gumming up. Diesels of this size were (and are) two-strokes.
Road diesel may generally be less taxed than gas in Europe, but here in England the Chancellor plainly decided he couldn’t bear the thought of not raking in all that lovely tax; though it used to be cheaper, it isn’t now.
Kubota makes a reliable diesel,used as powerplant in other ways besides transport.13 horse smallest to date.
My diesel backhoe,kept outside,yields about a day’s work for a gallon of fuel,and has never failed to start,even at sub-freezing temps.Even at current prices for low sulphur that seems good economy.
Diesels are the most amenable (with least conversion) to alternate fuels,i.e.,used fry oil.
I remember talk of micro diesel engines which would replace batteries in things like mobile phones, iPods, etc. Fuel consumption was somewhere in the order of a thimbleful weekly or fortnightly. This was a couple of years ago, and I’ve heard nothing since.