I wanted to get back to this when I had a moment. It demonstrates a common mis-conception about the whole issue.
ALL FOULS ARE SUBJECTIVE.
If you look at Law XII of the IFAB’s Laws of the Game, specifically the list of fouls which result in direct free kicks (and, thus, penalty kicks if committed by the defense in their own penalty area), they are almost all subjective in nature. The only two which are not are the bite/spit foul and the throw something at the ball/someone foul, both of which were recently added just to make sure everyone knows you can’t do those stupid things.
The first seven listed fouls (charge, jump, kick, push, strike, tackle/challenge, trip) are only fouls when done “carelessly, recklessly, or using excessive force”. That make clear that a subjective determination is required. Handling is only a foul when done “deliberately”; that as you note requires a subjective determination. “Holds an opponent” and “impedes an opponent with contact” don’t sound subjective, but the subjective nature of those two fouls comes from deciding when someone is actually holding or impeding someone. Thus, all the common fouls are subjective in nature.
The trouble with the handling foul isn’t that it is subjective. After all, people debate all the time about penalty calls for tripping and holding, yet those fouls don’t get the ire of everyone. The trouble with the handling foul is that it gets called on the basis of an inference. Is the players arm where it is as the result of conscious or subconscious choice by the player to try to potentially influence the ball? Or is it there because, as people put it, the arm has to be somewhere, and the player is merely trying to do something else other than use the arm to influence the ball, and the arm just got in the way? But in the end run, it doesn’t really matter. Penalty calls for handling are usually no more or less disputable than penalty calls for tripping, charging, holding, pushing, etc. It’s just a slightly different debate.
In all cases, they are subjective. It’s not possible for that element to be taken out of the decision.