If the problem is the OP’s unruly passenger, possibly the only people who’d need to move would be some cops to come aboard and gather up one passenger. Could the plane then be refueled where it parked and then sent on its way?
For a medical event it’s plausible to bring on the paramedics, remove just whoever needs to be removed, then the airplane can proceed. Subject to a bunch of details below.
For a law enforcement event there may be a more extensive need for the police to investigate, interview the crew or passenger witnesses, etc. Note that this is the last time everybody will be neatly corralled in one place and readily identifiable. If nothing else, they’re going to want to collect name and contact info from everybody involved. Which might take quite awhile to gather after the problem child(ren) is/are safely in a car going downtown. So a hefty delay is on the cards.
For the very simple (and by far the most common) case of bad weather at the destination forcing a landing elsewhere to buy more fuel and wait out the storm or whatever then proceed, none of the above passenger handling issues apply. But all the below still does.
Once all the above is done we have several hurdles to departure.
One is fuel as you say. That can certainly be trucked out to the airplane, although at most terminals its pumped up from a hydrant fixture embedded in the ground. So much more conveniently & quickly done at a normal parking space, particularly for a really big airplane.
The next issue is “paperwork”. Back in the day it really was made of paper, had to by signed by the captain, copies left behind, etc. So it’d have to be printed, couriered out to the airplane, reviewed, file copies signed and couriered back to the terminal, etc. Nowadays we can get it all on our tablets via cellular data service or the aircraft’s WiFi-to-satellite service. If they’re working wherever we happen to be. Nowadays it’s rare to have both unavailable at the same time, but it occasionally happens. Cue a clusterf*** even if it’s an otherwise normal departure from the terminal.
The next issue is maintenance. If anything broke during the flight it’ll have to be fixed or officially recorded as broken and acceptable for a deferred repair on an FAA-approved schedule. Which would require suitably licensed / trained maintenance people to come to the airplane, inspect it, secure it in a safe inoperative state, play with both physical and virtual paperwork, etc. Or actually fix it if deferred repairs aren’t authorized.
For some deferable malfunctions there is a requirement for a documented inspection before each departure. We may have left the previous station with such a thing already on the books and a documented inspection for that flight. But having landed at the divert location, we’re stuck until a suitably licensed / trained worker can do the inspection and the physical and virtual paperwork to document it.
The last issue is crew legality. We can only be on duty so many hours in a row and only actually fly so many hours during any one duty day. A typical limit might be 14 hours on duty and 9 hours of airplane-in-motion within that 14 hours. (Although the details look more like something the IRS would write, not the FAA. It gets wicked complicated quickly.) Our normal workday schedules are often pretty close to those limits before anything goes wrong. If any of those limits would be exceeded by any one of the 3 to 15-ish crewmembers before we got to the next stop (probably the stop of intended landing before the divert reared its head), well … we can’t leave here.
Bottom line:
The list of ways things can go wrong is long. The later in the workday, the more likely the divert will snowball into a customer service mess. Fuel is an obvious requirement, but is one of the easier ones to satisfy.