As others have said, libertarians aren’t in favor of no government, but in general are in favor of smaller and more localized government. One of the many benefits of having a smaller government is that it ought to be more efficient, there’s just less overhead with a smaller government with less moving parts. Thus, from a libertarian perspective, given the sheer size of California, in terms of the geography, population, and economy, it has an already difficult time, and having a larger government just makes it worse.
That said, I think most libertarians would agree that the courts are an essential function of government (some will disagree, but that’s pushing borderline anarchy). As such, given that the government wouldn’t be funding other things that aren’t an essential function of the government (perhaps public schools, various regulatory agencies, etc.), then there would be more funding for the essential functions.
There’s also other proposals that could go along with this from the small government side. For instance, there’s always the complaint about frivilous law suits, perhaps two birds could be killed with one stone by having more of the costs of court fall on those involved in the civil case. So, for instance, if I sue my neighbor for something legitimate and I win, I receive appropriate damages and he is also responsible for paying my costs in taking him to court (lawyer fees, filing fees, lost work, etc.) and court costs (building upkeep, pay for the judge, jury, and court staff). Similarly, if I file a frivilous suit and lose, the burden of those costs fall on me instead. Thus, the civil court remains fully funded and available for those that need it.
There would still be the issue that criminal court, in general, needs to be publically funded because it’s not just settling a dispute between two parties, but an issue between the party and the state. That would obviously have to be paid for by whatever means the government gets it’s revenue, likely taxes and fees.
The bigger issue would be what should and should not fall under which. In my opinion, it should be broken down strictly by who is and is not wronged, whether the state has a say in it. So, for instance, I think the state does have a say in a polluted water source since it has potentially wide-spread repercussions, but perhaps some things that are traditionally seen as crimes that the state prosecutes might be better served through the other method, like perhaps an assault charge or the like.
Anyway, that all said, I think the biggest issue here is just the misconception of what libertarianism represents. Even if it were conceded that reduced regulations would increase the costs in civil courts and the like, it does have reduced costs in actually applying the regulation, or at least in applying it on a smaller and more local scale. In fact, a big part of the idea here is that by taking some of the onus off of the government to perform regulations and put it on the parties themselves, they’ll be more inclined to make and follow reasonable restrictions because the cost is coming out of their pocket rather than being distributed to everyone via taxation.