I think a big part of it comes down to your view of the proper role of government in society.
Liberalism tends to see government as a big ‘problem solver’, to be applied to any issue that the public sees fit to address. Moms can’t afford daycare? Government should fund it! People get paid too little? Government should set wages! People drive cars that are too large? There should be a tax! People can get hurt on the job? Government should protect them! Etc., ad infinitem.
Conservatism tends to see government as a necessary evil, and conservatives are generally suspicious of government ‘solutions’. Conservatives prefer vouchers for education. Conservatives prefer market-oriented solutions for environmental problems. Conservatives don’t think the government has any business funding day cares or paying for everyone’s health care.
Libertarians believe that the initiation of force is wrong. Taxation is at best a necessary evil when the money goes to paying for essential services Libertarians agree with. But they find the notion of holding a gun to someone’s head and forcing them to pay for someone else’s child care to be abhorrent.
So on a basic philosophical level, Libertarians are much more in tune with conservatives.
As for drug legalization and other civil liberties, that cuts across both parties. IMO, there is almost as much support for drug legalization on the right than there is on the left (with the exception of the current crew in the white house). The National Review, possibly the most well-known conservative magazine, has supported legalization of drugs for over ten years.
But in the end, Libertarians find the biggest threat to liberty to come from the left in the form of a nanny-state morass of regulations, and high taxation that forces them to work half their lives just to pay the bills of the state. Many libertarians will point to ‘tax freedom day’ and say that the days before that are basically involuntary servitude to the state. They are disgusted by many major policies pushed by the left, such as the Americans with Disabilities Act, OSHA, the heavy-handednes of the FDA and the EPA under Democratic governments. In their day-to-day lives, they are far more likely to run into laws and roadblocks erected by liberals. Want to open a business? Better get that doorway widened, and put in a handicapped bathroom. Want to hire someone? Make sure you follow affirmative action guidelines. Want to start a business making pharmaceuticals? Good luck, unless you have a billion dollars to comply with the incredible tangle of obstructions set up by the government.
Or here’s another way to look at it - I’m a Canadian, and generally Libertarian in outlook. If I was forced to move, and had to choose between the United States, where conservatives have held more power over the government, or Europe, where Liberals have had more power, I’d choose the U.S. in a heartbeat. My outlook on life is that I own my own life. I am not a slave to anyone else. Society has no right to take what I earn or restrict what I do for the benefit of others. I do not want or need to be protected from the harshness of the real world. I want the freedom to sell my goods to whoever is willing to buy them, to hire whoever is willing to accept the wages I offer, or to accept a job from someone else if I accept the terms of employment. I do not need government setting limits on my actions as long as I do not hurt others. I am willing to accept responsibility for my failures, and reap the benefits of my successes.
I do not believe the rich are evil. I find class warfare to be obnoxious bigotry. I do not believe people should be judged by the color of their skin, and therefore find affirmative action and racial pandering to be obnoxious. I believe that allowing people to fail is an important part of life, and too soft a safety blanket damages society and more importantly creates a large a dependent class.
Republicans fit my philosophical outlook much better than do Democrats. I dislike their pandering to religion and their social conservatism, but see those things as much, much smaller threats than the gradual death-of-a-thousand-cuts of the modern regulatory state.