In many cases, the brand-name drug is little or no different from the generic, and there is no good clinical reason to prescribe the more expensive drug over the cheaper one.
In some cases, there is a real difference, albeit small. For example, many epileptics will have an adverse reaction to a switch from one brand to another even if the active ingredient and dose are technically the same. If this produces only one tonic-clonic seizure, it’s enough for the person to lose his driving license, so many doctors will be very reluctant to switch a patient to a cheaper brand (or generic) version of the same drug. The same is true, to a lesser extent, with diabetics and insulin.
As for other cases, there are a few reasons; some good and some bad:
Time pressure/lazyness: the doctor might remember the brand name but not the generic name. For example, Betamethasone Valerate is marketed (in the UK) as “Betnovate”. It’s commonly use for eczema and similar skin diseases and most doctors (that I know) remember “Betnovate” much more easily than the generic name. Add an extra couple of quid to your prescription cost.
Marketing: In my household we have approximately 64 items which bear the brand name of one drug or another. I recently wrote a note to our neighbour on paper that said “Endocort Enema” and thought nothing of it. Funnily enough, none of these things carry the generic name of the medecine.
Beanz Meanz Heinz: As a general rule, Glaxo’s own brand drug is at least going to be of reliable quality and, from time to time, you’ll find that Bloggs brand comes up with a dodgy batch. Why buy Coke or Pepsi rather than supermarket own brand drink? It’s the same reasoning: you’re getting a quality product from an established manufacturer.
Compliance: Many patients, when asked what medicines they take, say somethig like “pink ones in the morning and yellow ones at night.” Given the potential confusion it could cause, you don’t want to try to switch them to green ones in the morning and blue ones at night unless there’s some pretty big clinical benefit. (Why were they started on the brand-name drugs? See time pressure/lazyness & marketing.)
Evidence: Like Dr_Paprika says, the major clinical trial might have been conducted with brand X but not brand Y. There may or may not be a significant (undiscovered) difference in effectiveness between the two. Drug advertisements often play on this: “the only drug proven to increase five-year survival post-MI” and so on. We’re back to the “Brand X” factor.