So there is this family of enzymes, mostly living in your liver, that are responsible for metabolizing most of the drugs prescribed. These are the CYP450 enzymes. This family has lots of enzymes which go by similar names (CYP2D6, CYP2C19, CYP3A4, etc.), each of which takes primary responsibility for metabolizing a drug or group of drugs.
As with most things, there are different polymorphisms of each of these enzymes. One polymorphism (one type) might be a really active, efficient enzyme, while another polymorphism might be really slow and not very effective at all. A third polymorphism might be somewhere in between. Everybody gets two different versions of each enzyme (one from mom, one from dad) and these combine to give each person a general level of enzyme activity. If you got two fast versions, you might be REALLY quick at metabolizing drugs worked on by that enzyme, but if you got two slow versions, the drug might stay around in your system forever while your sluggish enzymes tried to decide whether to metabolize them or not.
This is all well and good, but not the complete story. These CYP450 enzymes metabolize drugs that you ingest. “Metabolize” can mean activate or inactivate, depending on what kind of drug you are talking about (some drugs are packaged as pro-drugs, which need to be activated by enzymes before they work, while other drugs are active right out of the bottle, and are metabolized into inactive metabolites.)
The clinical significance of all this is, that people will have different responses to different drugs, depending on what kind of drug it is (pro-drug or active) and depending on the genetic make-up of the metabolizing enzymes.
For example, codeine is packaged as a pro-drug, that needs to be turned into morphine by CYP2D6. For people who are slow metabolizers with CYP2D6, they get ineffective pain relief with codeine, because they never actually convert it to the active drug and it just gets excreted unused. For people who are ultra-fast metabolizers, on the other hand, they can get codeine intoxication, because their liver is too enthusiastic about converting codeine into morphine.
All of this gets even more complicated when you start throwing in other things that can induce or inhibit these enzymes. SSRIs inhibit CYP2D6 (the enzyme that activates codeine). So if you are a slow metabolizer already, and happen to be taking an SSRI, you will be even more slow. Each enzyme has a list of things (other drugs, grapefruit juice, alcohol, herbal supplements) that inhibit OR induce it.
So, yeah. TLDR: genetics.