I used to be involved in background investigations for security clearances. Just to be clear: I was not an investigator, nor was I involved in deciding to issue or deny clearances. I handled the initial applications, reviewed the final determinations, and had contact with the investigators. This was also about 15 years ago. With all of that being said:
There were a lot of kinds of derogatory information where there were no bright lines. If you had a shoplifting conviction 10 years ago and no other criminal record, you’d almost certainly be cleared. If you had two convictions in the last five years and were currently awaiting trial for a third arrest, you’d be denied. It’s always a judgement call. It’s not always “fair”, but that’s how judgement calls work.
From what I understood of the process, there are a number of factors taken into account. Marijuana use was always treated more leniently than other Schedule I drugs (and this was 15 years ago, before decriminalization and legalization were widespread). Occasional or casual use were treated differently than habitual use (and yes, that distinction was also a bit squishy, but there were questions about frequency - once a week, two or three times a week, daily, etc.). Recency was a big thing - as I recall, one of the big bright lines was use within the last six months (I may well be misremembering, but I think there was actually an explicit and formal question on the SF86 about drug use in the last six months).
Now, that was all for security clearance applications in the U.S. Army, which is not the same as the background investigation for a White House position. But it’s not entirely different.
And this is all well-established procedure with decades of precedent. It’s not like the Biden White House decided on this policy out of the blue. Drug use, including prior drug use, has been grounds for denying and terminating White House employment for decades. It’s been a standard element of background investigations and background questionnaires throughout the Executive Branch for decades.
What’s new is that most states have decriminalized or legalized marijuana use or authorized it for “medicinal” use (wink, wink, nudge, nudge), and there’s been a sea change in public perception of its use. A lot of people (including many in this thread) don’t think twice about recreational marijuana use, so there are a now a lot of ambitious young would-be staffers who have openly used marijuana, and are running into trouble when they apply for sensitive positions in the Federal government. Because it’s still a Federal crime, and the policy on that hasn’t changed in decades, and can’t change until the Federal law is changed.