Final frontiers of fields/endeavors run non-scientifically

So I was reading this thread about how the same item priced at 4.95 will outsell the one priced 5.00, and had the thought that there’s a LOT more business-related research out there than a lot of people think, and that I wouldn’t be surprised if someone has quantified just how much each cent below $5 increases or decreases sales.

That set me to wondering what fields/endeavors are still NOT based on science/math, and are run via tribal knowledge, lore and Kentucky windage. The ones that come to mind for me are multiple sports. Baseball is somewhat notorious for the rise of Sabermetrics, which is a sort of statistical system for valuing players relative to their salaries, but most sports seem to be run based on a sort of body of tribal knowledge that doesn’t seem to be based on anything other than (possibly flawed) empirical observations. Football seems to be overwhelmingly based on this kind of thing, both in the player selection aspects (college recruiting/pro draft), and in the aspects of play calling while actually in the game.

What do you think? Are there any other fields out there that are not run based on scientific application or analysis? Do you disagree about football? Something else?

I probably wouldn’t have thought of this if you hadn’t mentioned sports: any of the Arts

There are probably scientific studies out there about the relative value of a Romance novel or movie vs a Western or a Biography, but there are so many variables involved in what people like and don’t like, that the statistics may be futile. So much of what goes into it is just gut preference.

I would be happy to learn that I’m wrong. It would be a fascinating read.

Do you mean in terms of book/movie/play sales? That’s interesting; I wouldn’t be surprised if studies on how to price the books have been done, but not about how plot affects sales.

Exactly.

Similarly for other arts. Pandora claims to play music similar to that which you’ve shown a preference for, but in my experience, that is based purely on the music style, and it totally ignores the content of the lyrics, which is very important to me.

Trial skills. Although people have been using focus groups more to test ideas, it’s still based mostly on lore and instinct.

I agree that publishing, movie studios, music producers, and the like do the worst job of surveying their customers of any consumer businesses.

AIUI, it’s not so much that they don’t understand, as it is that workplace politics get in the way. Most Hollywood directors probably understand that audiences are sick of the 6th, 7th, 8th sequels, yet the company management finance people would oppose anything that breaks new ground because it’s a “risk” - they would rather try to wring blood out of a dry stone by making endless sequels than “take a risk” by trying something new.

I think economic punditry - the kind of stock market / financial news you get as 5 minute fillers on TV - has less to do with any of the analytical techniques of the academic field, and much more with ‘here’s a graph - its going down - my prediction is … it will go down tomorrow as well’ and [an actual example from this week] ‘its been stable all month so lets assume it will go up before Christmas or possibly fall if it changes at all’.

I think lyrics do play a small role, since I once managed to get a channel that was somewhat consistently humorous novelty songs by starting with one of them and downvoting anything that wasn’t, but certainly they aren’t as important as the music behind the lyrics. Only when the lyrics prominently feature in the categorization of the song as a whole do they probably get much weight in their algorithm.

Mainly this is because a lot of the factors that go into how the stock market will do on a given day are what newly released economic indicators are and what the newly released earnings reports of companies look like. This information is unknowable by the pundits until it’s released, and once it’s released the computer algorithms will have already for the most part decided what will happen based on what the news is. All the pundits can do is point to the news and point to the graphs and claim “XYZ is up on news that ABC has happened” even if ABC has nothing to do with XYZ. There’s a lot of “post hoc” fallacy reporting in financial news it seems to me.

Usually the market does not move as much during the day as it does at the very beginning of the day, where all the news and information since the last close is condensed to reflect prices on the first trade of the day compared to the last trade of the previous day. Derivatives markets are active all night and are generally highly predictive of what the markets will do when they open. Movement of stocks not derived from news data is just a random walk based on extra-market factors (who needs cash, who has extra cash, and what are the idiosyncratic views of such people?)

I haven’t read much about the academic pursuits of those crazy enough to get PhDs in accounting and study what sort of new things we might do to better represent the ongoing profitability of the companies we report on, but what I have seen looks entire stupid. When new accounting standards are announced, I feel it’s mostly because the ivory tower types have decided that technology has advanced to the point where it’s no longer necessary to accept the kinds of estimations we have in the past because it was too difficult to get exact figures because of various corner cases. Now they have finally enumerated what they think all the corner cases are and have issued broad guidance in how to treat various transactions that invariably results in very little actual change except in those corner cases.

Now we can finally know how exactly to handle recognizing the revenue when we lease someone a copying machine subject to cancellation within 2 years but having a 5 year term with an upfront payment of X and ongoing service charges of Y that are prepaid every quarter with a Z per copy additional cost at a N per month minimum rate, etc, when before we just booked revenue when we billed them because that’s what we normally do. Now we have to analyze the contract and break it out into each component piece and determine when we have performed each of our obligations under the contract, etc. I’m mainly referring here to ASC 606 which changed a lot of things in theory but basically nothing in practice for the vast majority of people. It seems to be more scientific, but so much of accounting is a cost-benefit trade-off where you can get more accurate if you put in more time, but does it really matter? The PhDs seem to be all about telling us actual practitioners what we absolutely must spend time on regardless of any actual benefit.

Wine. The ranking and pricing of wine isn’t based on anything but a few influencers who are more often than not shown to be frauds if they ever consent to a blind taste test.

On the subject of movie studios, the MBA-trained executives think they are managing their creative slates according to objective criteria. They do, in fact, have spreadsheets where they have attempted to quantify various entertainment factors related to past projects, in order to make future movies more effective (i.e. profitable).

This is why so many blockbuster movies in the past few years have included weird little nods to China, either including a Chinese star or setting a scene in China or including a subplot where Chinese assistance contributes to the success of the overall story. It’s because box office in China has become a significant contributor to the bottom line, so the executives think they can boost this performance by directly pandering to those viewers.

But it goes well beyond that, all the way down to the level of plot elements and how they’re received by the audience, based on test-marketing movies in progress. People have comment cards where they identify what they liked or disliked about a movie, and sometimes they have a response dial or some other electronic device where they swing back and forth between positive and negative reactions as the movie is actually playing.

For example, this is how you get the warped version of Peter Parker in the Andrew Garfield movies written by Orci and Kurtzman and directed by Webb. The whole point of the original character is that his life is a dichotomy. Although it’s fun being Spider-Man, living as Peter Parker totally sucks, but he lives with his choice because he learned from the death of Uncle Ben that he has to be responsible with his powers. This suffering makes the audience feel bad, though, so in the reinvented version, Parker is a much cooler nerd who achieves popularity and recognition at his high school. (Executives actually discussed making Parker a Tough Mudder star.) You can quibble about whether adherence to comics purism is “correct” or “incorrect,” but it’s definitely different, and this is where it comes from.

Of course, all of these supposedly-objective factors are really just well-organized bullet lists of the same old gut-opinion guidelines. By covering the traditional process with a quantitative veneer, executives can insulate themselves from judgment; if the movie fails, they can point to a process where they “did everything right,” in order to direct blame elsewhere. But of course, none of this does anything to make the movie better, and frequently interferes with the process and makes the movie worse.

The live-action remake of Mulan is an excellent example of all of this in action. It’s a handsomely mounted film, well-cast and frequently beautiful to look at. (Niki Caro = good director.) But as a movie, it’s airless, joyless, and lifeless. When you take it apart, you realize immediately that it is not at all an organic story, and rather a mechanical execution of the Disney Executive Slide Deck for Combining Girl Power, IP Exploitation, and Global Box Office Leverage.

  • Based on beloved original — make nodding references in remake (e.g. frequent quoting of “Reflection” in the score) — check
  • Girl power movie — need female director — check
  • Strong-willed girl defying male authority — need to include male love interest so nobody gets the, uh, “wrong idea” — check
  • Fighting girl able to beat male opponents — need to include something in the plot to make this “justifiable and realistic” — result, bunch of nonsense about “chi” — check
  • Wuxia-influenced movie set in China — cast established Chinese action stars and include scenes where they get to show their stuff, even if it’s extraneous to the plot (e.g. Emperor Jet Li briefly fighting the barbarians) — check

And on, and on, and on. The point is, Mulan is a perfect example of a modern corporate movie project assembled and produced according to the spreadsheet mentality, where the limitations of the approach are clearly apparent even to the casual viewer.

“The single most important fact, perhaps, of the entire movie industry: NOBODY KNOWS ANYTHING.”
     – William Goldman

Goldman wrote the screenplays for Butch Cassidy and the Sundance Kid (1969), The Stepford Wives (1975), All the President’s Men (1976), Marathon Man (1976), The Princess Bride (1987) and other major movies.

I think you have a basket of mixed fruit here. Sports is its own animal and not subject subject to many of the principles that govern regular retail

I think retail is based on supply and demand. If my widget is badly needed, the cost is going to reflect that. If my widget is markedly more popular than yours, the price is going to reflect that. If the economy crashes like in 2008, the price of my widget is going to reflect that as well. A few decades ago, “pet rocks” were the fad. Because of that popularity, you could actually sell a rock in a box.

Quality governs sports, so analytics is the thing. Players are tracked from childhood. They are coached and conditioned mercilessly. They have to fit the parameters that computers say yield optimum performance at the positions they play. Game play is analyzed to death, and adjustments made in player movement and what they key on. Science is the thing there.

Until recently, and partially still today: Professional mind sports such as chess and go. There is, of course, literature on strategies and tactic, but they are based on the experience of (and mostly written by) high-level players, not on scientific analysis. At least that’s how it used to be until recently; nowadays artificial intelligence has entered these fields big time and is upsetting centuries of human experience.

The same may go for poker. Obviously, computers are a big help here for calculating probabilities and pot odds, but bidding strategies are still largely human judgment and instinct. To my knowledge there is still no AI that can, in the longer run, beat world-class human players, but such AI is certainly not far away.

AFAIK, most of that sort of prognostication is more along the lines of infotainment, not actual actionable information that anyone sensible would base any sort of investment decisions on.

I’m sure there’s LOTS of analysis of the stock market and trends, and how to predict things based on it, etc… That’s in many ways, probably one of the Holy Grails of scientific analysis in the business world- someone who could figure that out stands to become VERY, VERY rich. It’s just that it’s a problem of astounding complexity that involves a huge dose of group-think. It’s not a statistical problem, it’s really a statistical problem compounded by psychology, as the stock market is as much or more driven by people’s perceptions and feelings as it is on actual hard financial and statistical data.

That’s kind of the opposite of what I’m getting at; that’s essentially a movie-by-analysis, when truthfully, movies are essentially a sort of artwork. Even if they had good analysis on plot elements, etc… it still wouldn’t make a good movie just because they included all the moneymakers.

But that’s my point- I’ve seen ZERO evidence that any player characteristics are analyzed by computer, or that there are such things as “optimum” parameters by position. Or that gameplay is scientifically analyzed. Except for things like Sabermetrics and its nascent cousins in other sports.

From what I can tell, it’s all lore, tribal knowledge and guesswork. I mean, some things are known, like you’re not going to have a very successful 5’9" 175 lb NFL tackle. But so much more is just unknown- nobody can really account for Tony Romo coming from the Cowboys’ third-string after not being drafted into the NFL. Similarly, everyone thought Tim Tebow would probably be mediocre at worst, and not that he’d stink like a slaughterhouse dumpster in August. Or any number of high school recruiting busts in college football. And a similar number of walk-on successes and long-shot scholarship players who end up being spectacular.

I can’t help but think that we’d have much more consistent teams and trades and what-not would be a lot less risky if this was actually scientific.

I mean, things like nutrition, workouts, etc… are things with scientific underpinnings. But the actual coaching and recruiting seems to be governed by personal knowledge, intuition and tribal knowledge.

Is there any evidence of that beyond subjective opinion? (That’s a rhetorical question, please don’t answer it.) You make a good point about trying to use evidence-based criteria with subjective matters. But that applies to any criticism as well. We can’t really know it was corporate checklists that caused poor profits. That is, there’s no evidence that not using checklists causes works to be more profitable.

Speaking of workplace politics, I suspect that the vast majority of hiring and promotion decisions, in all areas, are made non-scientifically.

The first thing I thought of when I saw the thread title, before actually reading the OP, was mathematics. You could say that the field of mathematics is run non-scientifically, in the sense that the scientific method is not and never will be how mathematical claims are verified within the discipline.