I have a set of measurements of several organization (around 50). The organization is similar to a company. The same measurements are taken of each organization. Unfortunately there is no way to control the size of the organization-some serve 30,000 clients some serve 3,000 clients. The measurements were taken to estimate the “quality” of the organization-whatever that means. I need to come up with a statistical means of ranking the organizations based on those measurements. I will leave it to others to argue whether that is a ranking of quality or not (I don’t think so, but I wasn’t asked).
Unfortunately retirement has gummed up my mental statistical gears-I believe there are tests for this purpose that will tell me whether there are significant differences between the organizations but I can’t find such a test.
Each organization was scored on a series of questions/tests. IE, they provide training in a particular subject-what percent of the students complete the training. How much do they spend/client, etc.
For ranking, I’ve found a weighted decision matrix to be helpful:
For example, from 1 to 10, how important is training? How important is per-client spend? Then you multiply those weightings by their scores for each company.
For example:
Training is 10/10 important
Cost is 8/10 important
Sustainability is 3/10 important
And then:
Company A:
___Training: 4x10=40
___Cost: 98=72
___Sustainability: 23=6
___Total: 118
Company B:
___Training: 9x10=90
___Cost: 48=32
___Sustainability: 23=6
___Total: 128
Company B wins because you decided that training is more important to you than cost (by 2/10 weighting points).
Impossible to say, without a degree of information that you’re probably not comfortable sharing with us. But if you have multiple measurements for each organization, and you’re combining all of those measurements into a single figure of merit of some sort, then you might be able to get something out of comparing the results of the different measurements. That is to say, if organization A scores better on some measurements than B, but worse on other measurements, then it’s difficult to say that there’s a significant difference between them, but if A outscores B on ten out of ten different measurements, then that probably is statistically significant.
Of course, statistically significant isn’t the same thing as real-world significant, either.
Thanks. the information isn’t private, in fact it is publicly available but rather mundane for purposes of discussion. The data are school performance scores, but I am interested in the general concept of statistical ranking.
Say I have 10 measures of each of 50 processes. I am only interested in ranking the processes. I can choose one measure and rank the processes using that. I can use two measures to rank. I can average, take the median, weight the measures, etc. I can rank the processes in hundreds of different ways. My question is: is there any way to determine which ranking is better? By better, I mean statistically significant-whatever that means.
I think I remember discussions of ranking of ordinal data (I don’t have ordinal data but that is part of my memory of this idea) and ways to distinguish among the ranks. At least enough to say whether in a ranking of A B C D E F there is a significant difference between two adjacent values.
I may be completely misunderstanding your post, but if you want to see whether one or more groups are ranked significantly different from how the other groups are ranked, a standard test for this is the Kruskal–Wallis one-way analysis of variance. Post hoc testing is needed to determine which groups are actually the different ones, Dunn’s test is one of those used for this purpose. Though it sounds like you might be trying to compare the efficacy of different ranking schemes based on weight, so then you’d need a more complex model.
thanks.
I am certainly trying to keep it simple.
I have a group of processes and a fixed set of measurements of each. All that is fixed.
I can rank the processes in any way I wish.
My main question is what test can I perform to be able to state that the differences between processes is significant. That is, when someone questions my ranking, I want to at least be able to say that the measures do in fact indicate significant differences.
In any stats thread you (OP) need to be real careful with the word “significant”. Ref Chronos’ excellent “Of course, statistically significant isn’t the same thing as real-world significant, either.”
Taking your example of spend/client. If entity A spends 2x what entity B does, does that prove A produces twice the result? Heck no. they might be 1/3rd as efficient as B and therefore have a worse outcome.
In the real world, what matters is outcomes or results. Not resources expended. If your data already has a single fairly objective measure of merit, say profit margin, then you can certainly run a multivariate analysis of your various inputs vs. that output to get in effect a multi-dimensional best-fit line that describes how the “average” entity would convert any given mix of inputs to the output. And from there develop a measure of fit: “A is doing it right, B is doing it a little wrong, and C is way off the ranch.”
IMO absent some idea of what an entity output looks like, all you’ve got is a bunch of pseudo random numbers in a howevermany-dimensional cloud. Speaking of which, you said you’ve got about 50 entities. You haven’t said how many parameters you’ve collected on each. Whether that’s 2, 5, 10, or 50 parameters will affect your complexity a bunch.
Thanks.
I understand the importance of being careful with what “significant” means. Statistically significant is not the same as functionally significant. But it is a useful measure to make.
There are usually 9-10 parameters for each process. Of course I can subset those parameters, but I would like to consider them all in my analysis.
