Looking for some suggestions on how to implement what should be a fairly simple task, but for which an obvious solution doesn’t come to mind.
What I need to do is put together a system which allocates applicants for a site to a group of reviewers, who assess their applications. Ideally, the allocation is random, but each reviewer gets the same number of applications to process - so, for example, if we have four reviewers and twenty applicants, each reviewer will have five applicants to process. We don’t know in advance exactly how many applications we’re going to get, and each application should be allocated as soon as it comes in.
Now, if I just allocate the applicants completely at random, we’ll probably end up with a situation where one reviewer gets ten applicants, one gets six, and the other two only get two each - not really acceptable. If, on the other hand, I only assign new applicants to reviewers who have shorter lists than the others, the fourth, eighth, and twelfth applicants will only have one possible reviewer they can be allocated to, which destroys the random element of the process.
For example, in the first batch of, say, seven, you assign one to reviewer A, then B then C then D, then A then B then C. D only has one review. However, for the next batch of , say, 15 you assign again starting at D. This means that for the next batch of four you have to start at C.
At the end, of the 26 applications, A and B have reviewed 7 each, while C and D have reviewed 6 each.
The application is assigned to the reviewer with the lowest rank_number. factor would be chosen for the maximum permissible difference between reviewers’ workload which would be (off the top of my head, not checked) (maximum_random_number - minimum_random_number) * factor + 1. e.g. with a six-sided die and factor = 0.2 the maximum difference would be 2.
Put the reviewers in a queue. Assign the first applicant to the head of the queue and remove the head. Do this until you have no reviewers left in your queue. Create a new queue in a random order and carry on. Not perfect, but simple.
jjim and Dominic’s suggestion has the disadvantage that there will be one application per “cycle” that always goes to a definite reviewer - this is a less acceptable problem than an uneven distribution would be, I’m afraid. However, tschild’s idea sounds good and easy to implement - I’ll see what it looks like in practice.
In your system, the fourth applicant knows they’re definitely going to get Reviewer D. The eighth applicant, similarly, knows which reviewer they’re going to get as soon as the seventh application is assigned.
Yes, but, unfortunately, they’re not.
nods * - as long as it produces distributions that are less dramatically skewed than simple random allocation, it’ll be OK. It also only takes a couple of lines to implement, which is another advantage.
Again, if you do it in batches, simply figure out how many reviews every reviewer will get. Then, for each applicant, assign them the required number of reviewers, randomizing amongst those who don’t have the average number yet. They could have 1, 2, or 20, but if they don’t have the maximum yet, they are still considered.
Once they have the maximum, however, drop them from the randomization process. At the end, every reviewer will have the same number of reviews (well, within 1 anyway, most of the time.)
Applicants must purchase their right to be reviewed, payable to a disinterested third party (e.g. you). A reviewer with no applicants in queue costs $1. A reviewer with one applicant in queue costs $2. A reviewer with two applicants in queue costs $4, etc. It’s the only truly fair way
This implies that applicants have that information: the number of applications submitted by others, the list of reviewers, and the order of assignment. Is that the case? Why would applicants have that information?
Ludovic - Your suggestion would be OK, but it means additional work for the site management - someone would have to decide the “batch size” and keep it updated as the number of applications and reviewers changes. A fully automatic system is preferable.
jawdirk - Would work wonderfully - in fact, we could just admit anyone who came up with the appropriate fee. However, in the competitive world we live in, the number of people who are prepared to pay for a service they can get for free is rather too limited.
c_goat - I don’t decide site policy, I just write the code. That being said, having the list of outstanding reviews publically available does give an incentive to the slower reviewers to catch up. And even if we hid the information, it would still mean that the process wasn’t random; it’s better to solve a problem than cover it up.
