From another thread, reproduced here to avoid a hijack:
A bit of a hijack but I took the PMP exam. It requires a certain number of hours of project management experience. Anybody can study a book and take a test. Just because you can barf out the book doesn’t mean you know how to do anything. If you have actual experience the chances are better that you can demonstrably use the book knowledge in real life.
OTOH I am Java certified and have never written a production line of Java code in my life. That’s because I mastered the material out of books, but no assurance that I could function as a Java programmer on a job.
If I can take a stab at that, only someone with real world experience knows when and how to apply best practices. This works better for certs like PMP and the business analysis certs than the technical ones. Basically if you tried to use every best practice on the PMP to a small (for example) 40 hour project, you’ll turn it into a 5000 hour project. You need to know what you can safely skip.
So then why not have case study questions like this on the exam? E.g. “You are in charge of a 40 hour project with the following characteristics: <list>. Would creating a formal Change Management Plan be likely to help or hinder this project?”
Obviously, there would be uncommon or rare examples of 40 hour projects where you would really want to implement lots of formal best practices that would otherwise be reserved for larger projects, but the point you seem to be making is that there is a general trend. Why not test for knowledge of that trend, then?
Think of it this way. The certification exam tests your knowledge of the toolset needed to manage projects, but not your ability to manage projects. It’s like giving a carpenter a test to see if he can hammer a nail without bending it, saw straight, and drive a screw without stripping the head. A good carpenter must do these things well but that doesn’t mean he can build a structure. Most of the problems that you encounter on real-life projects are unique and require unstructured thinking to solve effectively.
Here’s an example from PMP. The PMP training tells you how EVMS (earned value) works. It tells you what data is collected, what metrics are calculated, how to interpret it. But it can never tell you what to do when you realize your project is behind schedule. There are the generic answers–add staff, reduce scope–but the real answer for any given project requires more understanding of your project than what you can know from reading a two-sentence exam question. You have to understand your customer’s priorities, limitations of the technology you are using, availability of additional staff with the needed skill sets. And you might also need to know that your customer’s contracting officer likes single-malt scotch.
These are things you can only learn from experience on real projects. IMHO you can test for mastery of the material, but mastery of material is not what makes good project managers.
For my scenario that apparently triggered this sub-thread, I suppose it’s possible that any fool could pick up the books or do some online “bootcamp” thing and cram in enough raw data to have a stab at passing the CISSP exam. (six hours and 250 questions) I’ve helped proctor a couple of these exams, and generally, the older people who presumably have been in the security field for a while finish first. At the last exam I proctored, I could almost hear a bunch of young brains lock up when the lead called one hour remaining.
The five year experience requirement certainly helps weed out the fools with book smarts but uterly clueless in real life - I’ve had to work with college grads who did not know what odd or even numbers were. I still don’t know how someone reaches legal drinking age without understanding that a number ending in 3 is an odd number and one ending in 4 is an even number. I really wouldn’t want them designing security policies.
A CSM (Certified Scrum Master) requires taking a two or three day class and passing a fairly easy test. Its an intentionally worthless certification - they wanted to get a lot of them out there.
CPAs vary by state, but generally require a lot of difficult tests PLUS work experience PLUS a minimum number of completed credits in the field.
I’d been working for a security software company for about 6 years when I took the CISSP exam. I whizzed through the cryptography and PKI sections but couldn’t have passed the rest without cramming from the book & practice exams. I’m going to be letting the CISSP lapse this year, I’m not getting anything out of it and keeping up with the CPEs is boring.
A general framework for worker utility is KSA: “knowledge, skills, and abilities”.
If management can catalog the specifics it needs for a successful worker in a particular job along those three categories, then HR can screen for those things. And probably get a good hire. Or at least avoid a hopeless hire.
The problem is that typical testing is much better at K than S or A. Especially the standardized multiple choice style of testing.
A certifying organization can either invest a lot of time and money in more elaborate case-based testing that can more accurately measure S & A. At the risk of adding a large cost to their grading and inserting a much larger fraction of subjective grading. Which in turn opens them to charges of favoritism, discrimination, slack of standardization or repeatability, etc.
Or they can simply tack “5 years on-the-job experience” onto their requirements. Which in effect subcontracts the S & A testing to your former employers; if they kept you around that long you couldn’t have been too clueless. Or so we hope.
If you were a certifying entity, which approach would you take? I thought so.
In general you’d expect to see more K-only testing regimes in entry level jobs or certifications and more holistic KSA + relevant experience testing in more senior level jobs or certifications. And IME that’s what we do see.