Most bones, in most people, only fracture after experiencing a substantial amount of force. Such force is often sufficent to cause local damage to nerves or small blood vessels; damage which may be transient, permanent or very slow to heal. Bones also do not always heal in their “original” position (a broken collarbone can functionally heal but leave a very substantial bump), and this may place further stress on local structures. Bones which are placed under uncontrolled stresses (patient decides to take off the cast; bone bears a lot of weight and patient did not use or was not advised to use crutches, etc.) may also heal badly. Sometimes healing is poor even if everything is done correctly (poor nutrition, just because, etc.)
The scaphoid, as mentioned, is a special case. This bone is highly susceptible top damage and poor healing due to a tenuous blod supply. Even minor injuries to this bone (often missed) still require a cast for six weeks. Fractures may not show up on an initial X-ray; patients may require a great deal of persuasion to keep a cast on for six weeks when this is the case.