is it theoretically impossible to engineer a 100% reliable solution out of unreliable parts?

[Voyager]

@Mangetout

Failure appears to be an emergent property of broad/intensive use with the system I have in mind. All of the individual parts actually test out OK; any end-to-end test of a combination of parts, when performed alone in a test environment, will work. When 150 users are all trying to do different things, it fails all over the place at random. Retry the exact same thing that failed once, and it works next time, probably.

It’s not a capacity issue in the machinery - all the dials are in green - it’s most likely a collection of memory leak/overflow type issues or badly-scoped/poorly-controlled limits on something like file or record handles somewhere deep inside the database engine.

ICs get tested very thoroughly - more so than software - and still fail when put into a system.
Your problem is a test coverage problem. A component testing out okay either means it is okay or, more likely, the test isn’t good enough to find all failures. In hardware we have fault coverage metrics, which measure what percentage of certain defects a test can detect. When they first got introduced it showed that tests for ICs that the designers thought were good in fact stank. There are some test metrics for software, but not nearly as useful.

For testing designs, which is more like software testing than testing for manufacturing defects, we found that random tests are very effective. Hand written tests tend to mirror the assumptions of the designer (oh this will never happen - until it does) and have big holes. I know there are tools out there, but I’m not an expert. It is a big area…

[Voyager]

@Dr.Strangelove

It happens.

The lifetime of an IC is very sensitive to voltage and temperature. A 10% difference in voltage may mean the difference between a chip that lasts 20 years vs. one that lasts 5 years. That may seem a reasonable price to pay, but 10% can mean the difference between a viable product and a complete flop. So in practice, the chip is run right at the ragged edge of its expected useful lifetime.

Our parts go into big systems, and the voltage for each part is set individually, based on the performance of that part. If something is out of spec I don’t count it as a reliability fail - like if a chip fails because some guy is overclocking it.
We’re pretty conservative, and our FIT rate is always better than target unless there is some specific issue. A few years into one product our reliability guy sent me an email saying that according to his equations we should be seeing field returns. I looked at the return and repair data and told him, nope.
But it depends on your industry. In some too much reliability means you are leaving performance on the table.
At system level fans and power supplies are much bigger problems than ICs.

[Voyager]

@Francis_Vaughan

Non-deterministic is the key here.
Even perfectly reliable hardware is subject to non-deterministic behaviour without extreme effort. In reality is is extraordinarily hard to work around this. (My PhD was concerned with an allied aspect of this, and it is grim.)

This is very true. As feature sizes get smaller, SEUs (Single event upsets - like from cosmic rays) get more important. Memories take this into account already. Though there have been some solutions proposed for logic, I’m not aware of any generally implemented. Of course things that are going into hostile environments pay special attention to this, hardening features and doing the well known checking and redundancy stuff.
Luckily SEUs in random logic are not likely to cause a problem, because they will only effect the system if the spike happens when the signal is clocked onto a storage element. SEUs in flip-flops are going to be worse.
We used to keep data on the elevation of where our systems were installed. You never know.
I’ve actually expected to see problems before now, but it looks like I get to retire before I worry about it. But I hired someone who did this for his PhD work, so if it does happen we have a resource.

[Mangetout]

@Learjeff

First, give up on 100% reliability, because it doesn’t exist. Next, figure out the reliability rate you do need. Third, measure the reliability of the components you have.

Those are the easy first steps. If you don’t know how to do them, then you need to hire someone who does.

It can be simpler than that; The measurement of reliability doesn’t need to be terribly precise on the first pass; the next thing to measure is the capacity for change - can the solution actually be made better at all?