is there a large, sordid underbelly of codebases without / with insufficient unit tests irl?

and, of course, if the answer is yes, is there a market for hypothetical clever bandaid solutions to this problem courtesy of a certain Doper programmer bored with unit tests and his low on architecting cool stuff / high on keeping crud code bases running job in general :slight_smile:

But seriously. Is the amount of unit testing going on in the industry a known quantity? If we were to seek to identify widespread deficiencies related to this, would the typical deficiency be “no unit tests at all” or let’s say “a few tests that are so incomplete as to be meaningless compared to the scope of codebase”? Or maybe “some codebases well covered, while others will get covered by 2020, cross my heart and hope to die?”

How about the Indian outsourcing trend, what is this doing to the state of the unit test? E.g. they say that Windows Vista got built in India and it shows (yeah, that’s what they say - don’t look at me). So is the sheer might of Microsoft sufficient for getting decent unit tests out of South Asia?

Offshore coders aren’t capable of quality unit tests any more than they’re capable of writing quality code.

They give a DVD of the code to a guy with a T-Shirt that says “Unit Tester”, and he sneakernets it from the Development Lab right to the Production Server Room.

Third party applications are no better. Never have been.

I can count on one finger the number of companies I’ve worked for where unit-testing was a priority. And I’ve worked at a lot of different places.

A lot of companies use tools to analyze the code coverage of their unit tests. If enough companies were to publicly release this information, you might get an upper bound for the amount of coverage typically achieved. Of course, companies with this level of attention to unit testing are probably more thorough than those companies that do not place such an emphasis on it. Also, getting your hands on this data would be next to impossible, as I can’t think of a good reason the company would want to release this data.

I see. So sounds like there is indeed quite a potential market for bandaids.

Well, on second thought the “bandaid” already exists, known as the class of software and practices called QA. So I guess the next step in trying to capitalize on this problem would be to try study major limitations and imperfections, if any, of the way QA is being done nowadays.

“But automated testing is too expensive!” -> what makes you think these companies are willing to invest in any kind of solution?

ETA: the answer to the title of your OP is: Much larger than you might think. No, really. No, larger than that.

Superfluous Parentheses, could you please clarify your point? Are you saying that there are lots of companies with QA without any tools beyond screenshots and note taking? Or do you mean rather that, whereas those that have QA do in fact try to use some tools, a significant percentage of codebases have no QA of any kind beyond the actual programmer and his project manager?

If there were inexpensive and useful band-aids they would already be in place. There aren’t any even though lots have people have tried to come up with them.

hmm, actually it sounds like what Superfluous Parentheses meant to say is, unit tests are expensive. If I took him as commenting on QA, I may have been wrong. Sorry about that.

Both. And then there are the companies that do use some tools / automated tests, but not nearly enough to give the kind of guarantees that unit testing could theoretically give.

Testing, in whatever way, is often perceived (probably wrongly) as being “too expensive”. “Cheap enough” automated testing is probably never going to give you more guarantees than “it doesn’t immediately crash”. Keep in mind that you’re not selling to programmers; you’re selling to managers.

Bandaids of course can never give any guarantees, but they still can be demonstrably much better than open, festering sores. Or they can be much worse than that, if they are actively harmful. So I would argue that if the potential need is there it makes sense to try develop the ones of the first kind. But yes, of course it has to give us a lot more than just run the thing automatically and see it doesn’t crash.

Selling to managers is a separate issue, but then this may even be an advantage in some ways. At least for people who can do sales in the first place, myself unfortunately not being amongst them. My hypothesis here is that programmers are too conservative, busy and skeptical, whereas managers are technically clueless (meaning nobody told them that bandaids don’t work, period, end of story, and no contrary evidence whatsoever will be accepted), not as happy as they want to be with their own job performance and prone to magical thinking. So a competent salesman can sell bandaids to managers as “a small magic step that gives you some improvement in code reliability at minimal cost to all involved”. Which is basically what it would be, if it works. Nothing more, but also nothing less.

Moving to IMHO from GQ.

Colibri
General Questions Moderator

I see a huge need for bandaids, but I’m having trouble seeing the “market” part of it. The question any manager is going to ask is “how exactly is this going to help my bottom line?” If you can’t say an increase of X% in profits in the next quarter, then anything else you have to say doesn’t matter.

Many companies have enjoyed great financial success by skimping quite a bit on the QA/testing themselves and letting their customers do much of their software testing for them. Doing anything to increase testing costs isn’t going to fly at most companies these days.

code_grey, I don’t want to rain on your parade, but I believe the existence of unit tests are a serious problem industry problem. They are usually too simplistic to detect more than the most basic errors that would turn up in general pre-release testing. They can serve as a regression test in updated software, but any problems undetected in the unit tests then carry forward, while giving a false sense of security about the thoroughness of testing. IMHO the vast majority of serious software problems and the resources spent on identifying and fixing them are due to problems of complexity not at all addressed by unit testing. I find their only use as a means of forcing programmers to do the most basic level of testing that they might otherwise ignore, and your suggestion of selling ‘band-aids’ would then eliminate that minimal utility.

let’s start with the following argument and refine it from there:

  1. good programmers are hard to find (especially with the dumb HR coming with up with dumb requirements etc :slight_smile: ) and so are paid high hourly salaries
  2. your good expensive programmers are over-worked and project deadlines are slipping. (Show me a project that doesn’t have a slipping deadline anyway)
  3. so if using the bandaid saves 4 hours per week per programmer (the bug caught faster and did not blow up when least expected), let’s say it’s a saving of $240 per week per programmer
  4. this miraculous solution is yours for just … err, that depends on more knowledge of psychology and signing power of the typical manager than I possess. $50-100 per seat? $500 blanket license for the whole company of a certain approximate size?

Of course here we may also be coming up against the cost of the sale. Are there known statistics about how much do minor B2B sales cost?

Naturally an alternative solution would be to build this for free distribution at the expense of some R&D-for-public-good bureaucracy. That might make more sense for people having clout in such bureaucracies and be fully hopeless for those without it.

I think there’s actually a disincentive for third party apps to do much testing - buggy code (but not too buggy) sells support contracts and consulting hours.

I used to support an app which fills a Federally-mandated role in my industry. There are two vendors that make compliant apps, and they both blow. They both rake in the money on support contracts because the apps are terrible but every company in the industry must run at least one of them.

What a racket.

hmm, well, supporting a 3rd party app can mean different things depending on which parts of it are accessible and/or you are allowed to change. I am not sure what inkling meant.

E.g. if the 3rd party app exposes the database or even its full source code, the hypothetical bandaids would likely focus on helping programmers not very familiar with the system to troubleshoot bugs by themselves. Well, the exposure and permission to manipulate database I think is more likely, and here Red Gate has left some interesting low hanging fruit.

Not sure if an app can be meaningfully supported without both code and database, but I stand ready to be proven wrong and ignorant.

The very notion of overcharging 3rd parties is interesting from marketing standpoint. Jerkish behavior of the monopolist makes his customers more desperate and potentially interested in unusual solutions, whereas the boss can always hope that his own team will solve its own internal problems in due time and the costs involved would be more manageable and closer to reality than when he is taken to the cleaners by outsiders.

So I guess this points out a market niche that I simply never thought of before, due to limited experience.

While good programmers aren’t cheap, they’re not that expensive or hard to find.

I’m a project manager. Not all projects slip. Engineers don’t run projects anyway.

I have no idea where this number came from, but you failed to account for training time, extra time for builds, debugging false positives, updating and configuring software, additional hardware, and added complexity. And the fact that bugs in new features are rarely found by automated testing without specific unit tests that are written by engineers or QA.

No chance. You’d need to charge a lot more than that if your software is of any size and complexity. There are very few software houses that would buy this unless it integrated into their build tools, bug tracking, and QA software. You’re going to have to get that in place before you sell into big shops. And small shops are leery of using tools like this that don’t solve a major need. It’s too much effort for too little return.

I think you’re drastically overestimating the usefulness of this type of software and underestimating the effort it would take to create it.

I’m with **Telemark **and engineer_comp_geek on this; in order to sell this, you need to be able to define how it delivers value, and your hypotheticals don’t hold up well.

There are some really crappy codebases out there, but the people who have them aren’t interested in a band-aid solution. Either they don’t think there’s value to code quality, or they do. In the first case, spending more money to improve code quality is wasted, as far as they’re concerned - their outsource partner is CMM Level 5 certified, so of course they produce good code.

In the second case, they don’t want band-aids. They want real solutions, like functional test automation and code quality analysis. And there are already good solutions out there for providing those things.

For reference, the architecture team I work on is in the midst of doing just such a transformation.

Telemark, thanks for your points.

I will point out that standalone tools do exist, in both large shops and small. Red Gate database diff tool is standalone and I don’t know what it would mean for it to be “integrated” into anything. It certainly is not cheap, and I don’t know how they go about selling it.

Naturally, databases are a special case in more ways than one. For the build integrating in .net environment (unlike java) should not be a chore. I am not sure what you mean by integration with bug tracking - isn’t bug tracking done manually through a web interface?

QA is a separate topic. For now my thoughts seem to roughly divvy up into “new and improved Red Gate” and “what might become better QA, if I learn more of QA state of the art”. E.g. for QA I believe that state of the art tools for web apps are vastly superior to state of the art tools for desktop apps (well, not surprisingly, given the underlying mechanics). There are ways to design desktop guis to be more testable and there are some internal Microsoft attempts to make an instrumentable gui using a specialized widget framework, but I never heard of this getting lots of traction. So maybe a tool that would instrument .net desktop guis “as is” might make sense.

“False positives” and “training” sound like FUD for a sufficiently user friendly app. I mean we are not talking about Coverity reloaded in terms of the power and esotericism of models involved.

Bugs have been found since the beginning of programming, since before unit tests were invented. They are still found now in those shops where, as per engineer comp geek, unit testing is minimal or non-existent. Presumably “bandaids” would help finding them faster and easier.

All that being said, my position here should not be taken as a zealous sermon, more of a meditation on / desire to learn more of unfamiliar topics. I believe in the possibility of clever solutions to complex problems, including both low hanging fruit type of situations (like Red Gate) and “years of research needed for breakthrough” situations. And then there are various intermediate situations, e.g. long build times in dot net compiler for less-than-optimally structured codebases.

I am also aware that this is not Bell Labs circa 1980, and where I am personally is not even one of the pale modern imitations of that. But looking at the world around me, seeking to understand it and exploring solutions to apparent problems is still interesting in and of itself.