# Poll: Do you recognize the term "Fuzz Testing", and are you a Software Developer **URL:** \ **Category:** In My Humble Opinion\ **Created:** [February 9, 2024, 10:43pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369 "2024-02-09T22:43:12Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Balance](https://avatars.discourse-cdn.com/v4/letter/b/ccd318/32.png) [@Balance](https://boards.straightdope.com/u/Balance)\ **Post date:** [February 10, 2024, 10:34pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/21 "2024-02-10T22:34:42Z")
> [@Dr.Strangelove](#): > > I’m a little surprised that less than half of developers have heard of it. It’s not _that_ common a general practice in my experience, but I’d hope most developers follow security news just to keep abreast of things, and it’s very commonplace there. Judging by the followup posts, most of us developers who said we weren’t familiar with the term are familiar with the method, just not by that name. I’ve heard it called “dump testing” (from dumping garbage into it) and “Maggie-testing” (simulating input from a notional toddler named Maggie with an input device), or (prosaically) just “input testing”. Jargon diverges with people working in different companies and specializations. Just because coders working in, say, a services management software lane don’t call something by the same name the security wonks use doesn’t mean they don’t know about it. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [February 10, 2024, 10:41pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/22 "2024-02-10T22:41:19Z")
> [@Dr.Strangelove](#): > > Fuzz testing is a form of black box testing. I.e., you don’t need to know anything about the internals–you just feed in inputs and see what happens. So it doesn’t require the same degree of domain expertise as the original software development did. But, you do need to know what it’s supposed to do. Feed garbage into the program, and it’ll do… something. Maybe it does what it’s supposed to do, or maybe it does something bad. But if, given a random input, you knew what it was supposed to do, you could just write a program that did that: And that would be a replacement for the code you’re trying to test. Or is it just that, of the things it’s not supposed to do, there’s some small subset that are things you know it’s not supposed to do? In that case, you could catch bugs that cause it to do those known wrong things, but still might not catch bugs that cause it to do other wrong things. ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [February 10, 2024, 11:05pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/23 "2024-02-10T23:05:27Z")
> [@Balance](#): > > Jargon diverges with people working in different companies and specializations. That’s fair, but still–it seems that anyone that follows security news at all would have encountered it. I’m not in security either, but I do read about the major exploits, and fuzz testing comes up a lot. Google’s Project Zero is a major proponent and has found a large number of exploits via that method. Maybe people have seen it before, but the jargon didn’t stick for whatever reason. That happens too, IME–you skim past stuff that you don’t immediately recognize. ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [February 10, 2024, 11:17pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/24 "2024-02-10T23:17:10Z")
> [@Chronos](#): > > Or is it just that, of the things it’s not supposed to do, there’s some small subset that are things you know it’s not supposed to do? That’s certainly a big part of it. They generally shouldn’t crash, for one. If an image decoder library crashes on a malformed file, then there’s something wrong with it–it needs to do more input validation or something. Fuzz testing can also be used to catch regressions. Suppose you write some code and it passes your fuzzed test suite–great! Now keep up that automated testing forever (with the same random seed) and compare against the “gold” outputs. If you ever add a bug, the fuzzing might catch it. Even if nothing seems “wrong,” if there’s any difference at all, there might be a problem. Fuzz testing isn’t a replacement for other types of testing, but can be a good addition. And it’s pretty cheap in terms of developer time. Unit tests are time consuming, and are themselves never the same quality as the code they’re testing, so there’s always a chance they’re really catching a bug in themselves. But a generic fuzzing framework can be written once at a high quality level and then used to produce an unlimited number of variations. ---
**Author:** ![echoreply](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/echoreply/32/3641_2.png) [@echoreply](https://boards.straightdope.com/u/echoreply)\ **Post date:** [February 10, 2024, 11:56pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/25 "2024-02-10T23:56:13Z")
> [@Chronos](#): > > But, you do need to know what it’s supposed to do. Feed garbage into the program, and it’ll do… something. I’m not a developer; I only manage some systems. Here is a real world example of when I used fuzzing (as I knew it). Using real names, because let the guilty be shamed. This was a long time ago, back when there was a big difference in memory and speed between servers and desktop computers. SPSS came out with a Unix version of their statistical software. One method of interacting with it was as a service running on a server. Windows computers would run SPSS, which would submit code to the service, which listened on a particular port. As a systems administrator I was very hesitant to have some random software running on my server that allows anyone to talk to it. So to test I ran the extremely simple fuzz test of ```auto nc localhost 12345 < /dev/urandom ``` All that does is feed random data into the port where SPSS was listening. The correct response is for SPSS to do its best Gary Coleman impression, and then close the connection. Instead, SPSS crashed with a segmentation fault. That strongly suggests to me that at the absolute minimum it is trivial for someone to implement a denial of service against the SPSS software. More likely, it is possible to send specially crafted data (instead of random) which gets the SPSS service to do anything you want. So running one simple fuzzing command let me know this was not something I wanted to offer to my users, and if they needed it, I would have to be extremely careful to insure it was isolated from potential bad actors. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [February 11, 2024, 1:58am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/26 "2024-02-11T01:58:11Z")
Come to think of it, a black-box test (of any sort) isn’t very thorough, either. For instance, for a lot of software, if you try to feed it random data, the correct response is for it to give an error message of some sort. And on the black box level, the error message might be all that you can tell it’s doing. But what if, with the right (wrong) input, it gives an error message, and starts overwriting files, or something? > [@echoreply](#): > > So running one simple fuzzing command let me know this was not something I wanted to offer to my users, and if they needed it, I would have to be extremely careful to insure it was isolated from potential bad actors. Sure, there are always cases where Test X definitively proves that you do have a bug. In fact, for any given bug, there will always be some test that proves that you have it. But no matter how thorough you are, you can never be sure that you don’t have any bugs. I guess the bottom line is to just run as many tests as you can (limited both by your imagination in coming up with tests, and by the testing resources you have available), and in that context, a cheap, easy test like fuzz makes good sense. ---
**Author:** ![Typo\_Knig](https://avatars.discourse-cdn.com/v4/letter/t/cdc98d/32.png) [@Typo\_Knig](https://boards.straightdope.com/u/Typo_Knig)\ **Post date:** [February 11, 2024, 2:42am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/27 "2024-02-11T02:42:29Z")
This is all great info, everyone! The term is not as well known as I’d thought. I now have some alternate names for it, some example tools, and some examples of how it helps everyone. Ignorance fought! And keep it coming! ---
**Author:** ![Balance](https://avatars.discourse-cdn.com/v4/letter/b/ccd318/32.png) [@Balance](https://boards.straightdope.com/u/Balance)\ **Post date:** [February 11, 2024, 5:00am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/28 "2024-02-11T05:00:59Z")
> [@Chronos](#): > > But no matter how thorough you are, you can never be sure that you don’t have any bugs. > [@](#): > > And the last bug in sight, an ant passing by, > Saluted his tombstone, and whispered, “Nice try.” (From “The Last Bug”, by Lou Ellen Davis.) ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [February 11, 2024, 6:21am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/29 "2024-02-11T06:21:33Z")
> [@Chronos](#): > > But no matter how thorough you are, you can never be sure that you don’t have any bugs. [https://lamport.azurewebsites.net/tla/tla.html](https://lamport.azurewebsites.net/tla/tla.html) [https://tla.msr-inria.inria.fr/tlaps/content/Home.html](https://tla.msr-inria.inria.fr/tlaps/content/Home.html) ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [February 11, 2024, 6:58am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/30 "2024-02-11T06:58:44Z")
> [@Sam\_Stone](#): > > Fuzz testing at least adds an element of randomness to the testing. But the fact that so many developers have never heard of it backs up the idea that not all developers are educated in the latest QA things. I never heard of it either - my BS is from 1973. But here is something to support your point on randomness. New microprocessor designs get tested by running programs on a simulated model of them. This is done on a compute ranch with thousands of processors all running tests. (It helps to be a company that makes computers to do this.) The best way of testing is not to run tests written by the designers, but to assemble random snippets of code to run. This finds all sorts of corner cases developers never think of. The programs have to be generated to make some sort of sense - so they are pseudo-random, not truly random, but this is the standard method these days. When are you done? When the bugs found per week levels off at a low value. I have heard of bebugging, which is when a team inserts bugs inside a supposedly finished program, and then measure how much of these known bugs the test cases find. The simulation of tests for hardware faults works exactly in this way. ---
**Author:** ![scudsucker](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/scudsucker/32/14101_2.png) [@scudsucker](https://boards.straightdope.com/u/scudsucker)\ **Post date:** [February 11, 2024, 7:14am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/31 "2024-02-11T07:14:06Z")
> [@Cervaise](#): > > after googling for a definition I certainly know the method, just not by that name. I’m a software developer and this exactly my response. But we tend not to use it because unit testing and functional testing already includes testing for error situations. (And we have QA for the rest!) ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [February 11, 2024, 9:52am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/32 "2024-02-11T09:52:07Z")
> [@DPRK](#): > > [The TLA+ Home Page](https://lamport.azurewebsites.net/tla/tla.html) Good. We can show that our programs are correct as long as the specification is correct. Now we just need a specification specification language to prove that the specification is correct. But then we need a specification specification specification to show that our specification specification is correct… ---
**Author:** ![TriPolar](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tripolar/32/3008_2.png) [@TriPolar](https://boards.straightdope.com/u/TriPolar)\ **Post date:** [February 11, 2024, 10:31am UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/33 "2024-02-11T10:31:39Z")
The thing to remember is that automated testing can be more thorough than manual testing. It’s much faster, can consistently test known problem cases and find new problems by providing random input. ---
**Author:** ![gnoitall](https://avatars.discourse-cdn.com/v4/letter/g/bb73d2/32.png) [@gnoitall](https://boards.straightdope.com/u/gnoitall)\ **Post date:** [February 11, 2024, 2:17pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/34 "2024-02-11T14:17:29Z")
It’s specifications all the way down. ---
**Author:** ![74westy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/74westy/32/3950_2.png) [@74westy](https://boards.straightdope.com/u/74westy)\ **Post date:** [February 11, 2024, 5:04pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/35 "2024-02-11T17:04:39Z")
> [@DPRK](#): > > > [@Chronos](#): > > > > But no matter how thorough you are, you can never be sure that you don’t have any bugs. > > [https://lamport.azurewebsites.net/tla/tla.html](https://lamport.azurewebsites.net/tla/tla.html) > [https://tla.msr-inria.inria.fr/tlaps/content/Home.html](https://tla.msr-inria.inria.fr/tlaps/content/Home.html) > [@Donald Knuth](#): > > Beware of bugs in the above code; I have only proved it correct, not tried it. … ---
**Author:** ![Sam\_Stone](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@Sam\_Stone](https://boards.straightdope.com/u/Sam_Stone)\ **Post date:** [February 11, 2024, 5:38pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/36 "2024-02-11T17:38:38Z")
I heard a rumor once of a program that was completely bug free. ---
**Author:** ![pjd](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pjd/32/3009_2.png) [@pjd](https://boards.straightdope.com/u/pjd)\ **Post date:** [February 11, 2024, 5:41pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/37 "2024-02-11T17:41:24Z")
That’s probably the “hello world” program. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [February 11, 2024, 6:27pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/38 "2024-02-11T18:27:53Z")
You can, perhaps, be confident in the source code you wrote for “hello world”. But that source code will call some routine like printf(), or something equivalent to it: How confident are you that printf() is completely bug-free? And how confident are you in the compiler you use to compile your “hello world”? Or the compiler that compiled that compiler? Or the opcode built into the processor (or processors) that are running all of those programs? Or, since modern processors are too complex for any human to understand in their entirety, the programs (or compilers or opcodes) used to design those processors? It’s maybe not quite turtles all the way down, but it’s still a heck of a lot of turtles. ---
**Author:** ![SunUp](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@SunUp](https://boards.straightdope.com/u/SunUp)\ **Post date:** [February 11, 2024, 7:41pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/39 "2024-02-11T19:41:13Z")
> [@Chronos](#): > > You can, perhaps, be confident in the source code you wrote for “hello world”. Most versions of hello.c have bugs, if there is a requirement to alert the user to an error if printf() fails, such as by being redirected to an output file on a full-to-capacity disk. How about a show of hands: Who here has _ever_ checked the return value of printf() for an error or incomplete write condition? Heck, let’s make it a poll: _Poll ([view on site](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/39))_ ---
**Author:** ![What\_Exit](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/what_exit/32/10652_2.png) [@What\_Exit](https://boards.straightdope.com/u/What_Exit)\ **Post date:** [February 11, 2024, 7:52pm UTC](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369/40 "2024-02-11T19:52:59Z")
That is a poll for C and C related programmers, not programmers. I think it is failing its error check. [Previous page](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369.md?page=1) [Next page](https://boards.straightdope.com/t/poll-do-you-recognize-the-term-fuzz-testing-and-are-you-a-software-developer/997369.md?page=3)