Jason Pollock, from the prag-prog Yahoo mailing list:
I tried espousing oracles once (on the Yahoo software-testing mailing list). :) I had the following bit of wisdom beaten into me: you either put heuristics into your oracle, or you end up with software just as complex as the software under test. So, you either end up having something that isn't an exact oracle, or you have to debug your oracle first. :)
They said that parallel implementation had the same problem. Even if they were developed by different teams, the software had a habit of frequently failing in the same locations.
So, yes, using this will result in software that will behave the same as
the oracle, but your oracle may be wrong!
...
[So what's a better option, I replied...]
...
Not relying on an oracle as the be-all end-all of test generation? :)
I'm pretty pragmatic. If you've got something that requires bulk, exhaustive testing, then an oracle is probably the only way you can go.
For example, exhaustively testing a new floating point engine, by using the already existing, peer-reviewed, tested and heavily analyzed
version.
If you're looking at testing something that's nasty with a lot of exceptions, you might be better off re-thinking the design. Using Excel as an example, even with their oracular testing, I still see problems with recalculations. :)
The discussion on the other list eventually ended up at:
Susan S. Brilliant, John C. Knight, Nancy G. Leveson: Analysis of Faults in an N-Version Software Experiment. IEEE Trans. Software Eng. 16(2): 238-247 (1990)
http://dependability.cs.virginia.edu/bibliography/00044387.pdf
After much teeth gnashing from various people on the list over the quality of the research in the paper, Cem Kaner posted a message that summed it up best:
“N-way programming may be more or less efficient than competing methods for exposing certain types of errors. I think it is pretty widely accepted that we can find *some* errors this way. However, N-version programming/testing/data-entry/direction-pecking/etc. does not provide a solid base for assessment of a product or system's reliability, or assurance of a low error rate or of only non-serious errors”
http://groups.yahoo.com/group/software-testing/message/2179?threaded=1
tags:
ComputersAndTechnology