How to test a program

[Attaullah Shah]

Is there a systematic way of testing a program that one writes. Are there some easy steps, data sets, or some other ways to test a program. I am facing difficulty in testing a program where the number of steps is so high that manually developing a data set, and then extracting results are simply not possible.

[ben earnhart]

Depends on the nature of the program. If you can't reasonably generate fake data, just need to look around for an appropriate dataset, perhaps the one you plan on using it o in the end anyway. Auto.dta is a favorite, of course, but small and simple in its structure. The NLSY79 (potentially re-shaped) is good if you need something more complex. The GSS is good, too. But it all depends on what your program does.

[Jeph Herrin]

I find that if share a program with others, the bugs are swiftly identified....

[Clyde Schechter]

Perfect timing! I need an excuse to briefly procrastinate a day-job task that I'd prefer to avoid, and I'd love to get philosophical for a few minutes.

My views about programming arise from my experiences back in the 60's and 70's with FORTRAN and spaghetti code, and I have been strongly influenced by the ideas of Edsger Dijkstra and PJ Plauger. No amount of testing can validate a program (although even a small amount is likely to find a lot of its bugs!) At best, after extensive tests, we can say that the code is not obviously incorrect. But what we strive to attain is code that is obviously correct. To that end, the constructs of structured programming were developed. But beyond those details, some of which are getting dusty with time, there are important principles of programming that bolster our ability to write correct code. Among these are:

1. Transparency. Use syntax that says what it means and means what it says. Avoid "clever tricks" that speed things up, only to get us the wrong answers faster. Maintain a high comment to code ratio. Use whitespace liberally. Reflect the nesting of blocks of code in indentation (whichever style of indentation you like). Avoid using the same name to mean different things. Use mnemonic names when possible.

2. Algorithmic chunking. Write short blocks of code that do one thing well (better, do it perfectly). Build your program from those through calls and returns. Make the top level of your code look like a pseudocode, calling on other blocks of code. This can be done either in a top-down or object-oriented structure: the principle is the same. Short pieces of code that accomplish limited tasks get put together into a coherent whole. Different people have different rules of thumb about how many lines of code are acceptable before you split into pieces. I don't like those rules of thumb so much as the principle that each block of code should have a clearly defined, single purpose, and should be no longer than it needs to be to accomplish that (and only that), and that subtasks should be liberally delegated to separate blocks that get called. A good way to check this is to show each block of code, in isolation, to somebody who doesn't know your larger context and ask them to tell you what that block of code does. If they can't "back translate" your code into English (or whatever language they speak to you in) it may need refinement.

3. Information hiding. Global constants/variables are for the most part a plague and should be avoided where possible. Never make it possible for a block of code to access or, worse, change information that is not mission-critical for that block's functions. Specifically in Stata, local macros are superb (notwithstanding my gripe about the interpretation of undefined macros as empty strings). Use them early and often.

So, good coding practices will make your code more robust in the first place. And algorithmic chunking makes it possible to test each segment of the code one at a time. This can be done much more thoroughly than testing the entire thing because you cannot even imagine, let alone try out, every possible pathway through a long program. But you can easily imagine and test all the pathways through a short code segment. And short code segments typically have clear "boundary values" for their inputs where they are at greatest risk of going astray: you can test those carefully as well. Long programs are much harder to test.

So you can write and thoroughly test short, simple-single-purpose routines. And then you build your full program out of calls to them. If you have named your routines clearly, the top level will almost look like pseudo-code and the logic often will be either obviously right or obviously wrong.

This is probably not all that helpful as a response to your question because it lacks specifics. But I think these principles will serve you will in the long run if you put them into practice. I may be a little tangential here, but I hope I haven't totally hijacked the thread.

[Richard Williams]

These are all good suggestions. Try to have examples where you know what the answer should be, e.g. because you did simulations or you are copying a worked example from some other source. Then when you make tweaks to your program, make sure to rerun old examples and make sure you get the same correct answers.

[Attaullah Shah]

I appreciate all the responses.
To Clyde: Happens to me all the time that something that I really want to do will somehow make its way to the top of my priority list. Your suggestions regarding program writing are invaluable as they are derived from your long-term experience, one of the great teachers. I am new to this field and will take my time in learning these good principle, sometimes through suggestions from experienced ones, sometimes through trial and errors. By the way, if you have a program and exemplify through that program what you have suggested will help me a lot. Thanks for such a detailed reply.

To: Richard. Yes I am trying to different things such as making values of all variables equal to 1 or some other things. As far as actual data is concerned, unfortunately google has not helped me with that.

[Maarten Buis]

One useful tool for testing your programs is writing certification scripts, which are nothing other than .do-files in which you perform various tests of your program. In my experience, even a basic certification script can find many (embarasing) bugs before I make my programs public. A good place to start is:

William Gould (2001) "Statistical software certification". The Stata Journal, 1(1): 29--50.

You can download that article here: http://www.stata-journal.com/article...article=pr0001

There are a lot of tools inside Stata that help you write certification scripts. Inside Stata a good place to start is help cscript. This will give you an introduction on how to write certification scripts inside Stata, and links to helpfiles for specific tools.

[Attaullah Shah]

Thanks Maarten for the referenced material. Seems relevant to what I am searching for.