Increment: You’ve had a multi-decade career in this field, and when you began, software testing as a separate discipline from programming was still in its early days. How did you get started?

Graham: I was put into testing accidentally. I was going to be a teacher—a secondary school teacher. But I decided all I’d ever been was inside educational institutions, and maybe I should see what the real world was like. I was fortunate enough to get a job at Bell Labs in New Jersey. I was hired as a programmer as a master’s graduate and put into a test group.

The group I was put into was developing software to interpret LOFARgrams [Low Frequency Analysis Representation], which are the signals that come from hydrophones, which are microphones underwater that detect submarines in the Atlantic Ocean. It used to be if I told you that, I’d have to kill you. It’s no longer an official secret. If there was a signal at 50 hertz, it was a European submarine; at 60, it was America’s submarine, and so on. I was asked to write two programs: one to execute some tasks, and the other to compare some results. So I accidentally got into test automation right from the start of my career.

There were no tools, I imagine. Were you developing everything from scratch at that point?

That’s right. The culture was, if you were developing, you wrote some code and probably should test it yourself. It was unusual, actually, to have a software test group. There were only four of us in this group, and [only] two of the guys were actually people who were experienced at assessing these LOFARgrams and deciding what the correct answer was supposed to be.

If you wanted to have some tool support for the testing, they would call it a utility and it would do something that you wrote yourself. A number of other people in other organizations were doing similar things. The idea that you’d have something from outside the company that was actually going to help support the testing in any form of automation wasn’t around until, I think, the ’80s.

You had to convince and educate people as to the benefit of testing?

Yes. The first book solely about software testing [The Art of Software Testing] was published by Glenford Myers in 1979. (Bill Hetzel had published something earlier, but it was a conference proceeding rather than a book.) That was the book that said you’re more effective at testing if you’re trying to break things rather than prove that they work.

I went to my first testing conference in 1991, and there was a push for [an IEEE] Computer Society special interest group in software testing that started in 1989. I discovered there are actually other people who like testing. I had decided to specialize in it because I thought people were neglecting it. There was an increasing awareness of testing being a good thing to do.

My most popular talk back in those days was called “Test Is a Four-Letter Word.” Testing was viewed more as a necessary evil which shouldn’t be necessary than anything that would be good to do and to do well, and [something to] recognize the benefits of.

What new trends have emerged over your decades in the field?

The widespread acceptance that testing is a valid thing to do and is a reasonable career choice and is a respectable profession nowadays—that is the biggest difference, because testers really felt that they were at the bottom of everything, not [getting] a lot of respect.

One of the important things was beginning to get qualifications for testers—the certification schemes, which are done now by the International Software Testing Qualifications Board. They got started in the 1990s with an internationally agreed syllabus.

Do companies now understand how to integrate testing into the development life cycle?

In the older days, there was the waterfall model, which, as I understand it, was somebody saying, “Well, it’s not like it was—it would be nice if it was like this, but it isn’t.” And people would see the diagram and think, “Oh, that’s how we wish it was, so we’ll adopt that.” [Or], “We’re sorry, that’s the design.”

But that was definitely testing tacked on at the end. The V-Model— which was popular, again, in the early ’90s—was trying to make the point that tests are developed down the left-hand side and you would be done only if they all could be executed on the right-hand side.

But now, with incremental development models—and Agile, in particular, which seems to be quite widespread—testing is part of the definition of development for a sprint. People working in that model seem to have got it that testing is something you need to do and integrate with development.

In your book Experiences of Test Automation, you and Mark Fewster wrote, “Continuous integration is test automation; regression tests are run every day, if not more often.” Is that a hard sell?

Some companies are releasing software dozens of times a day, and you need to have some testing built into that. That testing also has to be automated if you’re going to be able to do that.

How do you test without knowing exactly what flaws you’re looking for?

There’s something called high-volume automated testing, where you generate loads and loads and loads of input. It’s always easy to generate inputs. But just putting an input into the system isn’t actually a test, because the essence of a test is not “Does it do something?” but “What does it do?” and “Did it do it with the right thing?”

Unless you have some idea of what it should be doing, then it might be a performance test or a stress test, because you’re just throwing stuff at it, and if it doesn’t fall down, then it passed the test. But if you want to know exactly what it is doing, and if you’re sending lots of inputs to it, you need to [be able to] tell whether it’s done the right thing with those inputs.

[Working] with the oracle assumption in testing, you know what the right answer is supposed to be. There are ways to use partial oracles to find [out] if your results are in the right ballpark; then you [can] assume they’re okay. And if they’re wildly out [of the ballpark], then you think, “Ah, I’d better ask a human to look at this.”

So, if you design tests in which elements always pass—and you get green check marks showing they passed—you have to scratch beneath the surface to understand whether the test is valid at all?

Exactly. There are some great stories in the Experiences of Test Automation book, [including] one by Julian Harty. There, they had tests that were turning green but they were doing it too quickly. They got suspicious and thought, “Well, hang on. What’s going on here?”

When they looked at the results, there was one test that had failed, and all the rest would turn green really quickly. They kept trying to fix this one test that kept failing. In the end they discovered that the test had failed because it couldn’t make a connection to the network. And all the tests that had turned green—when they couldn’t connect to the network, they said, “Well, I don’t care. I’m just going to turn green because nothing’s gone wrong, so it must be right.”

If you’re going to have confidence in what the testing is telling you, you need to have confidence in the quality of the testing.

How do you approach advising people about building the kind of comprehensive testing regime they need when so many different devices are running iterations of the same code?

If you’re automating testing so that you’ve got tool support to help run the tests, that’s one of the great advantages of having automation. You can run the same test for the same basic code but on different platforms. You get slightly different screenshots back, you get maybe slightly different placement of things, but it fits the same basic functionality. A lot of those tests are actually the same. The design of that automation has to be done in a way that makes that possible and makes that efficient.

How you draw a line when people are excited about the idea that they can automate everything? How do you say, “No, we still need humans involved”?

This idea of automating everything is just something [that] appeals to the silver bullet syndrome in people. But not all things should be automated, and there is a line to be drawn. It’s not necessarily one that’s terribly easy to see because it depends on a lot of factors. There are tests that are very useful to automate—ones, for example, that are run every day or every release and are just the same thing repeated over [and over]. That’s an ideal candidate because computers are really good at doing boring stuff over and over again.

But then you get into more complicated tasks, and maybe it takes a bit more effort to automate it. Is it worth spending the effort [to set up] your automation [until it gives] us a benefit? Or is that going to be too much time? If it takes two weeks to automate something that only takes 10 minutes to run manually, and it’s only needed once a quarter, then that’s probably not a good candidate to automate.

In Experiences of Test Automation, one contributor wrote about his work on the Microsoft Exchange team and how they were actively “dogfooding,” using their developmental software for day-to-day work. How do you determine whether it makes sense to use your own products while you’re in these cycles?

It’s probably always a good idea. I think the term came out of Microsoft. As soon as they had a new release of whatever it was they were working on, people in the company would begin using it. They would be, in effect, the first users; they would be the ones who would find problems. If you can find the problems yourself, within the company, before you’ve released it, then obviously you have a better-quality product to release to your users later on.

Should testers become programmers? And should programmers become testers? Or are those the wrong questions to ask?

I believe that any programmer should be testing their work to the best [of their ability]. Whether testers should become programmers— that’s quite a hot topic at the moment, because with the test automation tools coming along, even to use these test execution tools directly, you do need to be able to be a programmer. And it’s a very popular idea that testers should all be able to write the automation code directly in the tool, and therefore should become programmers.

Now, I actually happen to disagree with that, but I’m very much a minority there. I think this attitude that all good testers can write code is wrong. It is denigrating to people who perhaps have a business background and have been in testing for a long time. They have a lot of skills and a lot of knowledge about the domain. They like testing. They like breaking things. They don’t want to become programmers. If you force them to become programmers, they won’t be very good at it, and they won’t enjoy it.

I have no objection to people who want to do both. And I think in an Agile team, [having] people who can both do testing and be programmers is extremely useful. What I object to is the current prevailing attitude [that all testers should become programmers].

In fact, I was talking to somebody at the [STAREAST] conference last week who was very unhappy about their management saying that all the testers have to learn how to program. “Where should I start? Should I learn Java? Should I learn Python?” She didn’t want to. She actually wanted to stay as a tester. She said, “We’ve already lost about 10 people from our group because of this, and we’re losing the best people in town.”

I’m not all in with that. I don’t think I’ll change the trend, but I feel quite passionately about that.

What do you think is still missing from your discipline?

Two things which have almost always been missing in the testing world are how to measure all these benefits and costs—how to do it well, in particular with response to automation—and having it taught in universities. Some universities are exceptional at teaching testing; people get PhDs in testing. But the ordinary [computer science] degree you get often doesn’t cover a lot about testing.

I think it’s probably more important for high-level managers because the high-level managers set the tone [for an organization]. And if they don’t understand testing—if they don’t realize the benefits that could be had—they won’t invest in it properly, and it won’t give them those benefits.

What keeps things fresh in the software testing world?

[In] the testing community, it has felt like we’re in this together, I think partly because to start with it was not a very well-regarded thing to do. Now there are many communities of testers. It’s grown a tremendous amount in my lifetime. I certainly enjoyed being in it.