Photographic Testing: Lessons learned from Belgium Testing Days 2011

During Belgium Testing Days 2011, I learned a lot about teaching and - surprise, surprise - about software testing. When I took a walk around the city of Brussels with my camera the day after the conference, I reflected what I had learned while taking some photos and a little parable came to my mind which I would like to share with you.

Almost every person took a photo sometime. Lots of people nowadays think taking a photo is easy. The same applies to software testing. A lot of people tested some software and a lot of these people think, testing is easy. Well, that depends.¹

It is relatively easy to do simple regression testing (what should better be called regression checking). And it's relatively easy to take a simple photo. The advantage is: these things can be easily automated. This is state of the art in photography. Take a look at autofocus or automatic exposure. And automation in testing is becoming more and more popular (see TDD, BDD, ATDD and so on). The disadvantage is: both of them seldomly convey interesting information. A simple picture is almost always boring.² And regression testing has a 30% chance of finding a bug at most. To find out interesting information, you need (human) skill.² The first skill for a photographer and a tester is knowing his tools. There are lots of them in photography, often paralleled in testing.

Change your aperture
You can change the aperture of your lens, resulting in more or less depth of field. In software testing, you can change your scope. You can choose to take a broader or a more narrow view. See the application as a whole or pick out specific functionality or specific qualities.

(c) http://www.digicamguides.com/

Change your shutter speed
Changes in the shutter speed result in "frozen" or "fluent" images. In testing you can take a static or dynamic look on your product. Watch it with a static number of users or ramp up the load. You can spend less or more time on testing your area of choice. But beware! Sometimes longer exposure results in more blurry results ;-)

(c) Flagstaffotos 2007 licensed under GNU Free Documentation License

Change your focal point
Different focal points make near or far objects sharp. You can focus on different levels of depths of your product likewise. Just walk the happy paths or explore the whole functionality in depth. Test your system through the UI or "crawl under the skin" and test the API.

(c) Brien Szabo - http://www.natureimages321.com

Change your sensitivity
With modern cameras, you can easily adjust the sensitivity of your sensor, too, resulting in brighter or darker images with more or less visible details. As a tester, you can work at different levels of sensitivity, too. You can be open to any problems you see or you can be focused on specific types of bugs in specific areas, such as usability problems in the order function of your web shop.

(c) Andreas Simon 2011

Change your colour
With modern cameras you can easily adjust the white balance of the sensor, which influences the colour of your image in consequence. Your "testing balance" influences your results, too. More exploratory testing will show you more bugs than simple regression testing.

(c) Spiritia licensed under GNU Free Documentation License

Change your perspective
One of the most important means for a photographer - and a tester - is the perspective. You can move nearer to the subject or you can step back. You can go to the left or to the right. And you can kneel down or climb on a wall. Every step influences your picture and eventually you reach an interesting or even awesome photo.
In testing, you should change your perspective from time to time, too. Look at the application from the point of view of different user personas. "What would Homer Simpson do?" (Lisa Crispin). What would Mark Zuckerberg do? What would the kind hacker from your neighbourhood do? These questions lead you to interesting results.

(c) Andreas Simon 2011

Use special tools for special purposes
Extraordinary situations require extraordinary means. These are very specialized, mostly very expensive and sometimes heavyweight. But you just need them at time. Be it large lenses like the one shown below, be it a special GUI testing library, be it the cloud application for generating load on your servers, be it training, coaching or services.

(c) Ryan Foong 2011

Interaction
You see, there are lots of instruments. Some influence a single property of your image, some influence several aspects. Sometimes you have several means to influence the same attribute. The same is true for testing tools and methods.

Due to the scarcity of resources, e.g. light in photography and time in testing, the tools interact absolutely. When you want a "frozen" image, you probably have to make it more blurry, in order to get a well exposed photo. When you cannot raise your man power and you want to do more exploratory testing, you will have to do less (manual) regression checking for example. (Hm, maybe automating dumb tests could also be an alternative. But, that's just a thought ;-)

Back and forth
Take a "big picture" first, to get an overview and a first impression of your subject. Afterwards, select interesting details and try to work out their characteristics. Zoom in, approach them and watch them from different angles. Search for the pictures within the picture. Focus on a special area of your subject. Get an overview of your system under test at first. After having more information about the intended use of your customer, you can enter the areas of interest in detail.

Build on your social skills
Watch the people in interaction with their environment. It's often very interesting and insightful, and sometimes even funny.

(c) Wladyslaw Sojka 2007, licensed under GNU Free Documentation License

Ask for help. Sometimes other people allow you to reach an angle of view that you cannot reach on your own. Try to go to a high flat in the opposite building for bird view.
Talk to the best informed people of your domain of interest. In order to find interesting places in a city, a local resident might be your help. In order to find interesting areas of your application, talk to your customer. He can tell you which areas might be risky (from the business point of view). He can tell you what areas are hard to use or buggy.

Be economic
Sometimes the best picture is not affordable. Renting (or even buying) an elevating truck in order to get to the right point of view is not worth the cost. And asking for another thousand users to test your product might not yield any relevant insight.

(c) Michał Derela 2007, licensed under GNU Free Documentation License

Explore
Be it the city you are visiting, be it a software product. You might find interesting places, that are not in your guide book / your specification / user manual. Sometimes you will come to a dead end. Then, take a step back and explore another direction.

(c) David Stowell 2007, licensed under Creative Commons Attribution-ShareAlike 2.0

Look out for interesting differences (Belgians cross the street on red light; this is very extraordinary for me as a German guy). Maybe the user experience in your application is not consistent either.

(c) Andreas Simon 2011

Test the boundaries
Sometimes, good pictures are only possible under unusual conditions. A bathing beach will be much more attractive at sunset than at bright noon light. Wild animals are very shy, you have to wait for hours or days for the special single moment when they cross your way. Try out the edge cases in testing alike. These are the region where most of the defects are found.

(c) RonAlmog 2006, licensed under Creative Commons Attribution 2.0 Generic

Be creative

Play around with your tools to reach better results, both images and test reports. Sometimes you are surprised by reality. I was surprised that Manneken Pis for example is actually relatively small, I supposed it to be life-size.

(c) Andreas Simon 2011

Finally, give a try to serious photography. "Photography is learning to see [properly and precisely]" (Andreas Feininger). And this is on its own a very valuable skill for testers.

This article was inspired by Belgium Testing Days 2011, especially:

• Nathalie Rooseboom de Vries van Delft: "Unusual Testing: Lessons learned from being a Casualty Simulation Victim"
• Lisa Crispin's keynote: "Learning for Testers": What is special about you that makes you a good tester?
• Jan de Wael and Markus Gärtner: Testing Dojo Session in the morning of day 2
• Julian Harty's keynote: "Alternative Testing: Do we have to test like we always have?": Try "crazy" things

Another source of inspiration was the interview with Michael Bolton and James Bach (Agile Record 05, January 2011): bring knowledge from other disciplines into testing.




¹↑ This is probably the most important sentence I was told at university and during BTD2011
²↑ "I believe that photography at its best is an Art, and photo-technique is but a means to an end: the creation of the picture. Today, even a fool can learn to operate any of our modern foolproof cameras, and produce technically perfect pictures -- but is this knowledge really all he needs for taking purposeful and pictorially exciting photographs? Naturally, as in any other art, there are artists and there are dabblers. If photography really were nothing but the simple and purely mechanical reproduction process the majority of people still think it is, why are there so many dull and meaningless photographs around?" - Andreas Feininger