Developers Aren’t Really Testing

In my fifteen plus years as a software tester, I’ve had the pleasure of working with some great and talented developers.  The large majority of them have sat down with me and gone over issues and code; searching for the root of the bugs that were preventing us from getting some crucial feature out the door.  There were, of course, some who simply deemed that sort of thing ‘a job for QA’ and wanted nothing to do with such mundane tasks.  “After all,” they would say, “it works fine on my machine.”

Those developers hold a special place in my heart.

Really.

It’s because of them and the attitude they had, I believe, that have made me into a better software tester.  I had to, on my own, figure things out and learn what the code was actually doing.  When I would approach them with my findings showing where their code was broken, you would think I had earned the right to be as condescending as they were to me.  But no, a war with a developer was not the best thing for my career or the project.  It was enough that I walked away with some new learned skill about software development.

Not to say that I haven’t been tempted one or two times.  Remember that scene with the Chocolate Pie in the movie, The Help?  I totally identified with Octavia Spencer’s character.

It was also good to know more about the developers.  Over time, I would become accustom to finding bugs quicker when I knew which developer coded which feature.  Programmer A, for example, always had issues with getting shell scripts to work properly while Programmer B would be notorious for using beta libraries in his code and therefore not deployable on production machines.

When it comes to actually testing the software, however, even the best developers aren’t really testing.

They’re checking.

Developers code software to perform specific tasks.  Once they’ve written the code, for the most part, they confirm that the code, on its own, performs said task.  They check it. Once they know that the code does what it’s supposed to, they move and code some other task.  Once they’ve completed all the tasks, they’ll check-in the code and somehow deliver the code to QA for testing; confident it does what it’s supposed to.

We (the software tester) may check the code as well, but we’re also testing it.  We make sure that it not only does what it’s supposed to do, but ensure that it doesn’t do anything else.  We test to see how well it does perform, how fast, how slow.  We expose the breaks, the errors that aren’t handled, and the paths the software was never supposed to take but somehow does.

We’re testing.

I think the best response to finding a bug I’ve ever gotten from a developer is, “How in the hell did you find this bug?”

Praise indeed

Has being a Software Tester made me a Skeptic (or has being a Skeptic made me a Software Tester?)

So I’ve recently upgraded to an iPhone 4s.

I have to say, for as much as a techie, gadget geek as I can be; I’m not one to run out to immediately buy the latest cool thing as soon as it comes off the assembly line. We’ve both had our 3G version of the iPhone about 2 ½ years and were due for an upgrade.  The battery life was starting to shorten and apps were all starting to slow down, so I knew that my phone was soon ready to venture off to that retirement farm where all smart phones go after a few years.  This would be the same farm where our family pets have all gone off to, usually when we were away at school or summer camp.  To make matters worse, my partner’s phone was no longer able to charge; no matter which cable or device we tried (Tested? Checked?), the phone would not charge.  As he was leaving that afternoon for an overnight trip to Boston for work, the need for a phone upgrade was now forced upon us.

So we took a trip up to the closest Apple Store to see if his original 3G phone can be fixed.  Plan B was the upgrade.  It was 2 o’clock or so and we were told that the next Genius Bar appointment available was 6:30 that evening.  Time was not on our side, so Plan B took effect; bring on the upgrade.  Two iPhone 4s phones, 32GB plus Apple Care (because you know one of us is going to drop his phone at some point in the next two years and break it) plus tax rang up to about $865.

Side note: I would love to see how it would work if, instead of a support desk and QA team at work, we had a Genius Bar where our developers would have to make appointments to get their builds tested.  I’m thinking that particular software model would be tossed out sometime during Day 1.

So cut to that evening; my partner made it safely to Boston and we were able to use the very cool FaceTime feature that night.  Both phones were working great.  I had backed up my old phone and restored all of my apps to my new phone.  I did spend almost two hours re-organizing them which was annoying, but overall the phone was faster and the battery was lasting much longer than my old phone (which was about to start its retirement on that farm I mentioned.)

Monday morning, on the way to work, it began.

No SIM Card was the message that greeted me when I thought I had just gotten a text message.  WTF?  48 hours in and already there was an error.  So I did what anyone in this day and age does when their new gadget starts flaking; I went to Google.

Oh yes, I do remember this from when the new phones first came out; a SIM issue on some phones.  For every forum or You Tube video post describing this issue, most dated as old as last October and November, you would have thought Apple would have gotten a handle on this already.  The new firmware, they said, should have fixed this.  I had the new firmware and still had the bug, so nope, nice try.  Also, there use of the word ‘should’ began to raise my skepticism as a tester; I’ve heard developers use this word far too often.  As in “This build should have fixed that bug” or “This should be the final release candidate.”

On Monday when this started happening, Power cycling seemed to work.  At first, I thought it was just a one off issue and, as I do with most bugs, I keep a close eye to see if/when it happens again.
I didn’t have to wait long.

That evening, it happened again.  I didn’t even notice at first as I was on WiFi and Words With Friends was still working.  Once I realized it, another power cycle got me back up and running.  No longer a one off issue, I figured I had a real problem with the phone.  But the late hour caused me to take an “I’ll deal with it tomorrow” attitude.

The next day I had no issues at all.  I had almost forgotten about it until the train ride home; Invalid SIM now appeared on my phone.

Once I got home after a train ride spent power-cycling my phone repeatedly, my first response was to go right back to Apple Store and demand first-in-line service at their alleged Genius Bar.  A phone call to the store, however, was what I ended up doing.  I quickly by-passed their automated phone system to get a real person on line.  I explained the issue and my frustration was quite clear.

Apple Support: “Well we have no available Genius Bar appointments this evening sir, but I can look to see if any other stores nearby are available.”

Brian: “I already did that.  The closest one is at the Fifth Ave. store back in the city.”

AS: “Oh.  Well if you come in, we can replace your phone with a new one”

B: “What’s to say that this issue, which should have already been fixed by Apple, won’t happen to this new phone?”
      Notice my skepticism and pointed use of the word should?

AS: “Well, I’m sure that this won’t happen again if we replace the phone.”

B: “Then why has it happened to over 40,000 people since last November?”

AS: “It should be fixed with the latest firmware. Do you have the latest firmware?”
     There was that word again; should.

B: “If I get a new phone, will I have to take two hours to arrange all the applications again?”

AS: “Did you back it up to the cloud?  If you backed it up, your apps should be restored and organized”
  Again, should.

B: “Yes, it was backed up the first time and I still had to re-arrange the applications.”

AS: “Well, it should have worked if you backed it up”
   Grrrrrr

While this phone conversation was going on, I was power cycling my iPhone and it finally came back with a valid SIM.  I carefully placed the iPhone down and ended my conversation with the Apple support person, telling her I would just go to the Apple Store on 14th Street in Manhattan when I went into work the next day.

Invalid SIM appeared to text me as I was on the train the next morning.  No matter, I thought, I’m going to the first available timeslot and getting this resolved; which I did.

Genius Bar spent all of 20 seconds with me before telling me to just replace the phone with a new one.  It was my only option as I didn’t even want to look at this phone anymore.  I felt bad for my old phone, enjoying its first days on that lovely retirement farm with all the other old phones.  I spent nearly two hours at the Apple Store, but I left with a working phone, so I think it was worth it.  I also figured out the REAL solution to Apple’s Invalid SIM / No SIM bug and it’s not just ‘Get a new phone.’

Step 1: Turn on and activate the new phone.

Step 2: Download and apply the latest Firmware upgrade (currently iOS 5.1)

If you have already downloaded the latest Firmware on the iPhone 4 or 4s and are still having the problem, start at Step 3. Make sure you have your phone backed up to Apple Cloud before doing Step 3, it will save you some headache later.

Step 3: Connect your phone to iTunes and Restore it to Factory Settings.
 Or
Step 3: Under Settings --> General --> Reset --> choose Erase All Content and Settings.

 This will set your phone back as if you just opened it up from the box except you’ll have the latest Firmware already installed.  The Firmware plus resetting to factory settings somehow fixes the carrier issues the SIM cards in the iPhone were having.

Step 4: Once the reset is done, chose “Restore iPhone from the Cloud” and login to your Apple account.  Choose your latest backup and your phone will be restored to how you left it; organized apps and all.

It should work.

Why Do We Test?

There have been a lot of recent talks and articles regarding the whole “Is Test/Testing Dead?”  Jon Bach gave a great keynote at STPCon in New Orleans last week regarding this.  “Testing isn’t dead,” he said, “but Bad Testing should probably die.”  Overall, testing as a practice, as a skill, seems to be on the decline as the actual activity is being done by just about anybody. 

But can just anybody test?

It depends.

I fall back on that response a lot, sorry, but it’s useful.  In this case, it depends on what you’re testing. 

A better question is can anybody test software?

No.  I personally don’t think so.  It takes someone with a good analytical mindset, technical knowledge, and an innate drive to find things out to be a really good software tester.  Again, that’s just my personal opinion based on my 15 years of software testing and working with a wide range of testers, developers, support engineers, and technologists.  Overall, I think some testers lose sight of the basic question, which I’ve conveniently put in the title of the post.

Why Do We Test?

A good interview question I’ve used in the past (along with my famous “How do you know when you’re done testing?”) I’ve heard many answers in response.

“To see if the software works”

This is a completely valid, logical response; also, the one that I hear the most.  Essentially, yes, we are confirming that the software works.  But isn’t there more to it?

“To prove the developers wrong”

I chuckled at this one especially because I’ve heard it on more than one occasion.  I’m not a fan of the adversarial tester versus developer mentality.  I try to squash it whenever I come across it in fact.  I like to think that we all have the common goal of delivering quality software to the client and should behave as such.

“Because it’s my job”

This one was an actual response from an interview I gave.  I had to quickly check to see if the candidate was in fact a two dimensional cardboard cutout.  If you’re just testing because you were told too, how well are you going to do your job? How inventive will you be in your testing? Going through the motions of testing is not time well spent at all.      

Ultimately, we test to learn something about the software that we don’t already know the answer to; to confirm or disprove something about the software that we think or had heard about. 

We test to learn.

The point of each and every test that we run on software should be to learn something about it.  Not just if it works, but how it works.  What happens when we expose the software to specific conditions?  How many users at one time will break it? Slow it down? Can I bypass the password protection? Will it run on my Mac just like it runs on Windows? Does the website look better on Internet Explorer than Google Chrome? Perform better on FireFox?

Endless amounts of questions can be answered by our testing.  And with each test, we learn something more about the software.  Sometimes, we learn things we weren’t even looking or testing for but we take the knowledge the test delivers to us and use it to create more tests.

At least we should be

Is Testing Dead? Are we doing our part to kill bad testing?

Ask yourself, for each test that you run, what have I learned from this test?  Did I learn anything new about the software?  Is this test telling me something that no other test is telling me?

If you have a suite of tests that tell you nothing about the software, ask yourself, Why am I running this test if I learn nothing about the software when I run it?

If you have questions about the software and you have no tests that give you the answers you’re looking for, ask yourself;

Why haven’t I written those tests yet? 

British Leap Bug

So I have to admit, I’ve become accustomed to my GPS.  In my car, while driving, the GPS has gone from being a distraction to a normal part of my dashboard display.  For the most part, I know where I’m going, but having it there to tell me about traffic jams and congestion, even beeping at me when I go over the speed limit are all a welcome array of features. 

So when I returned home from New Orleans and STPCon (which upcoming blog posts will talk about) I immediately noticed that the GPS map was way off and the display constantly read ‘waiting for a valid GPS signal.”

What could be the problem?  We immediately thought of and dismissed Sun Spot and Solar Flare activity.  As interesting as that might be, we figured that our cell phones and other devices would also be experiencing issues and they were all clear.  So we then narrowed it down to a recent firmware upgrade; since the upgrade, the GPS error appeared. 

I didn’t know about the upgrade (occurring while I was away at STPCon) otherwise I would have gone with that first rather than the Solar Flare hypothesis.

A quick search of TomTom’s webite revealed that we were not alone in our GPS signal issue.  From their forums:

We have identified the root cause of the issue which has caused some of our models to not get a GPS position. The issue is caused by a ‘leap year’ bug in our GPS firmware. We’re working on a fix and will provide a further update as soon as we know more.Since Saturday 31st March a limited number of models have been having an issue getting a GPS position. We’re looking into the cause and will post an update here as soon as we know more. If you click Notify me at the bottom of this FAQ, you will automatically receive an update.
We hear from some of our customers that a reset of the device temporarily resolves the issue

A ‘Leap Year” bug.  This is great.  The kind of bug that comes along only once every four years really.  Even better, because TomTom is a UK company, the Leap Year bug didn’t appear until the UK entered British Summer Time this past weekend, not when the US sprung ahead a few weeks ago.

Resetting the device didn't work for us by the way.

On testing this, though, I can identify with them.  Testing for Leap Year issues is always an interesting challenge. 

STP - The Arrival

An early Sunday afternoon flight is a great time to travel.  Arrived safe and sound in New Orleans at the Sheraton Hotel on Canal Street for the Software Test Professionals conference.  

While Day 1 is really Tuesday, tomorrow (Monday) is the pre-conference workshops.  I’m looking forward to all of them and hope I won’t be too overloaded before the actual conference begins. 

A big plus; the view from my room is fantastic!

STP Bound !

All next week, I'll be in the gorgeous city of New Orleans attending the Spring Conference of Software Test Professionals.  Be on the look out for some live blogging and daily summaries! 

Testing, one, two, three…..

So these past few weeks have been more about the management side of my job, rather than the testing side; I was starving for some great testing topic to blog about.  Well, who would have thought that a great testing debate would literally come to my office door?    

A colleague of mine, a developer on another team whom I’ve worked with before, came by my office and asked for five minutes of my time to “Settle a testing debate between himself and another developer on his team.”

A break from the monotony of management?  A chance to engage in a discussion where the outcome was brining two developers into the warm, bright light of testing?  

Try and stop me. 

So the debate was, essentially this:

The first developer thought that the test suite they had should be re-organized so that the tests run in a specific order; establishing dependencies on the outcome of each test and then only running the next test should the previous one pass. 

The second developer thought that the work involved in doing so was way too much and that the first developer was “crazy” 

His words, not mine.

So they asked me, an impartial (read ‘not on their team’) observer and ‘testing expert.’ 

Again, their words, not mine. 

But I totally ate up the compliment and offered my voice to their debate.  I told them:

“You’re both right! And wrong!”

At this point, they were re-thinking brining me in on this, I’m sure.  So I explained what I meant.

Organizing your tests is a really good idea.  It will help to keep track of your coverage and identify areas, when the tests fail (or don’t met the expected results) to zero in on.  However, I told them, you’re crazy to make all the tests dependent on the ones before. 

Developer A, the test organizer proposer, had offered an example of what he was trying to do. 

“Say you have a function that squares two numbers and a test associated with it.  You’re not going to run that test if the test for the multiplier function, which the square function is using, fails.  So the tests should be ordered and linked somehow so we don’t waste time fixing a square function when the real issue is the multiplier function is broken!”

I was beginning to see that maybe we were looking at this the wrong way.  We should, I said, look at this not from the tests, but from the functions point of view.  What you want to do is not organize the tests, per se, but rather organize what they’re testing; group them by the functions being tested.  Since these tests were essentially core unit tests, this was probably the best way of doing it.

I then told them my planned idea for my own team and the couple of hundred tests case that I was looking at.  I also had to pull in my Sci-Fi geek cred at this point.  Ok, we’re a tester and two developers debating test methodology on a Friday afternoon; there was bound to be some geek gauntlets being thrown.

My idea, which I thought would work for them, was to identify the core functions of their system and the tests associated with them.  Call these your “Level One” tests, which should all pass prior to running your Level Two tests and so on. 

So when you test, it’s like you’re running a Level One Diagnostic.

“Nice. Star Trek”

Like I said, there was bound to be a Geek Out at any time here.

So test organization I really not about the test, but what you’re testing.  If you order your tests to verify the core, crucial pieces first, then move on to secondary, tertiary, etc. then you can ensure that the basics are solid before testing the complex.  This will help the developers as well to identify what was changed/broken/alerted in the last build.

Developer B, by the way, still thought it was a crazy amount of work to do.