Planning Ahead

I just flew over the North Pole.  How amazing is that?

Over this past year, I've been working on the most challenging project of my career.  It’s taken so much focus that I've let a good number of things slip; including this blog. 

“I’m so busy with work that I can’t focus on my career!”

I said that to my boss a few months ago.  It was meant to be more ironic funny rather than the tragic funny that it sounded like  I’ve been doing software testing for roughly 16 years now and I’ve been so reactionary, that planning ahead to what I would do next seemed like something I would do “when I got older.”  A couple of years ago, I started to focus on that and immersed myself into the Software Testing community.  Honestly, I had no idea there even was a community, but there they were.  Veterans and ‘newbies’ who all shared the same passion, same focus and drive, that I never knew I had until I started finding bugs all those years ago.  My sanity was restored and my career seemed to actually be something real suddenly; not just a notion or line item on my resume.

Then the company I work for starting work on implementing a new Trading Platform; a Global Trading Platform not just in name but in actual fact.  On paper, it seemed like another project; similar to the ones I’ve done in the past while being able to focus on several other things in both my personal and professional career.  Global was just another word; an adjective to give the project more impact.  Right? 

To answer that, I just need to restate: I just flew over the North Pole!

“The pressure it increases the closer that I get.  I could almost go to pieces but I’m not quite there yet.  See I've been braving crazy weather, drowning out my cries.  I pull myself together. I’m focused on the prize.”

I’m on my way to Hong Kong to meet with our APAC (Asian Pacific) trading desk and support staff.  This comes only two weeks after coming home from my second trip to London ; the first trip there only ending a week prior to that.  In total, a little under 40,000 air miles since August 30^th. 

I’ve literally flow around the globe for this project.

I was supposed to speak at the Software Test Professional conference in Phoenix Arizona next week, but I had to pull out.  I struggled with that decision as I was looking forward to touching base with that community that so grounded and focused me not that long ago.  But after a week in Hong Kong and work to do when I get back to help ensure this projects continued success (we’ve already achieved some great milestones, but miles to go and all that) I needed to excuse myself from the conference.

I’m also missing New York Comic Con right now.  So sacrifices are being made all around.   

There have been so many things that I’ve learned (and relearned) in the past year as this project unfolds.  I’ve enough material for numerous blog posts, conference topics and maybe even a book on software testing.  Knowing that there is so much more coming out of this project is what’s getting me through it; that and the amazing people I’ve been working with who support, challenge and provide the focus needed to get anywhere on this project.

Now I just need to find the time to put all of this together.  To find a way to weave work and career together without burning out on either. 

 “Will I remain the same or will I change a little bit?  Will I feel broken or totally complete? Will I retain my name when I the biggest, hugest hit? Or will I blend in with the rest of the street?  The people all are pointing.  I bet they never guess that the saint that they’re anointing is frightened of the mess.  But even though I fear it, I’m playing all my cards.  Baby you are gonna hear it when I give them my regards.”

-          Song lyrics are from Broadway, Here I Come  from the TV Show SMASH

What I’m Thankful For

This past year seemed to fly by so fast.  I mean, I’m still getting used to writing 2012 on my checks, mostly because who writes that many checks anymore?  But still, this past year has been so full of things, events, and people that I must take the time to look back and reflect and just say Thank You!

I’m thankful for my wonderful family!  As we’ve all gotten older and moved further away, you really take for granted the once often gatherings that now seem more and more infrequent.  Just sitting around a table with good food, good wine, and great conversation can be the best time and we should remember to cherish it.  I’m amazed at how quickly my nieces and nephews are growing up and so happy that I got to see them this past year.  Even my nephew in Ireland who is just starting school I got to see via photos on Facebook and video chat on Skype.

I’m thankful to the doctors and nurses who took great care of me when I was sick!  Getting diagnosed with Type 1 Diabetes was probably one of the most out-of-left-field things that has ever happened to me.  Being told “If you hadn’t come into the ER today, you would have most likely been dead in a couple of days” was, as you can imagine, extremely scary.  Having all the attending doctors and nurses simply say to me; “You’re here now, you’ll be fine” and “We’ll get you fixed up” and even “No, you’re not going to put all that weight you lost back on now” was all very reassuring.  And having my sister, brother-in-law and three nephews drive all the way down to come visit me while I was in hospital was a great treat.  

I’m thankful for my friends and co-workers!  To say the economy is tough is an understatement so I’m happy to have a great job and work for a great company.  The people I work with are amazing and I’m happy to walk in each day and be a part of it all.  Some of the people I work with now I’ve known for over sixteen years and their friendship over the years has been irreplaceable in my life.

I’m thankful to my fellow testers and colleagues in the software community!  It’s so great when you get a chance to meet with other people who share your interests and passion of work.  To be able to sit and talk with them, learn from them, share with them your experiences is a wonderful thing.  Thankfully, Twitter allows us all to stay in touch and keep up the sharing and learning until we can all meet up again.  You don’t realize how much 140 characters can do when you hit a wall and need to reach out to others for help!

I’m thankful for my neighbors!  These past few weeks recovering after Hurricane Sandy or Superstorm Sandy or “That one-eyed Bitch Sandy” has been extremely difficult.  Having others who are going through the same difficulties nearby to talk and share information with definitely helped.  It’s amazing when the small things we often take for granted become so welcome and needed when they’re taken away and then returned to you.  We shared phone numbers and contact information for FEMA, disaster relief information, trusted plumbers and electricians.  We stood on lines together to get Red Cross and City Harvest food donations so we didn’t have to worry about running around to find a hot meal. We literally jumped up and down the day that power was restored to our street and houses.  We now have hot water and heat, but some neighbors don’t.  “Come in and have a cup of tea,” we said to one of our neighbors who lives alone and lost the entire first floor of her home, “sit down and warm yourself up a bit.”  These small gestures can have a huge impact on people.  And you never know when you’ll be on the other end and need help yourself.

Finally, I’m thankful for my most wonderful partner, spouse and husband!  This past July, after being together for ten years, we were married in a very simple ceremony in New York.  This past year had a lot of ups and downs and I honestly don’t know how I could have gotten through it all without him.

STPCon Fall 2012 - Day Zero

This past week, I attended the Software Test Professional Conference in Miami, Florida; an amazing event with some truly talented people in the field of Software Testing.  Arriving Sunday afternoon from a cold and wet New York to a hot and sunny Ft Lauderdale, I drove down to Miami and checked into the Downtown Hilton.  After grabbing some snack foods to stock up my room from the Publix across the street, I spent the evening relaxing and reviewing the presentation I would be giving Tuesday afternoon.  Unfortunately, AMC was not one of the channels on my hotel television so the Season Three premiere of The Walking Dead would go unwatched, for now.

Bright and early Monday morning and what I call Day Zero began with an 8:00am breakfast.  Today was the pre-conference workshops and I attended Mark Tomlinson’s talk on New World Performance.  Mark and I had previously only met in the virtual world, on twitter and a TWiST podcast we did with Matt Heusser.  A seasoned tester and presenter, Mark’s workshop centered on exploring the balance of the new and the old technologies and techniques we as software testers use specifically with Performance Testing.  The details of the workshop were so dense that, to do it justice, I’ll be writing about it in a separate blog post.

One of the best things about the workshop was the attendees.  I was lucky to sit alongside and partner up with several testers and a developer from Sandia National Laboratories.  STPCon is an excellent place to meet and learn from testers with diverse backgrounds; Tina Newlander, Shavawn Bell and Lupe Massoth contributed great insight and questions to our conversations.  Our team re-wrote the Test Approach, using the information Mark reviewed and presented, to incorporate a more robust view on testing.

This document identifies <your company> attitude towards continuous performance, functional, and integration testing throughout the project lifecycle.  This test approach utilizes real-world and production level test data to allow for various development methodologies to be implemented and adapted across various product teams.

The conversation and ideas presented was so vast that Mark took an extra hour and half to finish his talk.  Despite the time, most of the attendees remained and gave Mark the attention and applause his talk deserved.

Throughout the day, the entire conference actually, the tweets were flying.  Using hastag #STPCon, I was able to follow along with other testers in other workshops as well as share what we were talking about.

On Monday evening, an impromptu dinner outing was planned by Matt Heusser which I was both happy and honored to attend.  The eight testers seated around the table must have easily represented well over 100 years of software testing experience. JeanAnn Harrison, Andy Tinkham, Todd Schultz, Seth Eliot, Karen Johnson, Matt Heusser, and Mike Lyles provided great conversation over a fantastic dinner at Gauchitos Steakhouse.  The testing talk continued as we made our way back to the Hilton’s lobby bar.  After a couple of Irish Coffee’s I excused myself and returned to my room.  I was presenting the next day, the first time I ever presented at a conference, so I wanted an early night and another chance to review my slides.

Before falling off to sleep, I of course watched the season three premiere of The Walking Dead that I downloaded on my iPad.
  ….to be continued

I'm speaking at STPCon in Miami next month

In less than a month, I'll be traveling down to Miami, Florida to attend and speak at STPCon!  My presentation, The Trouble with Troubleshooting, is geared toward testers who are looking to learn and enhance their investigative skills when it comes to Software Testing.
It's part of the hands-on Track, so this won't be a lecture at all, but an interactive, audience participation session.

 You can read all about it here and, if you haven't registered yet, you can do so here.

 One of the most valuable skill sets a software tester can possess is the ability to quickly track down the root cause of a bug. A tester may describe what the bug appears to be doing, but far too often it is not enough information for developers, causing a repetitive code/build/test/re-code/re-build/re-test cycle to ensue. In this interactive session, you will rotate between learning investigative techniques and applying them to actual examples. You will practice reproducing the bug, isolating it, and reporting it in a concise, clear manner. Brian introduces specific skills and methods to track down the source of bugs, and testers can then use this information to build an analytical framework and mindset around root cause analysis.

During this session, I will discuss several investigative methods and show how they can be used as a means of information gathering in determining the root cause of bugs. Common tests that are designed to eliminate false leads as well as tests that can be used to focus in on the suspect code will be shown and discussed in interactive examples.

Quality Is In The Eye Of The Beholder

Of the several definitions of quality found in your copy of Webster’s (or dictionary.com for those of us who haven’t opened an actual dictionary since high school), the one that I believe best fits the world of Software Testing is:

  an essential or distinctive characteristic, property, or attribute

When we test software, to assure its quality, we are ensuring that the software not only works but delivers some intended feature to the user.  For the most part, testers are able to verify their findings against an established spec or meeting notes or some form of outline that will tell them if the software passes and, as a result, delivers on the intended feature.  However, often the case is that the test findings will result in questions as to whether the actual software features and the intended features are a match.  Thus begins the questions; Is it a bug, an undocumented feature or a complete misunderstanding between the developers and the stakeholders?

Oh the joys of Software Testing!

At this point, it might be helpful to look at two additional definitions of quality;

character with respect to fineness, or grade of excellence

and.

high grade; superiority; excellence

Often, and for some ironically, Quality Assurance or Testers are not actually the ones who define quality.  We can create tests, execute them and deliver our findings.  We can voice our concerns and/or approval based on the findings, but defining the quality may not actually be in our jurisdiction.  This is because, based on the definitions I shared earlier, Quality itself is defined by the end user.

A working piece of software that has no value to a consumer really has little to no actual value or quality.  Working doesn’t automatically bestow quality in software.

 Beyond the Testing – Quality Assured

This is why software testers, in order to grow, need to go beyond the test results.  Question exactly what you are testing; who will use it, how will they use it, how often?  Ask for defined, realistic workflows that the users will perform to better define the tests. See if you can even talk to some users as well and hear feedback directly from them.

When a bug or issue is found, ask yourself; will the user also find this bug? Will it prevent them from using the software? Will it affect their perception on whether the software has quality?

This will allow you to better prioritize your bugs to the developers and the project lead, leading to better bug advocacy from the tester.  Concentrating on fixing and testing issues that the user needs rather than get bogged down in the smaller, edge case issues.

Summer’s End

I cannot believe the summer’s over. I also can’t believe that I’ve gone this long without a blog post. My fault; it’s been a crazy, sometimes lazy summer.  I’ve got about 5 or 6 half-written blogs that I need to finish and the publishing will start again, I promise.  I’ve also been hard at work on my presentation for STPCon next month in Miami.  Hopefully some of you will be in attendance!  Check out my presentation description here: http://www.stpcon.com/Session/41/The-Trouble-with-Troubleshooting

But let me catch you up on my summer and some possible reasons while my blog has been silent these past few months.

           I was conducting interviews for an open position on my team that we managed to fill with a great candidate who started a couple of weeks ago!

           I got married in July!
         
           Diet and Exercise has allowed to me to shed nearly 50 pounds!

          I got diagnosed with Type 1 Diabetes in August and spent 4 days in the hospital; a good chunk of that time in the ICU!

          I’ve begun an outline of a book I’m attempting to write on Software Testing!

         Some changes at work have brought more responsibilities onto my plate and I’m being challenged daily with exciting new things.

All of these things have provided me some great material for my Blog so I hope that Fall and Winter 2012 will make up for my quite Summer.  Hope to see you all at STPCon in Miami.

It’s a Bug!

As a tester, there’s probably not a phrase more often heard than; “It’s a Bug!” or “I’ve found a Bug!”  Usually, the phrase is delivered more emphatically by people who aren’t actually testers.  The Project Manager, Tech Writer, Customer, and of course Developers seem to take a small thrill in finding bugs in the software.  I’m not sure if it’s because they enjoy finding errors in other people’s work or simply think that bug finding is a game or puzzle and they’ve just won a prize.

 “I’ll take Software Errors for $400 Alex!”

But the problem I’ve often found with the quick-to-label “It’s a Bug!” is that once that bell is rung, it cannot be unheard; especially if an end user is within earshot.  Suddenly, the whole application’s stability is called into question and quite often, there’s no proof that there’s even an actual bug.

When is a Bug not a Bug?

The term bug carries a very large stigma in Software Engineering.  It’s used to describe everything from the wrong color of a button to a leak in memory causing massive system failure.  Imagine the same label being applied to the code that caused the Pentium to fail Long Division to the code that made a red button look pink.  But this is done all the time.  Those of us in the know will quickly add to the label words like “minor”, “critical”, “showstopper” and “negligible.”  However, the common user will still simply hear the words “It’s a Bug!”

Ben Simo (@QualityFrog) recently tweeted:

 “There is a big difference between your code ‘working’ and doing what it’s supposed to do”

I think this may be where a lot of the breakdown occurs; somewhere between what the developer codes and what the user expects.  Inside this gap, there be bugs….or at least what everyone calls bugs.  And in all fairness, they’re kind of right.

If Something Doesn’t Work The Way You Expect It, Is It A Bug?

There’s the main question; when is a bug really a bug?  And who gets to decide the criteria?  

Developers Aren’t Really Testing

In my fifteen plus years as a software tester, I’ve had the pleasure of working with some great and talented developers.  The large majority of them have sat down with me and gone over issues and code; searching for the root of the bugs that were preventing us from getting some crucial feature out the door.  There were, of course, some who simply deemed that sort of thing ‘a job for QA’ and wanted nothing to do with such mundane tasks.  “After all,” they would say, “it works fine on my machine.”

Those developers hold a special place in my heart.

Really.

It’s because of them and the attitude they had, I believe, that have made me into a better software tester.  I had to, on my own, figure things out and learn what the code was actually doing.  When I would approach them with my findings showing where their code was broken, you would think I had earned the right to be as condescending as they were to me.  But no, a war with a developer was not the best thing for my career or the project.  It was enough that I walked away with some new learned skill about software development.

Not to say that I haven’t been tempted one or two times.  Remember that scene with the Chocolate Pie in the movie, The Help?  I totally identified with Octavia Spencer’s character.

It was also good to know more about the developers.  Over time, I would become accustom to finding bugs quicker when I knew which developer coded which feature.  Programmer A, for example, always had issues with getting shell scripts to work properly while Programmer B would be notorious for using beta libraries in his code and therefore not deployable on production machines.

When it comes to actually testing the software, however, even the best developers aren’t really testing.

They’re checking.

Developers code software to perform specific tasks.  Once they’ve written the code, for the most part, they confirm that the code, on its own, performs said task.  They check it. Once they know that the code does what it’s supposed to, they move and code some other task.  Once they’ve completed all the tasks, they’ll check-in the code and somehow deliver the code to QA for testing; confident it does what it’s supposed to.

We (the software tester) may check the code as well, but we’re also testing it.  We make sure that it not only does what it’s supposed to do, but ensure that it doesn’t do anything else.  We test to see how well it does perform, how fast, how slow.  We expose the breaks, the errors that aren’t handled, and the paths the software was never supposed to take but somehow does.

We’re testing.

I think the best response to finding a bug I’ve ever gotten from a developer is, “How in the hell did you find this bug?”

Praise indeed

Has being a Software Tester made me a Skeptic (or has being a Skeptic made me a Software Tester?)

So I’ve recently upgraded to an iPhone 4s.

I have to say, for as much as a techie, gadget geek as I can be; I’m not one to run out to immediately buy the latest cool thing as soon as it comes off the assembly line. We’ve both had our 3G version of the iPhone about 2 ½ years and were due for an upgrade.  The battery life was starting to shorten and apps were all starting to slow down, so I knew that my phone was soon ready to venture off to that retirement farm where all smart phones go after a few years.  This would be the same farm where our family pets have all gone off to, usually when we were away at school or summer camp.  To make matters worse, my partner’s phone was no longer able to charge; no matter which cable or device we tried (Tested? Checked?), the phone would not charge.  As he was leaving that afternoon for an overnight trip to Boston for work, the need for a phone upgrade was now forced upon us.

So we took a trip up to the closest Apple Store to see if his original 3G phone can be fixed.  Plan B was the upgrade.  It was 2 o’clock or so and we were told that the next Genius Bar appointment available was 6:30 that evening.  Time was not on our side, so Plan B took effect; bring on the upgrade.  Two iPhone 4s phones, 32GB plus Apple Care (because you know one of us is going to drop his phone at some point in the next two years and break it) plus tax rang up to about $865.

Side note: I would love to see how it would work if, instead of a support desk and QA team at work, we had a Genius Bar where our developers would have to make appointments to get their builds tested.  I’m thinking that particular software model would be tossed out sometime during Day 1.

So cut to that evening; my partner made it safely to Boston and we were able to use the very cool FaceTime feature that night.  Both phones were working great.  I had backed up my old phone and restored all of my apps to my new phone.  I did spend almost two hours re-organizing them which was annoying, but overall the phone was faster and the battery was lasting much longer than my old phone (which was about to start its retirement on that farm I mentioned.)

Monday morning, on the way to work, it began.

No SIM Card was the message that greeted me when I thought I had just gotten a text message.  WTF?  48 hours in and already there was an error.  So I did what anyone in this day and age does when their new gadget starts flaking; I went to Google.

Oh yes, I do remember this from when the new phones first came out; a SIM issue on some phones.  For every forum or You Tube video post describing this issue, most dated as old as last October and November, you would have thought Apple would have gotten a handle on this already.  The new firmware, they said, should have fixed this.  I had the new firmware and still had the bug, so nope, nice try.  Also, there use of the word ‘should’ began to raise my skepticism as a tester; I’ve heard developers use this word far too often.  As in “This build should have fixed that bug” or “This should be the final release candidate.”

On Monday when this started happening, Power cycling seemed to work.  At first, I thought it was just a one off issue and, as I do with most bugs, I keep a close eye to see if/when it happens again.
I didn’t have to wait long.

That evening, it happened again.  I didn’t even notice at first as I was on WiFi and Words With Friends was still working.  Once I realized it, another power cycle got me back up and running.  No longer a one off issue, I figured I had a real problem with the phone.  But the late hour caused me to take an “I’ll deal with it tomorrow” attitude.

The next day I had no issues at all.  I had almost forgotten about it until the train ride home; Invalid SIM now appeared on my phone.

Once I got home after a train ride spent power-cycling my phone repeatedly, my first response was to go right back to Apple Store and demand first-in-line service at their alleged Genius Bar.  A phone call to the store, however, was what I ended up doing.  I quickly by-passed their automated phone system to get a real person on line.  I explained the issue and my frustration was quite clear.

Apple Support: “Well we have no available Genius Bar appointments this evening sir, but I can look to see if any other stores nearby are available.”

Brian: “I already did that.  The closest one is at the Fifth Ave. store back in the city.”

AS: “Oh.  Well if you come in, we can replace your phone with a new one”

B: “What’s to say that this issue, which should have already been fixed by Apple, won’t happen to this new phone?”
      Notice my skepticism and pointed use of the word should?

AS: “Well, I’m sure that this won’t happen again if we replace the phone.”

B: “Then why has it happened to over 40,000 people since last November?”

AS: “It should be fixed with the latest firmware. Do you have the latest firmware?”
     There was that word again; should.

B: “If I get a new phone, will I have to take two hours to arrange all the applications again?”

AS: “Did you back it up to the cloud?  If you backed it up, your apps should be restored and organized”
  Again, should.

B: “Yes, it was backed up the first time and I still had to re-arrange the applications.”

AS: “Well, it should have worked if you backed it up”
   Grrrrrr

While this phone conversation was going on, I was power cycling my iPhone and it finally came back with a valid SIM.  I carefully placed the iPhone down and ended my conversation with the Apple support person, telling her I would just go to the Apple Store on 14th Street in Manhattan when I went into work the next day.

Invalid SIM appeared to text me as I was on the train the next morning.  No matter, I thought, I’m going to the first available timeslot and getting this resolved; which I did.

Genius Bar spent all of 20 seconds with me before telling me to just replace the phone with a new one.  It was my only option as I didn’t even want to look at this phone anymore.  I felt bad for my old phone, enjoying its first days on that lovely retirement farm with all the other old phones.  I spent nearly two hours at the Apple Store, but I left with a working phone, so I think it was worth it.  I also figured out the REAL solution to Apple’s Invalid SIM / No SIM bug and it’s not just ‘Get a new phone.’

Step 1: Turn on and activate the new phone.

Step 2: Download and apply the latest Firmware upgrade (currently iOS 5.1)

If you have already downloaded the latest Firmware on the iPhone 4 or 4s and are still having the problem, start at Step 3. Make sure you have your phone backed up to Apple Cloud before doing Step 3, it will save you some headache later.

Step 3: Connect your phone to iTunes and Restore it to Factory Settings.
 Or
Step 3: Under Settings --> General --> Reset --> choose Erase All Content and Settings.

 This will set your phone back as if you just opened it up from the box except you’ll have the latest Firmware already installed.  The Firmware plus resetting to factory settings somehow fixes the carrier issues the SIM cards in the iPhone were having.

Step 4: Once the reset is done, chose “Restore iPhone from the Cloud” and login to your Apple account.  Choose your latest backup and your phone will be restored to how you left it; organized apps and all.

It should work.

Why Do We Test?

There have been a lot of recent talks and articles regarding the whole “Is Test/Testing Dead?”  Jon Bach gave a great keynote at STPCon in New Orleans last week regarding this.  “Testing isn’t dead,” he said, “but Bad Testing should probably die.”  Overall, testing as a practice, as a skill, seems to be on the decline as the actual activity is being done by just about anybody. 

But can just anybody test?

It depends.

I fall back on that response a lot, sorry, but it’s useful.  In this case, it depends on what you’re testing. 

A better question is can anybody test software?

No.  I personally don’t think so.  It takes someone with a good analytical mindset, technical knowledge, and an innate drive to find things out to be a really good software tester.  Again, that’s just my personal opinion based on my 15 years of software testing and working with a wide range of testers, developers, support engineers, and technologists.  Overall, I think some testers lose sight of the basic question, which I’ve conveniently put in the title of the post.

Why Do We Test?

A good interview question I’ve used in the past (along with my famous “How do you know when you’re done testing?”) I’ve heard many answers in response.

“To see if the software works”

This is a completely valid, logical response; also, the one that I hear the most.  Essentially, yes, we are confirming that the software works.  But isn’t there more to it?

“To prove the developers wrong”

I chuckled at this one especially because I’ve heard it on more than one occasion.  I’m not a fan of the adversarial tester versus developer mentality.  I try to squash it whenever I come across it in fact.  I like to think that we all have the common goal of delivering quality software to the client and should behave as such.

“Because it’s my job”

This one was an actual response from an interview I gave.  I had to quickly check to see if the candidate was in fact a two dimensional cardboard cutout.  If you’re just testing because you were told too, how well are you going to do your job? How inventive will you be in your testing? Going through the motions of testing is not time well spent at all.      

Ultimately, we test to learn something about the software that we don’t already know the answer to; to confirm or disprove something about the software that we think or had heard about. 

We test to learn.

The point of each and every test that we run on software should be to learn something about it.  Not just if it works, but how it works.  What happens when we expose the software to specific conditions?  How many users at one time will break it? Slow it down? Can I bypass the password protection? Will it run on my Mac just like it runs on Windows? Does the website look better on Internet Explorer than Google Chrome? Perform better on FireFox?

Endless amounts of questions can be answered by our testing.  And with each test, we learn something more about the software.  Sometimes, we learn things we weren’t even looking or testing for but we take the knowledge the test delivers to us and use it to create more tests.

At least we should be

Is Testing Dead? Are we doing our part to kill bad testing?

Ask yourself, for each test that you run, what have I learned from this test?  Did I learn anything new about the software?  Is this test telling me something that no other test is telling me?

If you have a suite of tests that tell you nothing about the software, ask yourself, Why am I running this test if I learn nothing about the software when I run it?

If you have questions about the software and you have no tests that give you the answers you’re looking for, ask yourself;

Why haven’t I written those tests yet? 

British Leap Bug

So I have to admit, I’ve become accustomed to my GPS.  In my car, while driving, the GPS has gone from being a distraction to a normal part of my dashboard display.  For the most part, I know where I’m going, but having it there to tell me about traffic jams and congestion, even beeping at me when I go over the speed limit are all a welcome array of features. 

So when I returned home from New Orleans and STPCon (which upcoming blog posts will talk about) I immediately noticed that the GPS map was way off and the display constantly read ‘waiting for a valid GPS signal.”

What could be the problem?  We immediately thought of and dismissed Sun Spot and Solar Flare activity.  As interesting as that might be, we figured that our cell phones and other devices would also be experiencing issues and they were all clear.  So we then narrowed it down to a recent firmware upgrade; since the upgrade, the GPS error appeared. 

I didn’t know about the upgrade (occurring while I was away at STPCon) otherwise I would have gone with that first rather than the Solar Flare hypothesis.

A quick search of TomTom’s webite revealed that we were not alone in our GPS signal issue.  From their forums:

We have identified the root cause of the issue which has caused some of our models to not get a GPS position. The issue is caused by a ‘leap year’ bug in our GPS firmware. We’re working on a fix and will provide a further update as soon as we know more.Since Saturday 31st March a limited number of models have been having an issue getting a GPS position. We’re looking into the cause and will post an update here as soon as we know more. If you click Notify me at the bottom of this FAQ, you will automatically receive an update.
We hear from some of our customers that a reset of the device temporarily resolves the issue

A ‘Leap Year” bug.  This is great.  The kind of bug that comes along only once every four years really.  Even better, because TomTom is a UK company, the Leap Year bug didn’t appear until the UK entered British Summer Time this past weekend, not when the US sprung ahead a few weeks ago.

Resetting the device didn't work for us by the way.

On testing this, though, I can identify with them.  Testing for Leap Year issues is always an interesting challenge. 

STP - The Arrival

An early Sunday afternoon flight is a great time to travel.  Arrived safe and sound in New Orleans at the Sheraton Hotel on Canal Street for the Software Test Professionals conference.  

While Day 1 is really Tuesday, tomorrow (Monday) is the pre-conference workshops.  I’m looking forward to all of them and hope I won’t be too overloaded before the actual conference begins. 

A big plus; the view from my room is fantastic!

STP Bound !

All next week, I'll be in the gorgeous city of New Orleans attending the Spring Conference of Software Test Professionals.  Be on the look out for some live blogging and daily summaries! 

Testing, one, two, three…..

So these past few weeks have been more about the management side of my job, rather than the testing side; I was starving for some great testing topic to blog about.  Well, who would have thought that a great testing debate would literally come to my office door?    

A colleague of mine, a developer on another team whom I’ve worked with before, came by my office and asked for five minutes of my time to “Settle a testing debate between himself and another developer on his team.”

A break from the monotony of management?  A chance to engage in a discussion where the outcome was brining two developers into the warm, bright light of testing?  

Try and stop me. 

So the debate was, essentially this:

The first developer thought that the test suite they had should be re-organized so that the tests run in a specific order; establishing dependencies on the outcome of each test and then only running the next test should the previous one pass. 

The second developer thought that the work involved in doing so was way too much and that the first developer was “crazy” 

His words, not mine.

So they asked me, an impartial (read ‘not on their team’) observer and ‘testing expert.’ 

Again, their words, not mine. 

But I totally ate up the compliment and offered my voice to their debate.  I told them:

“You’re both right! And wrong!”

At this point, they were re-thinking brining me in on this, I’m sure.  So I explained what I meant.

Organizing your tests is a really good idea.  It will help to keep track of your coverage and identify areas, when the tests fail (or don’t met the expected results) to zero in on.  However, I told them, you’re crazy to make all the tests dependent on the ones before. 

Developer A, the test organizer proposer, had offered an example of what he was trying to do. 

“Say you have a function that squares two numbers and a test associated with it.  You’re not going to run that test if the test for the multiplier function, which the square function is using, fails.  So the tests should be ordered and linked somehow so we don’t waste time fixing a square function when the real issue is the multiplier function is broken!”

I was beginning to see that maybe we were looking at this the wrong way.  We should, I said, look at this not from the tests, but from the functions point of view.  What you want to do is not organize the tests, per se, but rather organize what they’re testing; group them by the functions being tested.  Since these tests were essentially core unit tests, this was probably the best way of doing it.

I then told them my planned idea for my own team and the couple of hundred tests case that I was looking at.  I also had to pull in my Sci-Fi geek cred at this point.  Ok, we’re a tester and two developers debating test methodology on a Friday afternoon; there was bound to be some geek gauntlets being thrown.

My idea, which I thought would work for them, was to identify the core functions of their system and the tests associated with them.  Call these your “Level One” tests, which should all pass prior to running your Level Two tests and so on. 

So when you test, it’s like you’re running a Level One Diagnostic.

“Nice. Star Trek”

Like I said, there was bound to be a Geek Out at any time here.

So test organization I really not about the test, but what you’re testing.  If you order your tests to verify the core, crucial pieces first, then move on to secondary, tertiary, etc. then you can ensure that the basics are solid before testing the complex.  This will help the developers as well to identify what was changed/broken/alerted in the last build.

Developer B, by the way, still thought it was a crazy amount of work to do.

If A Bug Is Found In The Woods, Will Anyone Hear It Scream?

Bug Tracking has been a common discussion at work recently.  Like many software companies, we've adopted the use of Atlassian’s Jira.  It’s flexible enough for all the diverse product teams to be able to use it, and intuitive enough where very little education is needed to start using it.  The hardest part, it seems, has been to get the teams to ACTUALLY USE IT.

 As a tester, and more importantly in this discussion, as the QA Manager, I want to enforce a policy of “Jira All The Things!”  (See what I did there?  I’m not above shameless plugs for the blog.) 

But we’re using Jra as a means to not only track bugs found in testing, but also:

-          Bugs Found in Production

-          New Features requests from our Stakeholders

-          Client Questions on the Product Suite and the subsequent responses sent to them

-          Updates to existing Features

-          Changes to the database configurations; in both system and client

So now the question from the team comes back;

“What really needs to be put into Jira and what can we just stick to endless e-mail threads on?”

OK, they don’t actually say it like that, but those are really the two options.  Endless e-mail discussion threads with no workflow involved and, subsequently, no obvious resolution or just put it in the Jira backlog.

I might have come across as biased there.

While I feel the need to track all the features, new and updated, in some efficient way, I’m nonplussed on the method of how we track.  But I’m holding the line at bugs.

Put all the bugs in Jira. 

Please.

Track them, follow them, know when they creep up, sneak back into the hidden recesses of the code and once again appear like some coded perennial flowers.

What’s in it for me, you ask? 

I've jokingly said, too many times according to some, that we unfortunately don’t get paid by the bug.  If we did, I'd Jira me up a new car and summer home on each build.  But I’ve found that bugs in the system sometimes live longer than the testers that stay on the projects.  Leave a legacy to those who follow you; live on in your bug reports.

Open a Jira.

How Long Will It Take You To Test…? – Part 2

“Do you have enough time to test?” 

Every time I hear this question, I can almost hear Admiral Ackbar yelling “It’s a Trap!”

As if we’re going to get more time if we need it.  Or, if the release date does actually slip to allow more testing time, inevitably scope creep comes into play and we’re back where we started; too many things to test and not enough time to test them.

“It depends” comes back into play as the “go to” response to the question.

“It depends, how much testing needs to get done?” 

This is a great opener to start the conversation on testing breadth versus depth.  Some managers or developer leads might start to get glassy-eyed as you ramble on about being able to execute dozens of test cases to ensure that one API is bullet proof but, in the time it takes to execute them, no time if left to test the 20 other APIs.  The other option is to take the same amount of time, and execute one test case against each of the 20 APIs.  This ensures that, in that one case, they all work as expected, but leaves a large risk as to how resilient the code actually is. 

Can you say that the code has been tested?

Sure!

But tested well?

Not really.

So, if you had more time, would you test more? 

I would hope so.

“It depends, is there something that doesn’t need to be tested so I can spend more time testing something else?”

Now, it’s the manager’s turn to hear Ackbar’s famous line from Return of the Jedi.  You’re asking for a pass; absolution if you will, on passing over chunks of code and functionality in favor of giving your time to some other features more deserving of your attention.  If a bug gets past you and it’s found in this “Manager Approved Pass Over Area” of the code, you can at least say “You told me not to test that.” 

Personally, I hate letting any bug get past testing and into Production, but it happens.  I’m sure to cover that topic in future postings. 

But overall, the conversation of “What are the risky areas in this build?” is one that all testers should have with their development counterparts.  It’s a conversation that I myself have almost daily. Seriously, just today, I asked this question to three different developers.  I got three different responses and cried a little inside, but the conversation was good!

Overall, letting the team know what was tested, how much testing was done (depth versus breadth),  and what was not tested or not tested as thoroughly due to time constraints, is highly recommended.  

How long will it take you to test….? - Part 1

Test Estimation can be taken as much of an art as it is a science.  In my younger, more wise-ass days (which some might say were, literally, just yesterday) I might have replied to my manager or the Dev lead; “How long will you give me?”

But I’d like to think that I’ve matured some.  That and the rolling eyes that I’ve received when I’ve given that response more than fifty times in the near past has caused me to come up with a way to give a better response.

But, honestly, what is a good response?

“It depends” is always a good one.  Of course, that response has met its fair share of rolling eyeballs as well.    

If you’re running a pre-arranged suite of tests and know that it takes about an hour to run them, you can say “About an hour.”  But what happens when your manager returns to your desk an hour later and you have no test results?  What happened?

Well, there’s always more to it than just running the test suite, right?  What makes a true estimate?  You’re going to need to make sure that you have a build first.  And that the build works and runs without crashing on startup.    And that the test suite that should take an hour to run, actually runs to completion.  You may have to run it a couple of times; restart it, make corrections, clean log files, etc.  And then, once it’s complete, you’re going to need a bit of time to look over the results.  Then, of course, you haven’t even begun to write, execute and add any new test cases and scripts to your suite to test anything that’s in this new build. 

So how long is that?

Stay away from saying “It will take as long as it takes.”  Your manager might roll his eyes as he or she walks away and finds a new tester while you’re left surfing Monster or CareerBuilder quickly before they cut off your Internet access. 

Your best bet might be to stay away from absolute times.  Rather, bring your Manager or Dev Lead in on the conversation.  Tell them that, once you’ve gotten a working build, you’ll run your suite and go over the results.  While you’re waiting on that build, you’re writing some new tests and will run them after the suite completes.  Give updates along the way.  Let them know when you’re hitting walls and road blocks.  Resetting the clock on testing time should be part of the conversation.  

Ironic Titles

 Combining a little tongue-in-cheek irony with a popular meme provided the inspiration for this blog site. Primary, I'll be using this blog to focus on software testing and all that may encompass.  Of course, for anyone who actually does Software Testing, you can't actually ever Test All The Things

 Hence, the irony.

However, that hasn't prevented many a manager, developer, stakeholder or all of the above to make the request of the tester involved. 

So welcome aboard for the ride!  I'm aiming to try and blog about twice a week.  Maybe I'll be able to have guest posters as well.