Inattentional Blindness

Jeff Fly made a great article in AST newletter about Inattentional Blindness phenomenon (the phenomenon of just plain not seeing something obvious because our attention is focused on something else) and explained what does it mean for us as testers. That  reminds me of ways to mitigate blindness effects in testing. Here pair testing, pair programming, perspective variane and James Bach’s “Heuristic Test Strategy Model” comes into play.

Summary of Software Characteristics

In this post, I would like to summarize typical software characteristics that a Quality Control Leader needs to understand in order to recognize typical risks, develop appropriate testing strategies and specify effective test cases. In general, I consider this is the backbone of Quality Control Engineering because in the end of the day, no matter whatever test design techniques testers use in whatever test levels, no matter whatever tools they use in whatever test types, the ultimate purpose is to estimate/control/monitor the following software quality characteristics.

Please note that when talking about software characteristics, we divide them into two categories, one for Functional Attributes and one for Technical Attributes.

Functional Attributes

Functional Accuracy:

• Objective:  test based on specified or implied functional requirements to evaluate whether the system gives the right answer and produce the right effects. Accuracy also refers to the right degree of precision in the results (Ex: computational accuracy)
• Techniques: All black-box test design techniques can be used

Functional Suitability:

• Objective: to evaluate whether the system solves the given problem and appropriate to intended tasks.
• Techniques: Use case, exploratory testing.

Functional Interoperability:

• Objective: to evaluate whether the system functions correctly in all intended environments. Environment includes not only elements that the system must interoperate with but also those that interoperate indirectly with or even simply cohabitate with. Cohabitation implies sharing computer resources (CPU, memory…) but do not work together.
• Techniques:
  - Equivalence Partitioning: to determine environment set when you know possible interactions between one or more environments and one or more functions.
  - Pairwise and classification tree: to determine environment set when you’re not sure about interactions and want to generate more arbitrary configurations.
  - Use case testing in each configuration

Functional Security:

• Objective: to evaluate the ability of the software to prevent unauthorized access
• Techniques: Attack and Defect Taxonomies

Accessibility:

• Objective: to evaluate the ability of how to use the system under particular requirements, restrictions or disabilities. These are often arisen from national standards, industry compliance by law or by contract.
• Techniques: Specification, requirement-based testing used in risk-based testing approach. Since it obligated strictly by law, it usually not sufficient to test just a few representative fields or functions but every field and function might be required.

Usability:

• Objective: to evaluate whether the users are effective, efficient and satisfied with the software
• Techniques:
  - Inspection, evaluation and review
  - Use-case testing along with syntax and semantic tests
  - Survey or questionnaire

Technical Attributes

Technical Security:

• Objective:
  - Technical Security is different from Functional Security is that Technical Security leverages technical knowledge and experience to take advantage of unintended side effects and bad assumptions to subvert or attack the software.
  - Here, we try to evaluate software security vulnerabilities related to data access, functional privileges, the ability to insert malicious programs into the system, the ability to sniff or capture secret information, the ability to break encrypted traffic and the ability to deliver virus or worms.
• Techniques and tools:
  - Information retrieval
  - Vulnerability Scanning tools
  - Security attacks techniques (Dependency attacks, user interface attacks)

Reliability:

• Objective: monitor software maturity and compare it to desired, statistically valid goals. Reliability is important for high-usage, safety-critical systems. Special types of Reliability tests are Robustness and Recoverability.
• Techniques:
  - Select an appropriate mathematical model from Reliability Growth Models or Software Reliability Growth Models to monitor software increase or decrease in reliability.
  - TAAF (Test, Analyze and Fix). Because the “around-the-clock” nature of testing process, reliability testing is almost always automated. It uses empirical test data performed in a simulated real-life operational environment.
  - Recoverability testing (Ex: failover test, disaster recovery, backup/restore): to evaluate system’s ability to recover from some hardware or software failure in its environment.

Efficiency:

• Objective: to evaluate whether system has good time response and resource usage or not.
• Techniques:
  - Review, static analysis before and during design, implementation phase
  - Performance testing: to evaluate time response within specified period of time and under various legal conditions.
  - Load testing: to see how system handles under different level of loads, usually focused on realistic or anticipated loads.
  - Stress testing: put the load to the extreme and beyond to determine system limit and observe its degradation behavior at or above maximum load.
  - Scalability testing: take stress testing further by finding the bottlenecks and then estimate the ability of the system to be enhanced to resolve the problem.

Maintainability:

• Objective: to evaluate the ability to update, modify, reuse and test the system.
• Techniques:
  - Static analysis and reviews (maintainability defects are usually found with code analysis tools, design and code walk-through)
  - Test updates, patches, upgrades and migration
  - Collect project and productions metrics (Ex: number of regression test failures, long closure periods of bugs, duration of test cycle) to determine analyzability, stability and testability of the system.

Portability:

• Objective: to evaluate the ability of the system to install to, use in and move to various environments.
• Techniques:
  - Equivalence Partitioning, Pairwise, Classification Tree, Decisions table, State Transition
  - Installability testing: install software using its standard installation, update, patch facilities on its target environments. The purpose is to check installation instructions, user’s manual and observe software’s failures during installation/uninstallation.
  - Coexistence testing: to check whether one or more systems that work in the same environment do so without conflict.
  - Replaceability testing: to check that whether we can exchange our software components for other 3rd party ones or not.
  - Adaptability testing: execute test cases to evaluate Functional Interoperability. The techniques are the same with those in Functional Interoperablity section above.

Usability Testing

As Test Analyst, one will design test cases concerning about following quality attributes such as: Accuracy, Suitability, Interoperability, Functional Security, Accessibility and Usability. Here, I would like take some notes about Usability Testing as it’s very common but still abstract for us to practically apply it. The challenge is while usability is common-sense and straight-forward for all testers (and perhaps all people who are using computers), what is the technique to design test cases for it? Is there any guideline or instructions so that we can effective measure the usability of our system?

Definition:
You want to test and measure the capability of your system to be understood, learned, operated and attractive to users when used under specified conditions. At the end of usability testing, you want to make sure that at least your target users are effective, efficient and satisfied with the software.

Understandability:

• The simplicity or difficulty of figuring out what the software does and why you might need to do it. Could you shortly read the manual or just look at the GUI and understand what the software’s supposed to do and why you want to spend time using it?

Learnability:

• The simplicity or difficulty of figuring out how to make the software do what it does. Is it intuitive, easy for you to learn how to use the software? Are there any hints, underlying metaphors which naturally guide to explore the software features?

Operability:

• The simplicity or difficulty in completing your tasks using the software’s feature set. Is it easy, effective and efficient for you to have your tasks done by provided features?

Attractiveness:

• Which is the extent to which the software is visibly pleasing, friendly and inviting to the user?

What you need?

• Knowledge about your target users: their background, working environment, computer literacy…
• Black-box Testing techniques
• Knowledge of sociology, psychology, culture, language and national standards (in plus but very helpful)

What are the techniques?

1. Inspection, Evalution and Review: consider system specifications and designs under usability point of view. Actual users could be invited to give comments on mock-ups or demo version.
2. Usability test scenarios: measure usability attributes. Testers receive a document including instructions, test procedures, guideline to take notes and log results or checklists. In the test script, there are also questions or checklists which ask users to evaluate the interface, the meaningfulness of messages and outputs.
3. Survey or questionnaire: gather observations of users when they interact with the system. There are Software Usability Measurement Inventory (SUMI) and Website Analysis and Measurement Inventory (WAMMI) which provided standardized surveys, questionnaire for usability testing. Unfortunately, these are commercial services so you might want to refer to some of free, sample questionnaire available such as Example to screening questionnaire, Sample SUMI questionaire, Sample WAMMI questionaire… to design a survey for your own needs. Unless there is strict regulation to use standard surveys, I think one has little difficulty in creating usability questionnaire for his/her system. All we need to do is just keep in mind the objective of usability testing and its quality attributes.

Read more:

• Usability Testing Checklist
• [Ebook] Handbook of Usability Testing – download at Vietnamese Testing Board

Classification Trees – Short Notes

I think we could call this black-box technique an advanced one developed from pairwise testing. Unlike pairwise which just offers us to test pair of all factors, we can use classification tree method to increase coverage by testing triples, quadruples… of some factors. In short, this technique and tool allows us to design test cases more flexibly if based on our risk scale, we want to test some factors more heavily than others.

Supported tool:
- CTE XL – Download here (you need to fill in some of your name, email, get license by link and email before using it)

Example:
A a web application based on 3 factors as below:
• The first factor: connection speed (2 options: dial-up and broadband)
• The second factor: operating system (2 options: Windows XP, and Windows Vista.)
• The third factor: browser (2 options: Firefox, Internet Explorer)

a) Design test cases for all pairs possible
b) Design test cases for all pairs of OS and Browser only
c) Design test cases for all triples possible

Results:

Note: For scenario a, what we got is exactly when we use allpairs tool. Scenario b, it’s because we just focus on pair of OS and Browser only, all pairs possible of these 2 factors are ensured while other factor (Speed) is ignored. Scenario c, using threewise method to generate all triples possible, we ensure all combination of every 3 factors existed in our test suit. Here, because we only have 3 factors with 2 options each, number of test cases is the same when we use completed combination which is calculated by 2³= 8.

Pairwise Testing

Reading through Advanced Software Testing (for Test Analyst) today makes me understand more about pairwise testing, its concepts and how it’s designed both manually or automatically (by tools). Since this is very useful technique which every tester should know at least the basic or the insights, I think it’s worth to make a short notes briefing some important aspects of this test design technique. You can find a lot of information about it overnet but the concepts, examples and expressions are difficult to catch up. Here, I try to explain in a short, simple way to make it easiest for you to follow. Just a little practice and you’re done.

Pairwise Testing

1. Definition:
• A black-box test design technique
• Design test cases to execute all possible combinations of each pair of input parameters.

2. Benefit:
• Prevent combinatorial explosion
• Generate test cases that are much more effective than random selection or guessing

3. Application:
• Frequently used in compatibility and configuration testing

4. Manual process (illustrated by example)
Say, we want to do compatibility testing of a web application based on 3 factors as below:

• The first factor: connection speed (2 options: dial-up and broadband)
• The second factor: operating system (2 options: Windows XP, and Windows Vista.)
• The third factor: browser (2 options: Firefox, Internet Explorer)

First, construct a table called an orthogonal array:

Number of rows = the product of two largest numbers of options. We have all factors have 2 number of options, so number of rows = 2×2 = 4

Second, map the factor options with numbers, replace each instance of 0 with the first option, 1 with the second option and so forth.

We try with pair Speed and OS first, fill in the table possible combinations of pair Speed and OS, we have:

Then, for the pair Speed and Browser, do the same thing: fill in all their possible combinations, we have:

Now, check the pair OS and Browser, obviously we see that there’re duplication in combination of those 2 factors, we have combination 00 in both row 1 and 3, combination 11 in both row 2 and 4.

Thus, we have to edit combination for pair OS and Browser a little bit so that this pair has all possible combination. Try to edit row 3 and 4 and the result is:

Finally, check if we miss any combination for each pair?
• For the first pair of factors, Speed and OS, we have 00, 01, 10, 11, in rows 1, 2, 3, and 4.
• For the second pair of factors, Speed and Browser, we have 00, 01, 11, and 10, in rows 1, 2, 3, and 4. All four pairs, just occurring in a different row.
• For the third pair of factors, OS and Browser, we have 00, 11, 01, and 10, in rows 1, 2, 3, and 4. All four pairs, again, just occurring in a different row than the previous pairs of factors.
Yeah, so you have test cases that execute all possible combinations of each pair of input parameters. That’s the pairwise testing. The final step is just to remap the number to option name and you have the final result of nice test cases below:

5. Automatic process:
Download allpairs tool, read its manual and use it to generate test cases for you. If you learn to run it properly, the result is the same.

6. Further practice:
You may want to practice more because at the end of the day, we never have such simple situations like above example. Let’s roll your sleeves, try with the following exercise:

Design test cases using pairwise technique for a configuration testing based on 4 factors as below:
• The first factor is connection speed. We have two options: dial-up and broadband.
• The second factor is the operating system. We have four options: Mac, Linux, Windows XP, and Windows Vista.
• The third factor is the security settings. We have four options: native operating system security only, Symantec’s product, Trend Micro’s product, and McAfee’s product.
• The fourth factor is the browser. We have three options: Firefox, Internet Explorer, and Opera.

When you’re done, check with the following table to compare your answer. There is only 1 special thing in this exercise is that when you have unequal number of options in every factors, just give every factors number to the maximum. For example, in our case, factor Speed just has 2 options but we have numbers for it 0, 1, 2, 3. Later when you map those numbers with real option name, place ~ (means whatever option) to exceeding numbers.

References:
1. Advanced Software Testing (Test Analyst) – Rocky Nook
2. http://www.satisfice.com/tools.shtml

If you have any question or comments, feel free to drop me a line here or tweet me @khuongtw

Thanks for your reading.

CaseMaker

A test case design and test data generation tool – which supports the black box techniques.

CaseMaker offers for free all the ISTQB Certified Tester a full license for a year. Using CaseMaker for free you can set the learned techniques in your projects and this can be also a motivation for the managers and also for the tester to make the certification.

Advantages:

1. Orienting to apply basic black box techniques techniques such as: Equivalence Partitioning, Boundary Check, Decision Tables and Pairwise in the real project
2. Helping to design the test case systematically by dividing into small components or input business rule logic
3. Creating a pretty big amount of Test Data

Disadvantages:

1. Does not cover White Box and Some of Ad-hoc testing.
2. Unfriendly UI
3. Pretty heavy software
4. Limited range of target users (users must have a back ground of Software Testing in order to use this tool effectively.)

A simple tool of Time Management

• quadrant I — manage: the quadrant of necessity; things are both urgent and important
• quadrant II — leadership and quality: the quadrant of focus; things are important but not urgent
• quadrant III— (AVOID): the quadrant of deception; things are urgent but not important
• quadrant IV—(AVOID): the quadrant of waste; things are neither important nor urgent

A No uttered from the deepest conviction is better than a Yes merely uttered to please, or worse, to avoid trouble. (Mahatma Gandhi)

Learning 2.0 with Web 2.0

I couldn’t believe learning could be (in fact has been and will be) so much fun, convenient and enjoyable like this with Web 2.0

Creative Web 2.0 Learning

View more presentations from Judy O’connell.

Motivation

motivationfactors

What is motivation? Yep, it’s just the simple word and is so straightforward that everybody can say right away its meaning. Say, motivation is encouragement, is just what motivates, right? Yet, if we think more about it, we can see how important motivation is to our life.

“Motivation is which makes us take action towards our desired goals and which gives purpose and direction to those actions.” Thus, what motivates a poor student to get high scores? What encourages an employee to work? What inspires an artist to be creative? What urges an entrepreneur to develop and run their business? No matter who they are, successful people share a common thing that they all have strong passion, strong motivation that like a power, long lasting, unlimited-energy engine to boost them overcome any challenges and advanced many others. If one wants to be successful, they must always keep motivated. People see someone got top graduation, award; someone got rich from limited availability and wonder how did they do that? Don’t learn from their tricks, luckiness, their available advantages because you’re different, situation changes overtime and it may not happen to you. Instead, figure out what are their motive forces and try to make your own ones. The underline factor is just motivation which includes passion, inspiration, love, competition, responsibility… anything which can naturally push you to get started and go ahead.

Every now and then, we lose our motivation and we allow procrastination to dominate our life. Without motivation, we stop taking action, lose focus, wander around, circle our thumbs… We feel we’re in the middle of nowhere, useless, dumb, inane, stupid, ugly… all the worst thing we can think and feel about. Just along with motivation, we feel that we have to do something, learn something to achieve our goal and enjoy our lives with happy working, studying and entertaining. We start to try, achieve things that we really want to and believe that we are capable of becoming whoever we want to be.

So just brainstorm and list out anything which can motivate you. Favorite songs, compliments, intimate friends, caring, meditation, sight-seeing, competitors, lovers, parents, self-esteem, ego, ambition, money, comfortable working environment, kind and talented coworkers, classmates, movies, successful stories… or just some motivation quotes sometime for a while. Every successful people have motivation quotes to whisper to themselves and share with others. Pick some for your favorites, the ones that you contemplate the philosophy and stir up your desire to get moving and take action.

The secret of getting ahead is getting started. (Mark Twain)

The best motivation is self-motivation. The guy says, “I wish someone would come by and turn me on.” What if they don’t show up? You’ve got to have a better plan for your life. (Jim Rohn)

People become really quite remarkable when they start thinking that they can do things. When they believe in themselves they have the first secret of success. (Norman Vincent Peale)