A Testing Mindset

Because Copilot can't do it all for you... yet.

In the previous installment, we examined testing from a high level and discussed:

• The goal of testing: Find as many flaws as efficiently as possible

• Improving efficiency with equivalence classes and front-loading your tests

• Some theoretical testing categories: static vs. dynamic and black box vs white box

Testing Theory

In this installment, we'll zoom in on how to generate specific test cases. This will be checklist-heavy. These can be great references, but remember that different projects require different approaches. The long-term goal is for you to build a testing mindset so this comes naturally to you.

So let's dig in.

Dall-E: A software bug in a partly-filled-out page of a coloring book

Input Values

Your tests will often focus on controlled inputs. These may be the parameters of a function, the query strings or body of an API call, or the fields in a UI form. When presented with a controlled input, I'd suggest you draw test cases from three categories:

1. Happy path: The expected use cases

2. Boundary values: The minimum and maximum values allowed

3. Corner cases: Unusual conditions and error states

Consider a login page that takes an email in an input element:

This is a picture, don’t enter your email

For the happy path, the main uses cases from our spec might be:

• An email for an account that authenticates using a password

• An email for an account that authenticates using SSO (Single Sign On)

• An email not associated with an account

For boundary values, zero characters are the minimum and the backend determines the maximum. Let's assume you are using a SQL database and the email column is VARCHAR(50). (An insufficient length, but your backend author didn't know that). So the boundary cases would be:

• Empty (no text provided)

• The email address is 50 characters long

• The email address is 51 characters long

Corner cases are where things get creative. Here are a few possibilities:

• Emails that are valid but strange or invalid but close

• A valid email, followed by more text

• An attempted SQL injection attack

Aside from your day job, it is worth noting that many interviewers love watching how you handle corner cases. Being able to generate them on the fly can really improve your coding interviews.

Example cases for common data types

Numbers

• -1, 0, 1

• Smallest allowed number, and one less than that

• Largest allowed number, and one more than that

• Smallest and largest possible numbers (eg. int.MinValue, int.MaxValue)

• Non-integer value

Strings

• Null

• Empty

• Single character

• Smallest/Largest allowed strings

  • If the function doesn't have a defined largest, try one 10x larger than what use cases call for.

• Repeated characters

• Each character is unique

• Symbols

• Whitespace

• Unicode

• If validated to a format (email, zip code, etc) consider valid but strange and invalid but close

Collections

• Null

• Empty

• Single element

• Two elements

• A large number of elements

• Duplicate elements

• Sorted vs Not Sorted

• Very skewed (for trees)

• Cycles (for graphs and linked lists)

• Disjoint sets (for graphs)

Input Combinations

It is also important to ask how the inputs relate to each other. Are they independent? Do specific combinations matter?

Consider this REST API endpoint:

PUT /users/{id}/profile
{
  "email": "string",
  "firstName": "string",
  "lastName": "string"
}

The three values in the request body may be completely independent. If so, you can test each one with variations while you provide a happy path value in the two others.

But what if the system concatenates the two name fields? I’ve seen this in systems where legacy components interact with first/last separately, but newer components treat FullName as a single value. In this system we have to consider the interactions between the two.

Code paths

If you have access to the source code, you should also examine the paths through the code. You can draw test cases that ensure that you exercise all the code paths.

Consider this Java function:

public static int calculateGrade(int score) {
    if (score < 0 || score > 100) {
        throw new IllegalArgumentException("Score must be between 0 and 100");
    }
    int grade = (score - 50) / 10;
    return Math.min(grade, 4);
}

There are four basic paths, so we make four cases

1. Score = -1, to enter the if statement by the first condition

2. Score = 101, to enter the if statement by the second condition

3. Score = 75, to skip the if statement, then Math.min returns its first parameter

4. Score = 100, to skip the if statement, then Math.min returns its second parameter

But what if we wrote the code like this?

public static int calculateGrade(int score) {
    if (score < 0 || score > 100) {
        throw new IllegalArgumentException("Score must be between 0 and 100");
    } else if (score >= 90) {
        return 4;
    } else if (score >= 80) {
        return 3;
    } else if (score >= 70) {
        return 2;
    } else if (score >= 60) {
        return 1;
    } else {
        return 0;
    }
}

This version has seven paths, so we need seven cases

1. Score = -1, to enter the first block by the first condition

2. Score = 101, to enter the first block by the second condition

3. Score = 95, to enter the second block

4. Score = 85, to enter the third block

5. Score = 75, to enter the fourth block

6. Score = 65, to enter the fifth block

7. Score = 55, to enter the sixth block

What if you used the test cases for one version with the code for the other version? Either you would leave some paths untested (missing any bugs in that path), or you would test the same path multiple times (potentially inefficient).

An important caveat with code paths (or code coverage metrics) is that testing a path doesn't guarantee its correctness.

What if the above code had a typo?

} else if (score >= 91) { // <= Oops
  return 4;
}

None of our listed test cases would fail, missing the bug that a score of "90" returned the wrong grade. Boundary tests at each of the grade boundaries would catch it, though. Mixing and matching these approaches will give you the best results.

Functional Testing

The above approaches are forms of Functional Testing, which ask the question:

When the user inputs specific data or takes specific actions, does the system function as expected?

Besides these primary techniques, we should consider other factors:

• State: Does the action depend on the internal state? Account settings, session state, data pulled from a database, etc.

• Environment: Will the behavior change for different devices? Browsers? Operating systems? Environment settings?

• Does the same set of user actions always return the same result?

When I was testing desktop software at Microsoft, we used a simple technique I referred to as "testing furiously". It goes like this:

1. Find the button you wanted to test (usually a submit or toggleable state).

2. Hover the mouse over that button.

3. Now click it as many times as you can, as fast as you can. Really jam on that thing.

It sounds silly. And it was! But we would find race condition bugs that way. Sometimes our product managers would frown and say that those weren't bugs that actual customers were going to hit. But goes who accidentally double-clicks buttons all the time? Customers.

Specialties

Beyond functional testing, we might focus on a particular aspect of the system. We'll describe some possibilities here, but only very briefly. Each section could become its own topic for a future week.

Accessibility

Is the system usable by people with disabilities, either directly or by working with assistive technologies?

Is your product fully functional for customers:

• With visual impairments using screen readers?

• With low vision using screen magnifiers?

• With color blindness? (pro-tip: there are different types of color blindness)

• With dexterity impairments? Can they navigate your product using only the keyboard?

Internationalization

Is the system usable by people in different parts of the world and from different cultural contexts?

There are a few numeronyms used for variations of this theme.

• i18n: Internationalization

• l10n: Localization

• g11n: Globalization

The human experience is vast, but here are some possibilities:

• Different languages. Focus on your target audience first. If you support many languages, pick a varied subset such as:

  • German (long compound words)

  • Japanese (non-Latin characters, encoded in two bytes instead of one)

  • Arabic (non-Latin characters, written from right to left)

  • Pseudolocalization, a fake English-readable language using non-standard characters

    • Example: "Account Settings" shown as "!!! Àççôûñţ Šéţţîñĝš !!!"

• Formatting: number, telephone, address, currency, and date

• Time Zones

  • These are the absolute worst

  • No seriously, supporting time zones is really painful

• Maps

Performance and Scalability

Performance: Does the system complete work with acceptable accuracy, efficiency, and speed?

Scalability: Does adding additional resources lead to a proportional increase in performance?

Different test types to consider:

• Load: Expected usage conditions

• Stress: Extreme load conditions

• Endurance: Continuous expected load over a period of time

• Volume: Significant increase (as might be expected over time)

Security

Is the system's data and functionality safe from malicious users?

If I get started with this one we’ll be here all day. For now, I’ll just say that we should consider these aspects of our systems:

• Authorization: A user can only access the things meant for them

• Availability: A user cannot do anything that affects the availability of data or services for another user

• Non-Repudiation: A user cannot do something and then plausibly claim that they didn’t.

From practice to more practice

The previous installment was theory, this installment was practice, and now there is nothing to do but, well, practice more. Testing is a muscle that gets built over time.

People tend to focus their testing on what has bitten them in the past. That engineer who always asks what happens if the values are null? They have seen site outages because of a missed null reference.

Try to learn from their experiences. Ask them for their stories. As the quote goes: “Better be wise by the misfortunes of others than by your own”

Do you have any fun stories about testing or interesting bugs that we might learn from? Let us know in a comment!