Alternative Paths, Negative Tests and Acceptance Criteria

Acceptance criteria in a user story are like an informal contract - if they are met and quality assurance engineers / testers are happy, then a delivery team can argue that the story is complete, their work is done, they are free to move on to the next piece of work. Yet there are many potential behaviours of the software that could still happen which would lead to an unsatisfactory result. As a business analyst, this puts added importance to what you put down in the story.

Consider this example: The French government tells Facebook that French users can only log in via www.facebook.fr, and that it would be illegal for them to log in from www.facebook.com. (One can only imagine - perhaps to protect the French language?!) The definition of a French user - by IP address or country of residence - is not addressed for the purposes of this article. As part of meeting the regulations, we could have the following user story:

As a Facebook executive,

I want French users to to get an appropriate message when they are logging in to facebook.com

so that they understand they need to go to facebook.fr instead.

Acceptance Criteria

1)  A French user cannot successfully log in to facebook.com.
2) There is a message explaining to the customer after trying to log in that they need to go to facebook.fr (text to be confirmed with customer communications team)
3) By default, the message is in French.
4) Upon clicking an “OK” button in the message, the French user is redirected to facebook.fr.

In this example, there are some potential user actions that are not mentioned, yet somehow need to be covered to satisfy the overall business picture...

a) What happens when a French user enters an incorrect password?

b) What if a user hits the back button on their browser (and potentially the code might let them into facebook.com)?

c) How does a user see the message in other languages?

To each of these potential actions, let’s imagine that we do not get the behaviour we expect, yet the delivery team are claiming that acceptance criteria are met, thus the story should be signed off.

For a) we can agree that a user should not be logged in. But should they get the “incorrect password” message, rather than the “France redirection” message? If the latter happened I would say the user would be unaware that they are closer to a limit that puts a security flag against their account. This is because, with my own knowledge of how security works, the customer would not be informed of the incorrect password entry, meanwhile an addition is made to the incorrect password count.

Therefore, the “incorrect password” message should come first. Should the QA engineer / tester know this? Shouldn’t they have picked this up in regression testing? Much here depends on the technology organisation - whether it can be expected that QA engineers know how the system works on such a level. On balance, I would argue that for this alternative path, we are introducing a business rule (the new France message follows the “incorrect password” message in precedence). Therefore, a separate user story, correctly linked to the first, should be created and tested in its own right. I would also then be more stringent in the original user story, stating that it only applied to “validated” customers, where “validated” is defined as having successfully entered in valid credentials.

For b), I would not accept a delivery team member saying “but you didn’t mention this in the acceptance criteria”. This is clearly a defect and against the spirit of the story. With agile software development, business analysts cannot do requirements to such detail, nor should they. There should be a level of common sense and understanding of how things should work, without it needing to be spelt out. Plus, if I did cover this scenario in the requirements, where would it end? (back button + “Enter key”, “Esc” button,  etc...) This behaviour should be found through “negative testing” - verifying that what shouldn’t happen, doesn’t. Acceptance criteria should generally focus on what should happen.

For c), here I would say it is something the business analyst should have mentioned in the acceptance criteria in the story. Having French users understand they should go to a different page is part of the story. Furthermore, the story has mentioned that the default language is French. It needs to be mentioned how a user gets the message in a different language. There could either be a dropdown in the message for switching language, or the user could select one of the existing language links towards the bottom of the Facebook homepage. Both are feasible options, yet a key stakeholder from the "user experience" team may feel strongly one way or the other. Thus it needs to be both documented and part of this story. Therefore an extra acceptance criteria is needed.

To summarize how to cover alternative user journeys (paths), I would ask whether the expected behaviour is minor and should be commonly understood by the delivery team, as well as if it is essential to the user story under test. See the illustration below...  

Please let me know your thoughts...