CSCI 265 Project Phase 5: user acceptance testing

CSCI 265 Project Phase 5: User acceptance test process document (and updates)

In this phase we're back to having a fairly substantial document (the user acceptance test document) plus creating the team repo test infrastructure and data files needed to carry out the plan (on the still-to-be-completed final product).

Free advice: start early on this one and divide up the responsibilities, there are lots of big but somewhat independent sections to be done.

It relies heavily on the requirements document being complete and up-to-date with fixes and revisions, so the team will need to finish any remaining gaps in that document in order to proceed with a solid test plan.

As always, you'll be fixing previously identified issues, and providing an update on changes to all past documents and on the current state of the product development.

Deliverables for phase 5 are divided into four key areas:

a project update and suitable revisions to earlier project documents (due by 8pm Friday November 22nd)
writing and submitting the user acceptance testing document (also due by 8pm on the 22nd)
creating the scripts, directories, data files, and any needed external resources (e.g. web/database servers)
giving a 10-14 minute team presentation on key updates and test process/plan (to be done in your team's lab section, Nov. 20th or 22nd)

*The individual contribution assessments are due 48 hours after the other documentation.

Marks breakdown [30 marks total]

Project update and matching revisions to the charter, standards, requirements, and design documents (plus appropriate matching changes to the repo content) [5 marks]
Test document + infrastructure [20 marks]
Presentation [5 marks]

Project update and document revisions

A formal project update should be provided in an update.md file, outlining

key changes in project requirements (and .md document) since phase 4
key changes in the team charter (and .md document) since phase 4
key changes in the standards (and .md document) and processes since phase 4
key changes in the design (and .md document) since phase 4

All changes noted should, of course, be reflected in the other relevant project markdown document(s) and team repo files/structure where appropriate.

The user acceptance testing document

The document will have four key sections:

known issues,
the test plan to be followed,
the test infrastructure to be used,
detailed descriptions of all the individual test cases.

Known issues

This is likely to be a large and highly detailed document, describing many scripts, files, and other resources on the team repo (and possibly elsewhere). It is likely that the team will not have the time to fully complete all the various repo-side test content, so the known issues section should clearly and precisely identify what is incomplete (along with any other known errors/omissions in the document).

The test plan

This section should provide the user with a full understanding of the process to be followed: starting with an overview of the process and an introduction to the key anticipated challenges, then laying out the intended timeline, the roles and responsibilities for the different testers, the process to be followed in detail, and a summary of all the test cases to be applied. (The full detailed description of each test case can then be included in a document appendix.)

Testing overview

This section needs to provide a general description of the testing environment, process, test cases, challenges, and timeline for the user acceptance testing of your project. In essence it summarizes the most important points of each of the other sections of the document and gives the reader the needed background/context/organizational structure before reading the various (more detailed) later sections.

Key testing challenges

In this section the objective is to give the reader a clear understanding of the biggest challenges/obstacles in the thorough testing of all the requirements for your product. (The challenges faced will be very particular to the specific product and its runtime environment.)

Testing timeline

This will be a purely hypothetical exercise that assumes the core product/features are completed by Dec. 6th and that each member of the team has about 10 hours per week available for the project testing after Dec. 6th.
Based on those assumptions, provide a short timeline that covers

the completion of each unfinished part of the test infrastructure (scripts, data files, etc),
the completion of each unfinished part of the test documentation (e.g. the test case appendices),
one complete run of every test case and all the associated reporting.

For each aspect, be sure the timeline notes which team member will be carrying out the task.

Test process

The intent of this section is to ensure the reader understands the process the testers will follow with each of the different types of test. This section should provide an overview of

the general roles/responsibilities of the testers,
the general test environment, tools, and infrastructure used,
the typical setup process prior to running a test case,
the typical process for running a test case,
the typical process for evaluating and reporting on a test case,
any details needed to ensure the test process adheres to the version control process laid out in the team standards.

This is essentially a high level/intuitive summary of the remaining sections of the document.

Test cases summary and organization

This section should describe the core different types of test case relevant to your product and how the test cases have been organized. As an example, a breakdown might be something like:

tests involving visual inspections of the screen elements,
tests involving navigation between different parts of the system,
tests involving interactions with the database,
tests involving interactions with the AI,
testing of the map structure,
testing of NPC behaviour,
testing of combat mechanics,
etc etc etc.

Since this is specifically for user acceptance testing, the organization of your test cases should make it very easy for a reader to confirm that every aspect of the requirements document is indeed covered by these tests. One suggestion would be to organize the test cases to parallel the structure of the requirements document (so you could have the two documents open side-by-side and as you read through the requirements sections you could see the matching tests in the test document), while another might be to provide an additional field in the list/table below to identify which (sub/subsub)section of the requirements is addressed by each test case.

Test case list

A table or list of all the test cases to be applied, organized into intuitive sections/subsections, with each entry providing the test case id/name (linked to its detailed description in the appendix), and a few words clarifying the unique purpose/aspect of the test case.

A couple of sample entries might be something like the following (links not represented here):

PwdInv01	First bad password	one entry of bad password by registered user
PwdInv03	Third password failure	third consecutive bad password by registered user

The test infrastructure

This section should outline everything needed in order to carry out the actual testing. The breakdown below provides a suggested structure, but (depending on the nature of your product and operating environment) you are welcome to organize it differently as long as equivalent/suitable information can all be quickly and easily found within the document by a first-time reader.

Software tools and environment

The objective for this section is to describe the environment(s) in which you'll be running the product while testing and any additional tools you'll be using to conduct/monitor/report the test results.

This can vary considerably depending on the project/team, e.g.:

For a simple client-side game this might just specify whose laptop(s) and OS/version(s) you're testing on and that you're following manual 'user action scripts' as the process.
For a product that connects to other elements (AI, databases, web servers, etc) it will also need to specify which ones and how (this can be an overview, with details given in the 'External resources' section below).
If you're using third party tools (e.g. to record/playback UI actions) then those need to be listed and their use outlined here.

User action scripts

To ensure test results are reproducible when carrying out manual user acceptance tests, we can prepare and follow 'user action scripts'. These can be thought of similarly to the script for a play: they detail precisely what actions each user/tester must take and when, with the assumption that during testing the tester will be reading through the script, performing the actions as described.

The scripts can be organized into logical sections (or scenes, if we're using a theatrical analogue), e.g. a script that just covers the registration process, a script that just covers the login process (and assumes a previous registration script has been followed), a script that just covers a user sending a message to another user (and assumes both users have been previously registered and the sender has already logged in).

For each script there should be (at least):

a unique id/name for the script
a short description/overview of the script
a list of which other scripts must have been 'run' (followed) prior to running this script (ideally linked to the descriptions of the other scripts)
the exact sequence of actions each tester must follow during the scene covered by the script
the expected behaviour/results and how to assess pass/failure

Note that the descriptions of the actions to be performed must be precise and detailed enough to guarantee we will get the same results if we run the same script in the future.

Executables (test programs/scripts, stubs and drivers)

This section details any/all the scripts and programs the team has created to help automate the user acceptance test process. This would include programs/scripts such as:

those used to run specific test cases or batches of test cases,
stubs and drivers used to apply user acceptance tests to isolated portions of the system,
programs/scripts used in the process of setting up, evaluating, or cleaning up after different test cases (or sets of test cases),
any special-purpose scripts/programs related to the unique nature of an individual product.

For each script there should be (at least):

the name and location of the script/executable,
a link to the necessary source code and build location for executables,
a short description/overview of the purpose of the script/executable,
detailed descriptions (or links to such descriptions) for all the setup actions that need to be performed prior to running this script/executable,
the exact sequence of actions each tester must follow during execution of the script/executable,
the expected behaviour/results and how to assess pass/failure,
any required cleanup after the script has terminated.

Version control and branch structure

This section should clearly describe how the testing process fits with the teams version control standards and processes (i.e. what branches are used for testing, how they are set up/brought up to date immediately prior to testing).

Directory and file structure and content

This section should clearly outline the directory structure and file organization for the project test content, providing things like:

an overview of what kinds of files are located where in the directory tree (action scripts, test executables and their sources, files containing test case input, files containing expected output, etc),
an overview of any naming conventions for the different kinds of files used: ideally names provide an indication of what kind of file it is (action script, stub, driver, test executable, user input data, expected output, etc) and what kind of testing it is used for (e.g. uix tests, navigation tests, etc).

Independent subsystems and external resources (project web/database servers)

This section needs to provide full details on all the external resources the client side of the product relies upon/connects to, including things like

any AI components, web servers, and/or database servers run as part of the project,
any third party web servers and database servers that the product interacts with,
any third party AI components the product interacts with.

This section needs to provide all the details needed during testing to setup/run or connect to the relevant resources. These details can be provided in this document section or the document can link to later sections (appendices possibly?) where those details are provided in full.

Appendix: detailed test case descriptions

For every test case listed in the earlier summary, this section should provide all the details about that case, typically with a one-page form for each test case. Depending on the nature of your product and the test case, this likely includes most of the following:

the test case id, name, one-line description,
an overview of the test covering both the specific purpose of this test case and the key steps involved in running the test case,
a link to the specific section of the requirements document that describes the portion/aspect/feature of the product that is being tested here,
how many testers are needed and what they are labelled as in the test case actions,
what scripts (action and/or executable) need to be run prior to running this test case and what (specific) arguments/options need to be used when running them, ideally with links to the scripts themselves,
exact details on all other settings/ actions that need to be performed prior to running the test case (e.g. what fixed seed is used by the pseudo-random number generator if the game/product makes use of any form of randomness), (these may link to files/documents detailing sets of actions that are used in multiple test cases),
what scripts (action and/or executable) need to be run to conduct this test case and what (specific) arguments/options need to be used when running them, ideally with links to the scripts themselves,
all expected results/output from the test case, in precise enough detail to allow pass/fail evaluation,
any cleanup scripts that need to be run or other actions that need to be taken to 'tidy up' following the test case (e.g. your test case might involve things like changing permissions or contents for specific files/directories/users: things you'll want to undo those following the test),
the reporting process to be followed: e.g. links to which forms are to be filled in, who to notify with pass/fail results, etc (this may be a link to one of several standard reporting processes the team has created for their project testing).

A note on completeness: ideally this appendix would have a completed form for each and every test case from your "Test case list" section, but you may find that list contains many dozens (if not hundreds) of cases, and filling out every field of every form will be an extraordinarily time consuming process for the whole team.
As such, for this phase's requirements (since time is tight), it will be considered acceptable if roughly a dozen test case forms are actually completed in detail, as long as they represent a good cross-section of the overall collection of test cases.
A note on automation: it wouldn't be a bad plan to have someone write a short script that reads a simple list of test case ids+names and generates two files you can then include in your test document:
1) The collection of empty markdown forms for the test cases (with just the id/name filled in)
2) The list of test cases ids/names/links (to put in the earlier "Test case list" section of the document).
That would be good scripting practice and will save you a lot of time and annoyance in generating the raw forms and links.
If you do undertake this, please be sure that script gets mentioned as part of your overall test process (to make sure you get credit for that automation effort).

Sample documentation: as with previous phases, sample documentation for phase 4 can be found in the example project documentation.
EDIT: document still in preparation at the moment

The presentation

As with previous phases, each team will provide a 10-14 minute formal presentation discussing their current deliverables.

This presentation should

begin with a summary of any major changes to the earlier product requirements, team charter, standards/processes, and/or design,
provide a short update on the state of development of the product,
outline the key elements of your test plan: the major components to be tested, the process to be followed/used, the most significant challenges to the testing of your product
outline the infrastructure to be used to conduct the testing: software/tools to be used, user action scripts, executable scripts (stubs/drivers/automation), external resources to be created/configured (web/database servers)
summarize the state of the document and supporting infrastructure (how much is still T.B.D. at the time of the presentation)

Assessment and marks

Be sure all team members have read the contributions assessment page, and complete and submit their assessments by 8pm Sunday November 24th (phase5.md in your individual contribution repos).