Appendix A:  Sampling Theory for Voice Routing Field Test

The theory behind the staged statistical sampling design for the conduct of the acceptance test of the voice routing system is discussed in this section.

The acceptance test developed for this application begins with the following considerations.

1. System failure rate (p₀) and probability of error in concluding acceptance test failure (α) – It is initially proposed that a system failure rate of less than one percent is desired.  For this acceptance test procedure, we will collect a data sample and count the number of failures found in this sample.  We will then compare the proportion of failures in the sample against the desired system failure rate.  If the estimate is much greater than one percent, we will conclude that the true system failure rate is likely greater than one percent.  However, there is a possibility that we could make this conclusion in error (i.e., even though the sample failure rate was high, the true system failure rate could be one percent).  The probability of this error is denoted α, and we desire that it be a low value. This value is commonly set at approximately five percent for acceptance sampling.
2. Tolerable upper limit on system failure rate (p₁) and corresponding probability of error in failing to reject initial assumption (β) – If the true system failure rate exceeds the desired one percent threshold, the sample failure rate may not be high enough to recognize this. For any value over one percent, there is a probability β that we will fail to conclude the system failure rate is over one percent.  We desire that β be a small value (as with α, the value is commonly set at approximately five percent for acceptance sampling) at some true system failure rate greater than the desired system failure rate.  This point is often referred to as the tolerable upper limit.  We initially propose a tolerable upper limit on system failure rate of five percent.
3. For sampling efficiency, the smallest sample size (n) and acceptable number of failures (c) that meet the criteria above should be selected.

Noting that the sample response measurements for this acceptance test fall into only two categories (success or failure), the acceptance test quantities can be related by the following equations, based on the binomial probability distribution:

From our initial requirements of the acceptance testing, we have established target values for p₀ (0.01), α (0.05), p₁ (0.05), and β (0.05).  Since these equations produce discrete values, it is not necessarily possible to solve the two equations exactly for desired n and c.  Instead, we develop a solution by fixing p₀ and p₁ and then experimenting with different values of n and c, searching for corresponding α and β reasonably close to the desired values.

For this particular acceptance test, an appropriate solution is found at n=147 and c=3 with p₀=0.01, p₁=0.05, α=0.06 and β=0.06.  This leads to the following test:

1. Select a sample of 147 items and count the number of failures.
2. If the number of failures is 3 or less, there is not sufficient evidence to disprove an initial hypothesis that the true system failure rate is less than one percent and the test passes acceptance.  It is possible this conclusion is incorrect (i.e., that the true system failure rate is greater than one percent).  However, if the true system failure rate is five percent more, there is only a six percent chance of this error.  Above a five percent system failure rate, this probability is even lower.
3. If the number of failures is 3 or more, there is sufficient evidence to disprove an initial hypothesis that the true system failure rate is less than one percent and the test fails acceptance.  It is possible this conclusion is incorrect (i.e., that the true system failure rate is one percent or less).  However, if the true system failure rate is one percent, there is only a six percent chance of this error.  Below a one percent system failure rate, this probability is even lower.

The practical interpretation of this test is:

• If the test passes acceptance, we are fairly certain that the true system failure rate does not exceed five percent;
• If the test fails acceptance, we are fairly certain that the true system failure rate is not less than one percent; and
• If the true system failure rate is between one and five percent, there is a definite risk that we passed acceptance when we should not have.

Reducing Tolerable Upper Limit

From the test above, true system failure rates between the p₀ and p₁ values are a problem because there is a definite risk in these cases that an acceptance sample will pass for these values when it should have failed.  Therefore, it is desirable to make p₁ as close as possible to p₀.  However, for a fixed p₀, α, and β, this can only be done by increasing the sample size n.  Modifying the acceptance sample plan above, p₁ can be reduced to four percent for n=294 and c=6 with p₀=0.01, p₁=0.04, α=0.03 and β=0.05.

Though this acceptance test has slightly better α and β and is not exactly comparable to the original test, it does show that we essentially need to double the sample size to get a one percent reduction in the tolerable upper limit.

Staged Sequential Sampling

If acceptance samples are not collected all at one time, it may be possible to evaluate the sample results as they are obtained.  The acceptance test criteria can be modified in this case to allow consideration of these interim sample results as a means of coming to an early conclusion of whether an acceptance test passes or fails.  The general procedure would be to collect a predetermined number of samples and then evaluate the number of failures.  If at or below a certain number, c_1A, conclude the acceptance test passes.  If at or above another number, c_1R , which is greater than c_1A, the acceptance test fails.  If the number of failures falls between c_1A and c_1R then complete another stage of sampling.  At the end of this second stage of sampling, compare the cumulative failures to a new set of acceptance limits to determine acceptance, failure, or continuation.  This process can continue for as many stages as desired before reaching a final stage, where the cumulative number of failures from all stages is compared to a single acceptance limit, c_T.  If the cumulative total of failures from all stages is at or below the limit, the acceptance test passes.  Otherwise, it fails.

If the desired acceptance plan has two stages of sampling, the previous equations can be modified to account for a potential additional stage of sampling by

A final solution that satisfies the equations above is n₁=147, n₂=147, c_1A=1, c_1R=5, c_T=6 with p₀=0.01, p₁=0.04, α=0.06 and β=0.06.  The test is completed as:

1. Select a sample of 147 items and count the number of failures.
2. If the number of failures is 1 or less, there is not sufficient evidence to disprove an initial hypothesis that the true system failure rate is less than one percent and the test passes acceptance.
3. If the number of failures is 5 or more, there is sufficient evidence to disprove an initial hypothesis that the true system failure rate is less than one percent and the test fails acceptance.
4. If the number of failures is 2, 3, or 4, no immediate conclusion is reached about acceptance or failure.  Instead, another sample of 147 items is taken, the failures are counted, and these failures are added to the original stage failures.

1. If the aggregate number of failures is 6 or less, there is not sufficient evidence to disprove an initial hypothesis that the true system failure rate is less than one percent and the test passes acceptance.
2. If the aggregate number of failures is 7 or more, there is sufficient evidence to disprove an initial hypothesis that the true system failure rate is less than one percent and the test fails acceptance.

5. It is possible that an overall passing acceptance result from this test (regardless of whether it happens in the first or second stage of sampling) is incorrect (i.e., that the true system failure rate is greater than one percent).  However, if the true system failure rate is four percent or more, there is only a six percent chance of this error.  Above a four percent system failure rate, this probability is even lower.
6. It is possible that an overall failing acceptance result from this test (regardless of whether it happens in the first or second stage of sampling) is incorrect (i.e., that the true system failure rate is one percent or less).  However, if the true system failure rate is one percent, there is only a six percent chance of this error.  Below a one percent system failure rate, this probability is even lower.

Relative to the single stage sample plan of 147 samples, this final sample plan provides the desired protection against the lower tolerable upper limit at four percent instead of five percent.  However, it could require two stages, or 294 samples to achieve this.  Relative to the single stage sample plan of 294 samples, this final sample plan provides the possibility of finishing sampling in half the total number of samples.  However, its α and β are not quite as good as the single stage sample plan.  Balancing these two considerations, the two-stage sample is recommended for this particular application.

Appendix B: Data Collected for Acceptance Test of FOT Voice Routing Functionality

Appendix C:Data Summaries for FOT Data Routing Performance