Sample Size calculation, defect analysis

[cinger]

I am testing a change made to a software application; a function had to be rewritten to perform in a completely different way. The specific part of this application iterates through input data sets and outputs the data to another application. In order to test that data was being sent correctly, 15,000 data sets were passed through. This is an time-expensive test, taking about 10 hrs of labor for processing, and 2-3hrs for data analysis. During this test, 1 defect was encountered where the specific function failed to send data. Subsequently, they reviewed the code and attributed the defect to a incorrectly double incrementing a counter where it should have only incremented once. The code was corrected, and another test of 15,000 was performed with 0 defects of this specific type seen. Other defects occurred, but they were completely outside the scope of the specific function in question. Once the software is complete and ships, it will be responsible for passing millions of transactions in this manner.
How would confidence levels be determined for the first and second tests, i.e., how confident can we be with 0 defects seen in the second test?
The first test showed 1 defect in 15000 samples, or a defect per million encounters of 67 defects per million; how confident is this defect per million estimate?
How would adequate sample size be determined?
Would p-charts (http://en.wikipedia.org/wiki/P-chart" ) be appropriately used here?
If p-charts are useful, I have attempted to use the adequate sample size calculation listed on the wiki page; I wasn't sure of the units but could post some of that as well if it is applicable here.

 

[mXSCNT]

It doesn't apply because you fixed the bug, so you should have no defects of that type in the future.

Also, your test data sets are not necessarily representative of the data sets that you will find in practice - a bug that happens once in 15000 runs might actually happen 1 in 100 runs for real data, or it might happen 1 in 2 million runs for real data. So you can't draw conclusions about real usage based on your test run.

 

[cinger]

The data from the tests had been selected to be as representative of actual field data as possible, with an increased percentage of input data that would catch errors associated with the type of bug encountered, due to the software's new functionality and software changes. This particular bug has been fixed, and performing the test again no defects were seen; but perhaps there are other bugs with smaller probability of error, it is very large software. May not have defects of this type attributed to this bug, but it is possible to have defects of this type attributed to perhaps different bug. Can any statistical information be derived from these tests? Can an adequate sample size be determined?

 

[mXSCNT]

The problem is still that your test data is not field data. You could use the student's t distribution to get a confidence interval for the bug rate... but it wouldn't be very meaningful. Especially since there may be many bugs that exist that you have not noticed.

You could just track the absolute number of unfixed functional bugs. Maybe divide it by the total lines of code or by the number of current developers to get a proportional figure.

 

[blue_raver22]

I would approach this situation by first considering the purpose of the testing and the potential impact of the software application. If the software is responsible for handling millions of transactions, it is crucial to have a high level of confidence in its performance.

In terms of sample size calculation, the initial test of 15,000 data sets is a good start, but may not be enough to fully assess the reliability of the software. It would be beneficial to conduct additional tests with varying sample sizes to determine the optimal number of data sets needed to achieve a desired level of confidence.

In terms of defect analysis, it is important to not only focus on the specific function that was rewritten, but also to consider the overall performance of the software and the potential for other defects. The fact that other defects were encountered during the testing process highlights the need for a thorough and comprehensive approach to testing.

To determine confidence levels for the first and second tests, statistical analysis can be used. This would involve calculating the confidence interval for each test and comparing them to determine if there is a significant difference between the two. Additionally, the defect per million estimate from the first test can be used to determine the probability of encountering a defect in the second test.

In terms of determining an adequate sample size, it would depend on the desired level of confidence and the margin of error. It would also be important to consider the complexity of the software and the potential for different types of defects.

P-charts could be useful in this situation as they are commonly used to track the occurrence of defects over time. However, it is important to note that they are typically used for ongoing processes, rather than one-time tests. In this case, it may be more useful to focus on the specific defect that was encountered and track its occurrence in future tests to ensure it has been fully resolved.

In terms of the calculation for adequate sample size, it would be helpful to have more information on the specific data and variables involved. It may be beneficial to consult with a statistician to determine the most appropriate method for calculating sample size in this situation.

Overall, the key in this situation is to approach the testing process with a thorough and systematic approach, considering all potential factors and using statistical analysis to determine confidence levels and adequate sample sizes. This will help ensure the reliability and effectiveness of the software application.