Skip navigation links
(Wind-US Documentation Home Page) (Wind-US User's Guide) (GMAN User's Guide) (MADCAP User's Guide) (CFPOST User's Guide) (Wind-US Utilities) (Common File User's Guide) (Wind-US Installation Guide) (Wind-US Developer's Reference) (Guidelines Documents)

(Programming Guidelines) (Documentation Guidelines) (Testing Guidelines)

Testing Guidelines for
NPARC Alliance Software Development

Introduction

This document provides testing guidelines for NPARC Alliance software development efforts. It is tailored to the code resulting from the NPARC/NXAIR code merger effort, but with some modifications, could be applicable to other software development projects. It is intended to be used in the development phase of the code, and provides guidelines for evaluating the following:

The test cases described in this document are intended to be run in a relatively simple and quick manner, as much as each flow case permits. They are not intended to provide an in-depth and highly detailed validation of the code. (Detailed validation will most likely occur at a later stage in the code development and will be modeled after the "NPARC Validation Standards" included in the Appendix.) Each test case will be documented and archived so that users/developers will have access to input/output files, plotted results, and other pertinent information. Wherever possible, the computed results for each test case should be compared with experimental data and/or a solution computed with another flow solver.

Testing Overall Software Functionality

In this document, the "functionality" refers both to the mechanical aspects of the software, i.e. it's ability to compile, run and produce the desired output, and to the software's ability to compute the correct flow physics. The following cases are to be used for testing overall functionality:

Procedure

When a developer makes changes to the code, the overall code functionality must be evaluated before the changes are incorporated into the controlled version of the software. The procedure for evaluating the code's functionality is described below.

Documentation

The first time a test case is run, it should be documented to include the items listed below. Subsequent runs will be noted in a log file. The suggested format for the documentation is PostScript, since the documentation will contain both text and graphics. The documentation will include:

Archive

The following files will be included in the validation archive. Unless noted otherwise, all should be plain ASCII files.

Testing Software Features

Much of the code development effort involves adding or improving specific code functions, and therefore test cases must be run to evaluate thesefunctions. The table below indicates test cases to be used for evaluating several code features. (Other cases and features may be added, as needed.) One test case is indicated for each code feature, with the exception of turbulence, which indicates one free shear case and one wall bounded case. Cases marked with an asterisk (*) have been validated for the NPARC code, and detailed information about these cases is available in the NPARC validation archive. The documentation and archival procedures described for testing overall software functionality should also be used here.

Test cases for evaluating code features.
Test Case 2D Axi 3D Var Width Unsteady Turb Chemistry Subsonic (M<0.1) Subsonic (0.1<M<1.0) Transonic Supersonic Hypersonic Multiple Blocks Moving Grids Artif. Viscos.
Laminar Flat Plate* x  x           
Transonic Airfoil          x    x
Oscillating Airfoil     x          
S-Duct*   x            
Subsonic Diffuser*      x  x      
Supersonic Nozzle*  x   x    x x  
Low Speed Flow (M<0.1)        x       
Simple Combustion       x        
Store Separation              x 
Re-entry Vehicle            x   

Appendix - NPARC Validation Standards

Introduction

The NPARC validation effort is intended to establish the basis upon which confidence in results produced by the NPARC code is founded, and the practical limits on the accuracy of its predictions. Such confidence can only be achieved through a continuous process of careful application of the code to a wide range of "unit" and "configuration-oriented" problems and complete documentation of results. Here, "unit" refers to problems focusing on a single phenomenon and simple geometries, whereas "configuration-oriented" refers to problems which focus on geometries and flows more representative of realistic aerospace configurations. A wide variety of problems representing a mix of flows have been identified as candidates for validation cases. Cases are being run to determine the strengths and weaknesses of the NPARC code for a variety of geometric configurations, and over a range of flow parameters. Computed results are being compared with benchmark-quality experimental data, well-accepted computational results, and/or analytic solutions.

For these validation cases to be of maximum benefit to the user community, the results must be made readily available. The information should be also be detailed enough to permit the calculations to be repeated with relative ease by independent code users. A formal electronic archive system has been established to meet these criteria. This archive is the central repository for the validation cases computed by and for the NPARC Alliance. It is intended for use by NPARC users and developers, and allows easy access to the results of latest validation studies. The archive includes all the input files used to run the cases, the output files, the experimental data used for comparison with computed results, and written documentation providing an overview of each case and a discussion of the results. The archive is accessible over the Internet from the NPARC WWW home page located at http://www.grc.nasa.gov/WWW/wind/index.html.

The validation effort is expected to be an on-going activity and, through the NPARC User's Association, users are encouraged to propose candidate validation problems and submit documentation and results from independent validation efforts. This document is intended to assist users submitting the results from validation efforts by: (1) defining the three types of cases recognized in the NPARC validation effort; (2) describing the documentation required for each of them; and (3) listing the specific files to be included in the validation archive. Users are also encouraged to examine the existing documentation and files in the archive, and use them as examples when submitting their own results for inclusion in the archive. Questions and/or comments should be directed to the NPARC Validation Team at nparc-validation@info.arnold.af.mil.

Model Validation Study

Definition

The term "validation" has been used in a variety of ways in the literature. In the NPARC effort, we are guided by the following definition, adapted from one given by Mehta (Mehta, U. B., "Computational Requirements for Hypersonic Flight Performance Estimates," Journal of Spacecraft and Rockets, Vol. 27, No. 2, 1990, pp. 103-112).

A code is said to be validated if the following conditions are met: (1) a comparison of computed results with detailed surface and flow field experimental data and/or other well-accepted solutions shows that the code is able to accurately model the critical physics of the flow; (2) the accuracy and limitations of the experimental data are known and understood; and (3) the accuracy and limitations of the code's numerical algorithms, grid density effects, convergence effects, and physical basis are known and understood. The range of applicability of the validated code depends on the range of flow parameters and/or geometric configurations for which the code has been validated.

Of course, in practice the accuracy and limitations of the experimental data and the computational results cannot be fully "known and understood." In addition, the degree to which the code must "accurately model the critical physics of the flow" will depend on how the results are to be used. These factors inevitably introduce some blurring of the line between the states of validation and non-validation. Nevertheless, this definition does serve to provide the necessary philosophy that guides the validation effort. NPARC validation studies which attempt to meet this strict standard are termed model validation studies.

Model studies are run to determine the strengths and weaknesses of the NPARC code for a variety of geometric configurations, and over a range of flow parameters. The accuracy and limitations of the code are investigated by examining the sensitivity of the results to various input options such as mesh density, turbulence model, and artificial viscosity model.

Documentation

Each model validation study is documented, in a (mostly) consistent format, as part of the validation archive. A consistent format for the documentation of validation studies is necessary to ensure an adequately thorough representation of a given study and to permit a comparison of conclusions drawn from a variety of studies. The documentation highlights the primary focus and pertinent findings, and, with the files in the validation archive, includes sufficient detail to allow independent repetition of each case.

The documentation for a model validation study should include the following elements:

Archive

For a model validation study, the following files will be included in the validation archive. Unless noted otherwise, all should be plain ASCII files.

Example Validation Case

Definition

Example validation cases are established and documented in coordination with the NPARC Support team. There are two primary goals which the example validation cases are designed to meet. The first is to provide users with quick, but limited validation of the NPARC software over a wide range of flows. These validation cases are indicative of the capabilities of the flow simulation program, but do not meet the definition of a model validation case in that they do not examine the sensitivity of the results to various input options. The second goal of the example cases is to provide the new user with clear examples of how to properly set up and execute the NPARC code for a variety of geometries and flow conditions.

Documentation

Like the model validation studies, each example validation case is documented, in a (mostly) consistent format, as part of the validation archive. Since one of the goals is to illustrate how to set up and execute NPARC, the documentation typically includes a little more detail about the mechanics of running the case, including actual input listings, than the documentation for a model validation study. The documentation for example cases is automatically provided with the NPARC code as part of the NPARC User's Guide and/or as a separate document. It is also available as part of the validation archive.

The documentation for an example validation case should include the following elements:

Archive

For an example validation case, the following files will be included in the validation archive. Unless noted otherwise, all should be plain ASCII files.

Check Case

Definition

Check cases will be established to judge the functionality of a newly installed and/or modified code. These will be developed, maintained, and documented in conjunction with the Development Team. The primary intent of the check cases is to provide the Development team with a tool to ensure the integrity of all mechanical aspects of code operation. At least one of these cases will be an installation check case that may be used by new recipients of the NPARC code to verify that the code has been properly installed on their computer system.

Documentation

Each check case is also documented as part of the validation archive. Since these cases are intended primarily for use by the Development Team, they do not require as much detailed documentation as the model and example cases. The documentation for check cases will be provided with the NPARC code as part of the NPARC Developer's Guide and/or as a separate document. It will also be available as part of the validation archive.

The documentation for a check case should include the following elements:

Archive

For a check case, the following files will be included in the validation archive. Unless noted otherwise, all should be plain ASCII files.


Last updated 16 Mar 1999