(Wind-US Documentation Home Page)
(Wind-US User's Guide)
(GMAN User's Guide)
(MADCAP User's Guide)
(CFPOST User's Guide)
(Wind-US Utilities)
(Common File User's Guide)
(Wind-US Installation Guide)
(Wind-US Developer's Reference)
(Guidelines Documents)
(Programming Guidelines)
(Documentation Guidelines)
(Testing Guidelines)
Testing Guidelines for
NPARC Alliance Software Development
Introduction
This document provides testing guidelines for NPARC Alliance software
development efforts.
It is tailored to the code resulting from the
NPARC/NXAIR code merger effort, but with some modifications, could be
applicable to other software development projects.
It is intended to be
used in the development phase of the code, and provides guidelines for
evaluating the following:
- Overall functionality of the software
- Functionality of specific features of the software
The test cases described in this document are intended to be run in a
relatively simple and quick manner, as much as each flow case permits.
They are not intended to provide an in-depth and highly detailed
validation of the code.
(Detailed validation will most likely occur
at a later stage in the code development and will be modeled after the
"NPARC Validation Standards" included in the
Appendix.)
Each test case
will be documented and archived so that users/developers will have
access to input/output files, plotted results, and other pertinent
information.
Wherever possible, the computed results for each test case
should be compared with experimental data and/or a solution computed
with another flow solver.
Testing Overall Software Functionality
In this document, the "functionality" refers both to the mechanical
aspects of the software, i.e. it's ability to compile, run and produce
the desired output, and to the software's ability to compute the correct
flow physics.
The following cases are to be used for testing overall
functionality:
- Laminar flat plate
- Turbulent flat plate
- Driven cavity
- Wing/airfoil
- Shock tube
- Simple combustion
Procedure
When a developer makes changes to the code, the overall code
functionality must be evaluated before the changes are incorporated into
the controlled version of the software.
The procedure for evaluating the code's functionality is described below.
- Choose at least 1 test case (whatever seems most appropriate).
It is left to the developer to decide how many cases to run.
- Run each case until residuals drop at least three orders
of magnitude, until residuals reach the level achieved in an
earlier computation of the case, or until some monitored flow
quantity stops changing appreciably.
- Visually examine solution (e.g. with Plot3d) to make sure
there are no glaring errors.
- Plot a representative sample of data and, where possible, compare
with a theoretical solution, experimental data and/or a solution from
another code.
- If the previous two steps reveal problems with the solution, fix any
problems with the code and repeat the above procedure until
all problems are eliminated.
- Document and archive the results. (See below.)
Documentation
The first time a test case is run, it should be documented to include
the items listed below.
Subsequent runs will be noted in a log file.
The suggested format for the documentation is
PostScript, since the documentation will contain both text and graphics.
The documentation will include:
- The version(s) of the code used to run the case, and the computer
hardware and operating systems used.
- A brief description of the problem being solved.
This should include flow conditions and descriptions of the
geometry and grid.
- A listing of the standard input file used to run the case.
(When a case consists of a series of separate computer runs,
changes made to the input for subsequent runs should be noted.)
- A description of the convergence history, preferably including plots.
- A representative sample of the computed results, and, where possible,
comparison with experimental data and/or a solution from another code.
Archive
The following files will be included in the validation archive.
Unless noted otherwise, all should be plain ASCII files.
- A README file that: (1) briefly describes the case and the
procedure used to run it; (2) lists and briefly describes all
the files in the archive relevant to the case; and (3) provides
the name, postal address, email address and phone number of the
contact person for the case.
- The mesh used for the calculation (32-bit IEEE Fortran
unformatted form).
- The initial restart file (32-bit IEEE Fortran unformatted form).
- The standard input and output from each run.
- The solution files from each run, including convergence history
information.
The files should be 32-bit IEEE Fortran unformatted and should be
in both of the following forms:
- Common File 3 (CF3) format
- Plot3d Format
- The documentation
- A log file of repeated runs.
Users/developers who re-compute a flow case should add the
following information to the log file: name and email address,
date the calculation was made, code version used, computer
software and hardware used, and comments about differences
from the documented results and any relevant findings.
Testing Software Features
Much of the code development effort involves adding or improving
specific code functions, and therefore test cases must be run to
evaluate thesefunctions.
The table below indicates test cases to be used for evaluating
several code features.
(Other cases and features may be added, as needed.)
One test case is indicated for each code feature, with the exception
of turbulence, which indicates one free shear case and one wall
bounded case.
Cases marked with an asterisk (*) have been validated for the NPARC code,
and detailed information about these cases is available in the
NPARC validation archive.
The documentation and archival procedures described for
testing overall software functionality
should also be used here.
Test cases for evaluating code features.
Test Case
| 2D
| Axi
| 3D
| Var Width
| Unsteady
| Turb
| Chemistry
| Subsonic (M<0.1)
| Subsonic (0.1<M<1.0)
| Transonic
| Supersonic
| Hypersonic
| Multiple Blocks
| Moving Grids
| Artif. Viscos.
|
Laminar Flat Plate*
| x | | | x | | | | | | | | | | |
|
Transonic Airfoil
| | | | | | | | | | x | | | | | x
|
Oscillating Airfoil
| | | | | x | | | | | | | | | |
|
S-Duct*
| | | x | | | | | | | | | | | |
|
Subsonic Diffuser*
| | | | | | x | | | x | | | | | |
|
Supersonic Nozzle*
| | x | | | | x | | | | | x | | x | |
|
Low Speed Flow (M<0.1)
| | | | | | | | x | | | | | | |
|
Simple Combustion
| | | | | | | x | | | | | | | |
|
Store Separation
| | | | | | | | | | | | | | x |
|
Re-entry Vehicle
| | | | | | | | | | | | x | | |
|
Appendix - NPARC Validation Standards
Introduction
The NPARC validation effort is intended to establish the basis
upon which confidence in results produced by the NPARC code is
founded, and the practical limits on the accuracy of its predictions.
Such confidence can only be achieved through a continuous process
of careful application of the code to a wide range of "unit" and
"configuration-oriented" problems and complete documentation of results.
Here, "unit" refers to problems focusing on a single phenomenon and
simple geometries, whereas "configuration-oriented" refers to problems
which focus on geometries and flows more representative of realistic
aerospace configurations.
A wide variety of problems representing a mix
of flows have been identified as candidates for validation cases.
Cases
are being run to determine the strengths and weaknesses of the NPARC
code for a variety of geometric configurations, and over a range of flow
parameters.
Computed results are being compared with benchmark-quality
experimental data, well-accepted computational results, and/or analytic
solutions.
For these validation cases to be of maximum benefit to the user
community, the results must be made readily available.
The information
should be also be detailed enough to permit the calculations to be
repeated with relative ease by independent code users.
A formal
electronic archive system has been established to meet these criteria.
This archive is the central repository for the validation cases
computed by and for the NPARC Alliance.
It is intended for use by
NPARC users and developers, and allows easy access to the results
of latest validation studies.
The archive includes all the input
files used to run the cases, the output files, the experimental data
used for comparison with computed results, and written documentation
providing an overview of each case and a discussion of the results.
The archive is accessible over the Internet
from the NPARC WWW home page located at
http://www.grc.nasa.gov/WWW/wind/index.html.
The validation effort is expected to be an on-going activity and,
through the NPARC User's Association, users are encouraged to propose
candidate validation problems and submit documentation and results from
independent validation efforts.
This document is intended to assist
users submitting the results from validation efforts by: (1) defining
the three types of cases recognized in the NPARC validation effort; (2)
describing the documentation required for each of them; and (3) listing
the specific files to be included in the validation archive.
Users are
also encouraged to examine the existing documentation and files in the
archive, and use them as examples when submitting their own results for
inclusion in the archive.
Questions and/or comments should be directed
to the NPARC Validation Team at
nparc-validation@info.arnold.af.mil.
Model Validation Study
Definition
The term "validation" has been used in a variety of ways in
the literature.
In the NPARC effort, we are guided by the
following definition, adapted from one given by Mehta (Mehta,
U. B., "Computational Requirements for Hypersonic Flight Performance
Estimates," Journal of Spacecraft and Rockets, Vol. 27, No. 2, 1990,
pp. 103-112).
A code is said to be validated if the following conditions are met:
(1) a comparison of computed results with detailed surface and flow
field experimental data and/or other well-accepted solutions shows
that the code is able to accurately model the critical physics of
the flow; (2) the accuracy and limitations of the experimental data
are known and understood; and (3) the accuracy and limitations of
the code's numerical algorithms, grid density effects, convergence
effects, and physical basis are known and understood.
The range of
applicability of the validated code depends on the range of flow
parameters and/or geometric configurations for which the code has
been validated.
Of course, in practice the accuracy and limitations of the experimental
data and the computational results cannot be fully "known and
understood." In addition, the degree to which the code must "accurately
model the critical physics of the flow" will depend on how the results
are to be used.
These factors inevitably introduce some blurring of the
line between the states of validation and non-validation.
Nevertheless,
this definition does serve to provide the necessary philosophy that
guides the validation effort.
NPARC validation studies which attempt to
meet this strict standard are termed model validation studies.
Model studies are run to determine the strengths and weaknesses of
the NPARC code for a variety of geometric configurations, and over a
range of flow parameters.
The accuracy and limitations of the code are
investigated by examining the sensitivity of the results to various
input options such as mesh density, turbulence model, and artificial
viscosity model.
Documentation
Each model validation study is documented, in a (mostly) consistent
format, as part of the validation archive.
A consistent format for the
documentation of validation studies is necessary to ensure an adequately
thorough representation of a given study and to permit a comparison
of conclusions drawn from a variety of studies.
The documentation
highlights the primary focus and pertinent findings, and, with the
files in the validation archive, includes sufficient detail to allow
independent repetition of each case.
The documentation for a model validation study should include the
following elements:
- Software/Hardware Description
The version(s) of NPARC used in the study should be stated, as
well as the computer hardware and operating system(s).
Any other software/hardware used, such as a mesh generation
package, should also be described.
For parallel operation, the PVM version that was
used should also be stated.
- Problem Description
The problem being studied should be described, including details
about the geometry and flow conditions.
The cases should be
summarized in tabular or similarly compact form, showing the values
of the relevant parameters that were used in the various sensitivity
investigations.
If the same or similar NPARC results have been
presented elsewhere, those publications should be cited.
The experimental data and/or computational results used for comparison
with the NPARC results should also be described and cited.
- Baseline Case
One of the cases run during the study should be chosen as a baseline
case.
This case does not necessarily have to be the one giving the
absolute best results, but should illustrate a reasonable way to
run the NPARC code for the problem being studied.
This baseline
case should be described in detail, much like an example validation
case (see below), with discussions of the reference conditions,
computational mesh, initial conditions, boundary conditions, solution
algorithm and time step, artificial viscosity, turbulence model,
convergence history, and computed flow field results.
The flow field results should be compared with experimental data
and/or other well-accepted computational or analytical results.
Unlike an example
validation case, the documentation for a model validation case need
not include a discussion of the Fortran include file, or a listing of
the NPARC standard input file.
- Sensitivity Investigations
As noted above, the accuracy and limitations of the code are
investigated by examining the sensitivity of the results to various
input options.
Exactly which input parameters should be varied may
differ from study to study.
As a minimum, though, the effects of the following parameters
should be investigated:
- The number of mesh points.
Ideally, for turbulent flow, the spacing between a
solid wall and the first interior mesh point
should remain constant.
- For turbulent flow, the spacing between a solid wall
and the first interior mesh point (i.e., the initial
y+ value).
The number of mesh points should remain constant.
- For turbulent flow, the turbulence model.
Other sensitivity investigations that may be appropriate include
the effects of boundary conditions, initial conditions, artificial
viscosity model and level, and time step option.
These sensitivity investigations should be described in detail.
Differences in the input options for these cases, from those used
in the baseline case, should be noted.
The convergence histories
should be discussed, and where appropriate, compared to the results
from the baseline case.
The sensitivity of the computed flow field
to the choice of input options should be examined by comparing with
the results from the baseline case, and with experimental data and/or
other well-accepted computational or analytical results.
- Summary
A summary section should be included, describing the lessons learned
in the validation study.
- References
A list of references cited in the text should be included.
Archive
For a model validation study, the following files will be included in
the validation archive.
Unless noted otherwise, all should be plain
ASCII files.
- A README file that: (1) briefly describes the problem being studied
and the procedure used to run it; (2) summarizes, in tabular or
similarly compact form, the various sensitivity investigations that
were done, showing the values of the relevant parameters that were
used; (3) lists and briefly describes all the files in the archive
relevant to the study; and (4) provides the name, postal address,
email address, and phone number of the contact person for the study.
- The source for the program used to create the mesh, if a user-written
custom mesh generator was used.
- The source for the program used to create the initial restart file.
- The initial restart file for each case, in 32-bit IEEE Fortran
unformatted form.
- The NPARC standard input (unit 5) and output (unit 6) files for each
run.
- The mean flow (unit 21) and turbulence model (unit 22) convergence
history files for each run.
These may be in ASCII or 32-bit IEEE
Fortran unformatted form, depending on the input parameters L2PLOT
and IFXPLT.
- The plot3d xyz and q files for the final run of each case, in 32-bit
IEEE Fortran unformatted form.
- Any other relevant input/output files, such as the input data file
for boundary conditions 30-32 (unit 3), or the output mass flow
convergence file for boundary conditions 90-96 (unit 11).
- A file or files containing any experimental data and/or other
well-accepted computational or analytical results, in tabular form,
that were used for comparison with the NPARC results.
- The documentation, in PostScript form.
Example Validation Case
Definition
Example validation cases are established and documented in coordination
with the NPARC Support team.
There are two primary goals which the
example validation cases are designed to meet.
The first is to provide
users with quick, but limited validation of the NPARC software over
a wide range of flows.
These validation cases are indicative of the
capabilities of the flow simulation program, but do not meet the
definition of a model validation case in that they do not examine the
sensitivity of the results to various input options.
The second goal
of the example cases is to provide the new user with clear examples
of how to properly set up and execute the NPARC code for a variety of
geometries and flow conditions.
Documentation
Like the model validation studies, each example validation case is
documented, in a (mostly) consistent format, as part of the validation
archive.
Since one of the goals is to illustrate how to set up and
execute NPARC, the documentation typically includes a little more
detail about the mechanics of running the case, including actual input
listings, than the documentation for a model validation study.
The
documentation for example cases is automatically provided with the NPARC
code as part of the NPARC User's Guide and/or as a separate document.
It is also available as part of the validation archive.
The documentation for an example validation case should include the
following elements:
- Software/Hardware Description
The version(s) of NPARC used to run the case should be stated, as
well as the computer hardware and operating system(s).
Any other software/hardware used, such as a mesh generation
package, should also be described.
For parallel operation, the PVM version that was
used should also be stated.
- Problem Description
The problem being run should be described, including details about
the geometry and flow conditions.
If the same or similar NPARC
results have been presented elsewhere, those publications should be
cited.
If the NPARC results are being compared with experimental
data or other computational results, these should also be described
and cited.
- Reference Conditions
The reference conditions used in the NPARC input should be stated,
and related to the geometry and flow conditions for the problem being
computed.
- Computational Mesh
The computational mesh, and the procedure used to construct it,
should be described.
Where appropriate, such as for cases with a
relatively simple customized mesh, this may include listings of the
mesh generation code and input.
For turbulent flow cases, the y+
value at the first grid point from the wall, and the method used to
compute it, should be discussed.
- Initial Conditions
The initial conditions, and the procedure used to construct them,
should be described.
Where appropriate, this may also include code and input listings.
- Boundary Conditions
The boundary conditions used for all boundary segments should be
described.
This should include, for each segment, the j, k,
and l grid point indices defining the segment, the
NPARC boundary condition
code number, and the values of the auxiliary variables.
- Solution Algorithm and Time Step
The solution algorithm and method for defining the time step should
be stated, especially for non-default values of ISOLVE and IVARDT.
When a case consists of a series of separate computer runs, the
values controlling the time step and the number of steps should
be given for each run.
The computer resources used, in terms of
run-time memory and CPU time, should also be stated.
- Artificial Viscosity
The smoothing parameters should be discussed, especially if any
non-default values are used.
- Turbulence Model
For turbulent flows, the turbulence model used should be stated.
The associated input parameters controlling the model, such as IFMAX
and the "EDGE" values for the Baldwin-Lomax model, should also be
discussed.
- Include File
For NPARC Versions 2.2 and earlier, the parameters used in the
NPARC.INC file should be listed.
For Versions 3.0 and later, this is
only necessary if changes were made to the nparc.inc file distributed
with the code.
- NPARC Input File
A listing of the NPARC standard input file used for the example case
(i.e., the information read from Fortran unit 5) should be included
in the documentation.
Since one of the goals of an example case is
to show a new user how to set up and execute the NPARC code, a short
description of each input item should be included, relating it to the
physical problem and/or the operation of the NPARC code.
When a case consists of a series of separate computer runs,
changes made to the input for subsequent runs should be noted.
- Convergence History
The convergence history should be described, including the residuals
for the mean flow and turbulence model equations, as well as any
flow-related parameters used to determine convergence.
This should include plots of the convergence parameters as a
function of time step.
- Computed Flow Field Results
A representative sample of the computed results should be shown,
and discussed briefly.
Extensive discussion of the results, such
as would be included in a report describing the solution of a flow
problem, is not necessary.
Discussion of specific elements of the
results that are related to the operation of the NPARC code is
appropriate, however.
- References
A list of references cited in the text should be included.
Archive
For an example validation case, the following files will be included in
the validation archive.
Unless noted otherwise, all should be plain
ASCII files.
- A README file that: (1) briefly describes the case and the procedure
used to run it; (2) lists and briefly describes all the files in
the archive relevant to the case; and (3) provides the name, postal
address, email address, and phone number of the contact person for
the case.
- The source for the program used to create the mesh, if a user-written
custom mesh generator was used.
- The source for the program used to create the initial restart file.
- The initial restart file, in 32-bit IEEE Fortran unformatted form.
- The NPARC standard input (unit 5) and output (unit 6) files for each
run.
- The mean flow (unit 21) and turbulence model (unit 22) convergence
history files for each run.
These may be in ASCII or 32-bit IEEE
Fortran unformatted form, depending on the input parameters L2PLOT
and IFXPLT.
- The plot3d xyz and q files for the final run, in 32-bit IEEE Fortran
unformatted form.
- Any other relevant input/output files, such as the input data file
for boundary conditions 30-32 (unit 3), or the output mass flow
convergence file for boundary conditions 90-96 (unit 11).
- A file or files containing any experimental data and/or other
well-accepted computational or analytical results, in tabular form,
that were used for comparison with the NPARC results.
- The documentation, in PostScript form.
Check Case
Definition
Check cases will be established to judge the functionality of a newly
installed and/or modified code.
These will be developed, maintained,
and documented in conjunction with the Development Team.
The primary
intent of the check cases is to provide the Development team with a tool
to ensure the integrity of all mechanical aspects of code operation.
At
least one of these cases will be an installation check case that may be
used by new recipients of the NPARC code to verify that the code has
been properly installed on their computer system.
Documentation
Each check case is also documented as part of the validation archive.
Since these cases are intended primarily for use by the Development
Team, they do not require as much detailed documentation as the model
and example cases.
The documentation for check cases will be provided
with the NPARC code as part of the NPARC Developer's Guide and/or as a
separate document.
It will also be available as part of the validation
archive.
The documentation for a check case should include the following
elements:
- Software/Hardware Description
The version(s) of NPARC used to run the case should be stated, as
well as the computer hardware and operating system(s).
Any other software/hardware used, such as a mesh generation package,
should also be described.
For parallel operation, the PVM version that was
used should also be stated.
- Problem Description
The problem being run should be described, although not necessarily
in as much detail as the model and example cases.
If the same or
similar NPARC results have been presented elsewhere, references to
those publications should be included.
- NPARC Input File
A listing of the NPARC standard input file used for the check case
(i.e., the information read from Fortran unit 5) should be included
in the documentation.
When a case consists of a series of separate
computer runs, changes made to the input for subsequent runs should
be noted.
- Convergence History
The convergence history should be described, including the residuals
for the mean flow and turbulence model equations, as well as any
flow-related parameters used to determine convergence.
Ideally, this
will include plots of the convergence parameters as a function of
time step.
- Computed Flow Field Results
A representative sample of the computed results should be shown, and
discussed briefly.
Archive
For a check case, the following files will be included in the validation
archive.
Unless noted otherwise, all should be plain ASCII files.
- A README file that: (1) briefly describes the case and the procedure
used to run it; (2) lists and briefly describes all the files in
the archive relevant to the case; and (3) provides the name, postal
address, email address, and phone number of the contact person for
the case.
- The source for the program used to create the mesh, if a user-written
custom mesh generator was used.
- The source for the program used to create the initial restart file.
- The initial restart file, in 32-bit IEEE Fortran unformatted form.
- The NPARC standard input (unit 5) and output (unit 6) files for each
run.
- The mean flow (unit 21) and turbulence model (unit 22) convergence
history files for each run.
These may be in ASCII or 32-bit IEEE
Fortran unformatted form, depending on the input parameters L2PLOT
and IFXPLT.
- The plot3d xyz and q files for the final run, in 32-bit IEEE Fortran
unformatted form.
- Any other relevant input/output files, such as the input data file
for boundary conditions 30-32 (unit 3), or the output mass flow
convergence file for boundary conditions 90-96 (unit 11).
- The documentation.
Last updated 16 Mar 1999