Embedded gradient-based optimization algorithms can out perform non-invasive black-box optimization algorithms by large margins in speed, accuracy and robustness in many cases. However, incorporating an embedded optimization algorithm in a complex production application code is very difficult to achieve, especially after a large simulation code already exists. Different heroic efforts to get embedded optimization algorithms into production codes have ultimately failed to achieve any real impact because the interface and capabilities were not maintained as the code was continued to be developed. There are many obstacles to getting embedded optimization into production applications and maintaining the capability (so that I can be used when it is really needed). Production codes may not use sufficiently smooth mathematical formulations or numerical approaches to provide smooth function derivatives. Even if the functions are smooth, the production code may not be able to efficiently compute accurate derivatives (e.g. using an automatic differentiation tool). The structure of the application code may be very inflexible and not allow the exposure of basic model functions and derivative operators or allow control by the embedded optimization algorithm. Related to this, when an embedded optimization algorithm is incorporated into an production application, if it is not intimately integrated into the basic infrastructure of the application, then the “hooks” will be lost through atrophy. Lastly, if the software for the complex embedded optimization algorithms and the production application are not kept integrated on a frequent schedule, then the integrated embedded optimization capability can disappear as the two codes continue to be developed independently.
In this presentation, I will describe experience at Sandia National Labs in trying to incorporate embedded optimization algorithms in the MOOCHO (once called rSQP++) package into several different semi-production application codes. Early failures of the form described above will be mentioned as well as current efforts to address the problems. The core problems are being addressed on all fronts and there are now examples of integrated embedded optimization capabilities that are being maintained over many years, even over long periods where active embedded optimization work is being done. The primary approaches that will be discussed are the concept of Abstract Numerical Algorithms and the Thyra interface layer, automatic differentiation of C++ codes using Trilinos/Sacado, the Trilinos/Thyra ModelEvaluator interface to define the application infrastructure, and formalized frequent software integration strategies to maintain integrated embedded optimization capabilities.