NA-ASC-500-13 Issue 23
The Meisner Minute
Guest editorial by Bill Archer, LANL ASC Program Director (acting)
Los Alamos National Laboratory
Next-generation Technology Impacts on the ASC Program
Recently I’ve had reasons to review the goals of ASCI from 1995. There were two related goals: develop a 100-teraFLOP/s computer and develop 3D weapon codes to run on it. The program succeeded at both of these goals by 2007. What we then found was that there was a long way to go for a practical 3D capability, in both hardware and software. Even with Cielo, a 1.3-petaFLOP/s system, certain high-resolution 3D simulations are still beyond our reach.
The 3D simulations on Cielo have taught us that the 100 terabytes of memory available on half the system is not enough, and that I/O and archiving of 25-terabytes files is a problem. We also learned that node failures during a run must be managed to ensure throughput. These lessons have been incorporated into the mission needs of the next ASC system, Trinity, which is currently being procured by the Alliance for Computing at Extreme Scale (ACES) Los Alamos/Sandia partnership.
At this time all we know for sure about Trinity is that it will not be a conventional cluster like Cielo.
The ASC program has two next-generation technology machines, Roadrunner, the world’s first 1-petaFLOP/s system and Sequoia, the first 20-petaFLOP/s system. Trinity is likely to incorporate features similar to both machines. The weapon codes will feel the full impact of these technology changes.
As we move forward to Trinity the weapon codes and supporting models will require a significant amount of re-work to be able to efficiently use the next generation of hardware. At the same time we have to continue supporting the Stockpile Stewardship Program, especially the ongoing and planned Lifetime Extension Programs (LEP). This will require shifting the program from supporting assessments to supporting primary reuse, design and certification. And, we have to continue developing an improved understanding of both primary and secondary physics to improve our predictive capability. Obviously, doing all of this at once will challenge the program.
Don’t think that getting onto the next-generation systems is just an effort for the code teams, or even just the Integrated Codes (IC) program element; it will require the combined effort of the entire program to successfully execute this. Both physics models and data access routines will need to be modified by the Physics & Engineering Models (PEM). The reworked codes will have to be evaluated by Verification & Validation (V&V) program element. A new programming model will have to be provided by Computational and Systems Environments (CSSE), as well as resiliency solutions. Facility Operations and User Support (FOUS) will have to provide the machines, but also the supporting infrastructure for cooling and power, and new ways to handle large data sets for visualization, storage, and archiving. Moving our focus beyond stockpile assessment to design and certification will be equally challenging for the entire program.
The next few years are going to be both exciting and challenging for the ASC program. They will be very similar to 1995-2005 when both computing technology and stockpile stewardship were changing direction. We are at a time when new ideas must be tried, and everyone in the ASC program has to work together to build the future.
ACES and NERSC form partnership for next-generation supercomputers, an unprecedented collaboration between NNSA and the Office of Science
The Alliance for Computing at Extreme Scale (ACES), a collaboration among Sandia (SNL) and Los Alamos (LANL) national laboratories and the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory (LBNL) have fully integrated as a team as they work towards their respective next-generation supercomputer deployments. The goal of the new alliance is to acquire two supercomputers, Trinity and NERSC-8, using a common set of technical requirements and a single procurement process. In addition to strengthening the partnership between NNSA and the DOE Office of Science (DOE/SC) in the area of high performance computing, there are a number of advantages to combining the acquisition strategy. It leverages shared technical and risk management expertise and saves industry time and money in responding to a single request for proposal and a single set of requirements. There are also benefits associated with shared experiences in production operations.
Trinity will be the next-generation supercomputer to provide high performance computational resources in support of NNSA’s Stockpile Stewardship Program mission. Driving Trinity’s architectural requirements is the need to increase ASC predictive capabilities. Cielo and Sequoia (the current machines at LANL and Lawrence Livermore National Laboratory (LLNL) are unable to support the higher fidelity models, both in geometry and physics, needed to provide necessary predictions.
NERSC-8 will increase DOE’s ability to support rapidly increasing computational demands of the entire spectrum of DOE/SC computational research. It will also transition DOE scientific applications to more energy-efficient architectures. The NERSC-8 deployment will support well-established fields that already rely on large-scale simulation, but are moving to incorporate additional physical processes and higher resolution. It will also support scientific discovery performed using a very large number of individual, mutually independent compute tasks.
The ACES and NERSC team will issue a formal Request for Proposals in the third quarter of calendar year 2013, and plans to have contracts in place by the end of the year. Deployments are planned for late 2015 to early 2016.
Cielo Working at Full Tilt for Stockpile Stewardship
The ASC Program’s capability-class workhorse, Cielo, concluded its third capability computing campaign (CCC) on January 27, 2013. Cielo performed at approximately 80% overall utilization of the machine for CCC-3, which began on May 22, 2012. These campaigns support simulations for Los Alamos, Lawrence Livermore, and Sandia national laboratories.
As these campaigns go forward, there is a large and growing demand for large-scale, up to full-system jobs. Cielo is operated by the New Mexico Alliance for Computing at Extreme Scale (ACES), a partnership between Sandia and Los Alamos national laboratories. It is the petascale resource for conducting NNSA weapons simulations in the 2011–2015 timeframe. Built by Cray, Inc., it is installed at Los Alamos National Laboratory.
After a system outage for upgrading the Cray Linux Environment, CCC-4 started on February 4. CCC-4 has initiated an impressive workload for stockpile stewardship and weapons science. Initial requests oversubscribed the time available by a factor of more than five.
Numerous campaigns are increasing the job size to utilize the full-system, or near full-system, capability. Three users will use 102 thousand cores for their jobs: 3D weapons studies, a safety study, and 3D particle transport. Co-design scaling studies by Sandia National Laboratories will use the full system.
Results from Cielo’s Capability Computing Campaigns
In addition to unclassified projects, there were numerous classified stockpile projects done by weapons program scientists. The results from these projects help to understand and accurately calculate nuclear tests that had anomalous results. These results are important to validate our understanding of the physics relevant to nuclear weapons. Simulations such as these began with the inception of the ASC Program in 1995 (formerly the ASCI Program), but the calculations have needed larger machines than those available at the time. For example, LANL researcher Bob Weaver ran a particular weapon simulation on the Blue Mountain machine, which had to be run as a “cutoff,” or truncated, problem because the calculation was too big to run on the machine. In addition, the level of resolution was below that desired, due to the limitations of the machine size. Since that time, he has run versions of the problem to utilize these ASC machines as much as possible: White, Roadrunner, and now Cielo. With Cielo, he is now able to run full-scale 3D simulations with remarkable fidelity, but he is still not able to run the complete simulation. It will have to wait until the next-generation supercomputer Trinity is deployed at LANL.
The need for running full-scale 3D simulations with increased fidelity is the design basis for the Trinity platform. The following five projects with unclassified results from the CCC3 tri-labs are highlighted below.
1. Integrated Simulation of Laser Plasma Interaction (LPI) in NIF Experiments (Steve Langer, PI; Code pF3D) — This project begins an investigation of how to account for backscattered light in a self-consistent manner in simulations of ignition experiments at the National Ignition Facility (NIF). The pF3D code is used on Cielo to simulate backscatter in overlapping beams, and interesting temporal and spatial structure is seen. The Cielo computer was recently used to simulate three interacting quads. The spatial variability of the backscattered light shows an enhancement from interacting quads. The runs on CCC-3 do a better job of simulating backscatter in regions where quads overlap.
2. Informing Predictive Models Using 3-D Simulations of Localization and Initiation in Polycrystals and HE (Nathan Barton, PI; ALE3D and mdef [crystal mechanics]) — This project enabled exploration of phase transformation and plasticity kinetics. To better model strength and phase dynamics of materials, they ran simulations to provide details that are not experimentally accessible at the rates and pressures needed. The results illuminate interplay between nucleation and growth and among interacting mechanisms. The results contribute to ASC Physics and Engineering Models milestones.
3. First Principles Diffusion Coefficients of DT-Pu (Christopher Ticknor, PI; ofmd.f90) — This work explores first principles calculations of the diffusion coefficients for mass transport in a deuterium-tritium (DT) plutonium warm dense plasma.
4. Mitigating Stimulated Raman Scattering (SRS) in Inertial Confinement Fusion (ICF) Hohlraums; Lin Yin, PI; VPIC — This work studies SRS, a type of laser-plasma instability of concern for laser-driven fusion experiments such as those underway at the (NIF).
5. Towards Accurate Simulation of the Abnormal Thermal Fire Environment for Stockpile Stewardship (Paul Lin, PI; Sierra/Fuego) — The goal for this project is safety and reliability of the stockpile for the abnormal thermal fire environment. It is a B61 LEP requirement to accurately predict the abnormal thermal fire environment.
Cielo was deployed for stockpile stewardship, and it has excelled as a stable and reliable platform for running capability computing campaigns for the tri-lab weapons community, particularly for problems that cannot be resolved at less than near-petascale size. Cielo’s architecture allowed easy migration of the existing integrated weapons codes, thus extending the ability to conduct critical simulations while the code teams work to adapt the codes to the changing computing architectures. The examples shown in this article illustrate that Cielo is excelling at the job for which it was intended.
Sandia, Los Alamos and NERSC Host Joint Mini-App Deep Dive with NVIDIA’s Developer Technologies Group
Hardware and software experts from NVIDIA’s Developer Technologies Group recently visited Sandia for a deep-dive on mini-application performance and porting activities. Attendees from ACES and NERSC learned the very latest developments in NVIDIA’s hardware solutions and approaches to optimizing codes for high-performance GPU architectures. NVIDIA also presented several optimization activities where they are using key ASC and ASCR mini-applications to inform future hardware designs. This gave the laboratories the opportunity to learn how modifications to the algorithms or programming models will improve future application performance.
The visit by NVIDIA builds upon a strong and growing collaboration between ACES and NERSC as the laboratories increasingly focus on the opportunities for co-designing future Exascale systems. The laboratories also continue active engagement with leading industry vendors to identify future application characteristics and requirements to enable hardware designers to improve performance, lower power requirements and increase machine reliability. Follow-up discussions from the NVIDIA visit are also helping to refine requirements for future directive-based programming models, a key facet of utilizing existing investment in legacy application codes, as well as programmer tools such as mathematics libraries and runtime systems.
New High-Order Hydrodynamics Algorithms Pave the Way for Future Computing Architectures
Through research funded at Lawrence Livermore, scientists have developed BLAST, a high-order finite element hydrodynamics research code that improves the accuracy of simulations and provides a viable path to extreme parallel computing and exascale architectures.
Scaling a computational physics code to extreme levels of parallelism on supercomputers such as Sequoia, which has over 1.5 million compute cores, is a major challenge. The computational science community widely recognizes that high-order numerical methods will be much better suited to existing and future heterogenous computing architectures, and the need for such methods will increase over the next decade. It is essential to invest in not only hardware development but also numerical methods research to develop novel algorithms designed from the beginning to take advantage of both current extreme parallel machines such as Sequoia, as well as new architectures.
High-order finite element methods use additional degrees of freedom per computational element (or zone) to increase the accuracy and robustness of simulations relative to low-order methods, which have historically been used. For example, in the image below, a very high-order calculation of a multi-material shock hydrodynamics problem is shown using Q8–Q7 finite elements (eighth-order polynomials for the kinematic fields, seventh-order polynomials for the thermodynamic fields). The high-order finite elements result in highly curved zones and sub-zonal resolution of the shock waves, which is simply not possible with a low-order method.
BLAST also provides a high performance computing advantage since high-order methods have greater FLOP/byte ratios, meaning that more time is spent on floating point operations relative to memory transfer, an important characteristic of numerical algorithms for exascale computing. Strong scaling data for the BLAST code on the Sequoia machine is shown below. In strong scaling, a single large problem is solved on more and more processors. Ideally, the time to solution should decrease as the number of processors is increased. In practice, there is a limit to how far an algorithm can be scaled as parallel communication overhead can dominate the run time of a problem as more and more processors are used. Because high-order methods are more compute intensive at the zone level, the communication overhead is not as significant as it is for low-order algorithms; thus, it is possible to achieve excellent strong scaling results down to very few computational zones per processor. “I don’t know of any other code that has such good strong scaling,” said BLAST Principal Investigator Tzanio Kolev. “And I think we can push it even further.”
Strong scaling data is shown for three cases: Q2-Q1 (quadratic), Q3-Q2 (cubic), and Q4-Q3 (quartic) finite element basis functions.
For more information about BLAST, visit https://computation.llnl.gov/casc/blast/index.php.
New ALE3D Release Means More Efficient Simulations
The ALE3D code team, a multidisciplinary team at Lawrence Livermore, recently released version 4.18, which added significant enhancements and functionality. ALE3D, a multiphysics numerical simulation software tool that uses arbitrary Lagrangian–Eulerian (ALE) techniques, is used to simulate many physical phenomena, including geological materials, rocket and munitions fragment impact, concrete fracture, and heat transfer. The code addresses 2D and 3D physics and engineering problems, using a hybrid finite element and finite volume formulation on an unstructured grid. The added enhancements and functionality support the Lab’s progress and performance in predictive capability for assessment.
The new version provides Lagrangian element erosion and automatic contact, with improved parallelism, robustness, accuracy, and coupling, which allow newly exposed surfaces to interact. These enhancements improve the modeling of complex systems, including material failure and fracture. For the longer time scales required by cook-off simulations, an implicit mortar slide capability was added for the structural components, while the incompressible flow
module was augmented to give users the ability to simulate cook-off of high explosives with low melt temperatures. Simulations using ALE capabilities will benefit from using a new condition number relaxer, producing higher quality meshes and leading to more robust and efficient simulations.
The code team also made several improvements that affect problem setup and initiation. Support for CUBIT (a geometry and mesh-generation toolkit produced at Sandia) has been expanded to include meshing of shell elements and the ability to shape geometry into 2D meshes. The ParticlePack library now includes several types of spherical particle defects, new statistical capabilities, and new particle/enclosure options and size-distribution options. The DSD (detonation shock dynamics) package supports a new dynamic capability for initializing explosive burn times.
Usability is improved by the addition of “for” loops in the user input and by initial support for dynamically linked user-defined functions. The flexibility of user controls is also improved. In addition, the code provides a common input syntax for element erosion, slide deletion, and nodal relaxation. A set of “rules” is available that can be combined to give the user very specific control over code behavior.
Code scaling on Sequoia is progressing. For this effort, the code team developed and implemented a new mesh-initialization algorithm and improved the memory control in Smooth-Particle Hydrodynamics. Preliminary results show two-orders-of-magnitude improvements in scaling.
Sandia Fracture Challenge Results Presented at 2012 ASME Congress Indicate Fracture Models Are Not Yet Predictive
Mid-summer last year, Sandia National Laboratories, in partnership with the US National Science Foundation (NSF) and Naval Surface Warfare Center, launched the Sandia Fracture Challenge. The goal of the Challenge was to benchmark the prediction capabilities for ductile fracture, including physics models, computational methods and numerical approaches currently available in the computational fracture community.
The challenge given to engineering researchers was to predict the onset and propagation of quasi-static ductile fracture of a common engineering alloy in a simple geometry (Figure 1) using modeling and simulation. Twenty-four international research teams initially signed up to participate; however, only 14 teams submitted their predictions before the deadline.
The 14 successful teams presented their methodology and predicted results at a special symposium during the American Society of Mechanical Engineers (ASME) International Mechanical Engineering Congress and Exposition in Houston, TX, November 9-15, 2012.
The methods for fracture prediction taken by the teams ranged from very simple engineering calculations to complicated multi-scale simulations. The wide variation in modeling results presented at the symposium (Figure 2) indicated computational fracture models existing in the mechanics community today are not yet predictive. In addition, predicting ductile failure initiation and crack propagation remains an extremely difficult problem.
Soon after the ASME symposium, discussions began within the research community about how to isolate the shortcomings in computational models and narrow the gaps in predictive capabilities for ductile fracture. A follow-up workshop, hosted by NSF, is in planning stages. It will document the lessons learned from the Sandia Fracture Challenge and define coordinated future R&D activities.
Mantevo 1.0 Release, Co-design Via Mini-Apps and the Launch of Mantevo.org Portal
Sandia researchers, along with colleagues from LLNL, LANL, the Atomic Weapons Establishment, NVIDIA, Intel and AMD, announced the first official release of the Mantevo suite of mini applications and mini drivers on December 14, 2012. The community portal (mantevo.org) for accessing Mantevo packages and related capabilities launched simultaneously.
Mantevo packages are small programs that embody one or more performance impacting elements of large-scale production applications. Mantevo 1.0 includes eight packages, including seven miniapplications: CloverLeaf, CoMD, HPCCG, MiniFE, MiniGhost, MiniMD, and MiniXyce, and one minidriver: EpetraBenchmarkTest.
While Mantevo miniapps have been around for several years, energy efficiency concerns are driving change at all levels of computing. Co-design has become an essential activity in order to jointly answer questions about how memory, processor, operating system, programming model and application designs can advance simultaneously. Miniapps have emerged as a critical central component in the co-design process, representing concrete, yet malleable, proxies for large-scale applications, enabling rapid exploration of a very complex design space.
The collection of Mantevo reference implementations continues to grow. The base implementation of each Mantevo miniapp includes OpenMP, MPI and sequential execution. As a part of our advanced systems testbed efforts, many other models are also supported, including AVX, CUDA, OpenACC, OpenCL, Intel TBB, qthreads and KokkosArray.
For more information, or to download these packages, visit the following links.
Livermore Computing Collaboration Tools Increase Productivity
Collaborating in Livermore Computing's (LC's) high performance computing (HPC) environment has become easier over the past year with the rollout of three Web-based tools at lc.llnl.gov. Confluence is a wiki, a Web site that allows users to easily create and share content using only their Web browser. JIRA is an issue-tracking system that allows users to track tasks and software bugs and also provides powerful project management capabilities. Stash is a source code hosting system (based on git) that allows users to store their code and to collaborate on projects through online code review.
“In a matrix company like LLNL, there are many small, multidisciplinary projects, and they often span divisions or departments,” said Todd Gamblin, who started the sites with ASC funding as a pilot program to enable HPC research and development teams to work together more effectively. Todd continued, “Web-based collaboration tools can help these teams work together, but the overhead of maintaining them is often too large for a small team. Strict security requirements make it difficult for LLNL employees to set up one-off Web servers.” Now, LC staff and users can quickly create wiki spaces, issue trackers, and source code repositories for their HPC-related projects. The login is the standard LC user name, and fine-grained access controls and groups make it easy to stay within LC security guidelines.
The impact of the tools has been far-reaching. Within LC, Confluence was used to rapidly produce and maintain documentation for the Sequoia supercomputer, and it has since been used on many other LC projects for documentation and brainstorming. Pam Hamilton, Software Development group leader, fronted an effort to use JIRA to improve communication between and among LC groups. Users from Los Alamos and Sandia National Laboratories can log into the system using cross-site credentials. This capability allows LC's TOSS operating system and Lustre file system teams to use JIRA across the tri-labs.
Since going live in January 2012, the lc.llnl.gov tools have grown to serve more than 560 HPC users. There are more than 120 wiki sites, 88 JIRA projects, and 64 Stash repositories. LC HPC users can start their own collaborative projects by visiting https://lc.llnl.gov/. There may be institutional tools in the future for those without LC accounts.
Traditionally, collaboration tools were seen as outside the scope of HPC, and LC did not support them on its platforms. This perception is changing.
“We see the advantage to the ASC program of a common system for the HPC community to share content, track issues, and work together on code,” said Kim Cupps, LC division leader. “We appreciate the work of Todd, Pam, and others in leading the way to establish this valuable set of tools, which have already had a strong, positive impact on LC users.”
LANL ASC Projects Showcased in Stockpile Stewardship Quarterly
Since February 2011, the Office of Stockpile Stewardship, NA-11, has published the “Stockpile Stewardship Quarterly” (SSQ). Four ASC researchers at LANL are featured in the February 2013 issue. Alek Zubelewicz in the Physics and Chemistry of Materials Group and coauthor Abigail Hunter in the Lagrangian Codes Group at Los Alamos contributed the article titled “Fracture Model for Beryllium and Other Materials.” Another LANL author, Kim Molvig, contributed the article titled “Knudsen Layer Reduction of Fusion Reactivity.” Molvig in the Verification and Analysis Group and coauthor Nelson Hoffman is in the Plasma Theory and Applications Group.
These articles can be viewed on the LANL public website at http://www.lanl.gov/asc under Documents.
Dealing with Data Overload in the Scientific Realm
Excerpted from Science & Technology Review.
We may think we have problems managing our ever-increasing stream of electronic personal “data,” whether that information comes in the form of e-mails, social network updates, or phone texts. However, the challenges we face are miniscule compared with those faced by scientists who must parse the growing flood of scientific data critical to their work. From sequences to simulations to sensors, modern scientific inquiry is awash in electronic information. At Lawrence Livermore, computer scientists supporting various projects and programs are developing methods and algorithms that provide new ways to tame, control, and understand these large amounts of data.
“Four key steps are involved in solving data-science problems,” explains Dean Williams of Livermore’s Global Security Principal Directorate. “One must organize the data—arrange the numbers and bits into meaningful concepts. One must prioritize—choose the most useful data when time and resources are limited. One must analyze—find meaning in images, graphs, data streams, and so on. Finally, it’s important to make it easy for researchers to use the data; that is, we must create the methods and systems that help users query, retrieve, and visualize.”
Three “V’s” can sum up the type of data and the challenges involved: the variety, the velocity, and the volume. For example, in biological mission areas, the variety is high, but the velocity is low, with the volume that needs to be manipulated ranging from gigabytes to terabytes. By contrast, in the cyber security arena, variety and velocity are high, and the volume, which is continually changing as data streams past, can be very large. For climate research, the variety and velocity of data are also high, with an enormous volume accumulating in databases worldwide (from petabytes to exabytes).
The Laboratory, with its broad expertise in analysis, experience in storage technologies, and strong institutional computing culture, is addressing the data-science challenges of its programs and projects. These efforts range from devising methods for predicting the evolution of viruses, to creating tools for tackling streaming data in the cyber security arena, to fashioning accessible intuitive portals and analytics for climate scientists worldwide.
For the complete article, see the January issue of Science & Technology Review: https://str.llnl.gov/january-2013/williams
Computation/Engineering Team Provides Expertise to DOE Carbon Capture
A joint team of Computation and Engineering staff contributed to the first software release for the Carbon Capture Simulation Initiative (CCSI) last fall. The release was a full year ahead of schedule, and the high quality of the tools has elicited praise from both industry partners and the Department of Energy (DOE).
In a February 2010 memorandum, President Obama charged a Carbon Capture and Storage (CCS) Task Force to overcome the barriers to widespread, cost-effective deployment of CCS technology within 10 years. To meet the President’s goal, DOE initiated the CCSI project to accelerate the realization of new carbon capture approaches from the laboratory to the power plant.
James Wood, from DOE headquarters, commended the quality of the initial software release: “The usefulness of the tools was demonstrated in a breakout session [at the CCSI Industry Advisory Board meeting] during which industry representatives began to voice plans for immediate use of these tools. Success in [this] goal has significant uses in many industries that use complex systems and offers a potential to reduce costs, risks, and the enormous time-lag presently required to bring commercial products to market.”
The release is the result of a productive multi-lab collaboration. The software includes uncertainty quantification (UQ) tools, carbon capture simulation models, risk analysis tools, a reduced-order model development tool, and the Turbine Gateway, which allows thousands of UQ simulations to run on the Amazon EC2 cloud simultaneously. Lab employees were involved with the development of many of these tools.
The Lab is a partner in CCSI with other national laboratories, industry, and academic institutions that use state-of-the-art computational modeling and simulation tools to accelerate the commercialization of carbon capture technologies from discovery to development, demonstration, and ultimately to the widespread deployment to hundreds of power plants.
The multi-lab CCSI team still has a significant amount of work to accomplish. Although the team has proven the viability of the simulation approach for improving carbon capture, they must now productize their tools, develop more detailed simulations, and develop dynamic simulations of the full power cycle of a power plant. The team also must broaden the simulations to include the solvents that industry partners are most likely to deploy in the near future.
Livermore Shares Seismic High Performance Computing Expertise with Academia
Lab researchers and university scientists gathered recently at a Lawrence Livermore-hosted workshop for “Advancing Seismology and Geodynamics through High Performance Computing.”
The workshop's goal was to create a strategy for advancing the state of the art in these fields by partnering the National Science Foundation (NSF) and academia with the high performance computing (HPC) capabilities of the Department of Energy.
Funded mostly by NSF, the November session brought to Livermore many of the nation's academic leaders in computational seismology and geodynamics, including the directors of the Southern California Earthquake Center; NSF Center on Computational Infrastructure for Geodynamics at the University of California, Davis; and the Western Region Earthquake Center, U.S. Geological Survey. According to Steve Bohlen, deputy program director, Energy and Environmental Security, who organized the event, rapid advancement of important Laboratory programs in nuclear explosion monitoring and nuclear reactor seismic safety relies in part on the development of more refined seismic wave propagation codes and whole-Earth models, both of which are also needed by academic researchers to improve researchers' understanding of the Earth's interior.
Discussions during the workshop focused on some of these academic needs. A tour of Livermore's computing facility showcased the HPC modeling and simulation capabilities that can be shared.
The workshop ended with an outline for a white paper that NSF will use to promote partnerships between academia and DOE. It will present a vision for major advances in seismology and geodynamics that can be enabled with the focused support of an HPC "ecosystem." The white paper will define key attributes and features of that ecosystem and a concept for a more robust partnership between the academic community and the Lab.
LDRD Funds Early Career Research Projects
LANL’s Laboratory Directed Research & Development (LDRD) organization awarded twelve Fiscal Year 2013 Early Career Research projects of which three awards were for ASC Program researchers. Proposals from Abigail Hunter, Chengkun Huang, and Christopher Ticknor were selected from a total of 27 proposals that were submitted to the competition by researchers across Los Alamos. They will receive up to $225,000 per year for 2 years. “Early career research is one of the most important mechanisms for retaining top-notch scientific talent and aiding in the transition to a staff position,” said Deputy LDRD Program Director Jeanne Robinson.
Abigail Hunter’s proposal “Novel Mesoscale Modeling Approach for Investigating Energetically Driven Nanoscale Defect/Interface Interactions,” was accepted. Abigail is in the Lagrangian Codes Group in LANL’s Computational Physics Division.
Chengkun Huang, in the Applied Mathematics and Plasma Physics Group in the Theoretical Division, will be funded for his proposal “First Principle Study of Relativistic Beam and Plasma Physics Enabled by Enhanced Particle-in-Cell Capability.”
Christopher Ticknor, who first came to the Los Alamos as a Metropolis postdoctoral fellow, works in the Physical and Chemistry of Materials Group in the Theoretical Division. His proposal title is “Stochastic Modeling of Phase Transitions in Strongly Interacting Quantum Systems.”
The Rapid Evolution of Supercomputing Displayed at SC12
If the annual supercomputing (SC) conference proves anything, it is that high performance computing technology changes with stunning speed. But then, the purpose of SC is to galvanize the forces that bring about that change. SC12 keynote speaker, Michio Kaku, a physicist, futurist and author, told conference attendees that "today's supercomputers are tomorrow's infrastructure."
"The rapid evolution of high performance computing should come as no surprise," said Dona Crawford, associate director for Computation at Lawrence Livermore. "The leadership role LLNL plays in supercomputing contributes significantly to the new developments in HPC highlighted at SC."
Sequoia retained its No. 1 Graph 500 ranking, showcasing its ability to conduct analytic calculations or finding the needle in the haystack by traversing 15,363 giga edges per second on a scale of 40 graph (a graph with 2^40 vertices). The system's capability also played a role in one of the conferences most watched competitions, the Gordon Bell Prize. Two of the final five submissions used Sequoia: a simulation of the human heart's electrophysiology using a code called Cardioid, developed by a LLNL/IBM team; and a cosmology simulation led by Argonne National Laboratory.
Sequoia also was selected by readers of HPC Wire, the high performance computing news services, for a 2012 Readers Choice Award. Michel McCoy, head of LLNL's Advanced Simulation and Computing Program, received the award from Tom Tabor, publisher of HPC Wire.
Lawrence Livermore Computation Scientists Tapped to Lead Significant External Committees
Jeff Hittinger (left) has been named co-chair of the Exascale Math Steering Committee for the Office of Advanced Scientific Computing Research (ASCR) within the Office of Science. Jeff will gather input from the applied math community and will then help organize a workshop to discuss future directions for applied math research as it relates to exascale computing architecture.
Martin Schulz (right) was tapped to chair the Message Passing Interface (MPI) Forum, a large, open committee with representatives from many organizations that define and maintain the MPI standard. Among Martin’s responsibilities will be overseeing the MPI 3.1/4.0 effort, which seeks to identify and document additions to the MPI standard that are needed for better platform and application support.
Dona Crawford Named HPCwire "Person to Watch" in 2013
Dona Crawford, Associate Director for Computation at Lawrence Livermore, has been named one of HPCwire’s People to Watch in 2013. This is the second time Dona has been given this recognition; she was the first woman to be named to the list in 2002, and she is the only woman who has made the list twice. The 12 people who HPCwire selects each year are among the most talented and outstanding individuals within HPC. Recipients are selected from a pool of potential candidates in academia, government, industrial end-user, and vendor communities. Final selections are made following discussions with the HPCwire editorial and publishing teams and with input garnered from colleagues and other luminaries across the HPC community, including nominations and guidance from past recipients.
Maya Gokhale's “Streams-C C-to-FPGA Compiler” Paper Is Honored
A paper by Maya Gokhale and her team, entitled “Stream-Oriented FPGA Computing in the Streams-C High-Level Language” and originally published in 2000, was recently selected as one of the 25 highest impact papers in the 20-year history of the IEEE Symposium on Field-Programmable Custom Computing Machines.
In Feb 2013, Maya was named a “Distinguished Member of the Technical Staff” at Lawrence Livermore.
To read the paper, visit
ASC Salutes Jeremy Templeton
The Verification and Validation (V&V) subprogram of ASC is pretty important. After all, we want to be highly confident in the predictions we make about the safety, security, and reliability of the weapons in the stockpile.
Sandia’s engineering researcher Jeremy Templeton wants to provide confidence in the safety of nuclear weapons systems in abnormal thermal environments.
State-of-the-art uncertainty quantification theory brings that confidence.
Quantifying margins and uncertainties analysis begins by working with systems engineers to understand the relevant safety themes and environments of concern. Analysts then obtain as much information as possible regarding the weapon’s design. This includes information such as the materials used and the knowns and unknowns about parts configuration. “All this information is used to build a realistic model for the heat transport physics throughout the system,” said Templeton.
After quantifying algorithm accuracy through rigorous verification methods, assessment of the impact of physical uncertainties on system response begins. “This helps us learn which parameters are important to determining the behavior of the systems and which are not, helping us make the best use of limited experimental and computational resources,” he said.
These results provide the basis for validation testing. Predictivity of the model is assessed quantitatively to confirm the assumptions made in the modeling process, followed by tens of thousands of targeted simulations. These simulations show how the system meets the defined safety requirements probabilistically.
Templeton’s work, and that of his team, in this area has inspired an LDRD project in a seemingly unlikely area – calibrating the engineering models of gas turbine engines. The idea is to inform models using high fidelity data and use computationally efficient methods to impact the engine design cycle, leading to increased efficiency and reduced emissions.
“I also work on developing multi-scale and multi-physics coupling methods for atomistic and continuum descriptions of physical phenomena,” he said. “The goal is to understand the fundamental behavior of electrical energy storage systems at a molecular level.”
Templeton has worked with the Sandia portion of the ASC program for five years. He came to Sandia after completing his Ph.D. in Mechanical Engineering at Stanford University. His thesis was in the area of turbulent flows, developing new wall models for large-eddy simulation (LES) using optimal control theory. “We demonstrated orders-of-magnitude computational savings in some canonical flows with minimal reduction in accuracy as compared to LES without wall models,” he said.
“I enjoy working at the intersection of physics, statistics, and computation,” said Templeton. “The constant process of learning and improvement makes work a lot of fun. I particularly like having the opportunity to think deeply about hard problems, such as how to assess very rare events, and then putting those ideas into practice to solve a real problem.”
His contributions at Sandia are noteworthy. “Jeremy and the teams he’s led have brought an outstanding level of rigor to the verification and validation of full systems thermal analysis for the W87,” said Greg Wagner, manager of the Thermal/Fluid Science and Engineering Department. “He’s pulled in some of the most up-to-date techniques in uncertainty quantification theory and used them to have impact on a very applied problem, setting an example that can be followed by other analysis teams across the labs.”
Wagner further observed, “I’m always impressed with Jeremy’s range of knowledge, his creativity, and his willingness to share his expertise. When I want help thinking through a tough problem, to talk about ideas for new projects, or to get some technical coaching or mentorship for a less experience staff member, Jeremy is always the first person I turn to.”
Wagner’s comments are quite a statement about the value Templeton brings to the ASC program, and the confidence we can have in our predictions about the safety of nuclear weapons systems in abnormal thermal environments.
ASC Relevant Research
Lawrence Livermore National Laboratory
Los Alamos National Laboratory
2012 Citations for Publications (previously not listed)
Sandia National Laboratories
Citations for FY13, Q13
Key: DOI = Digital Object Identifier; URL prefix of DOI is: http://dx.doi.org/
CORRECTIONS TO PRIOR SUBMITTALS
Submitted FY13 Q1
O’Hern, T., Shelden, B., Romero, L., Torczynski, J. (2012). “Stably Levitated Large Bubbles in Vertically Vibrating Liquids,” Bulletin of the American Physical Society, 65th Annual Meeting of the APS Division of Fluid Dynamics, Vol. 57, No. 17. Abstract ID: L11.00005. SAND2012-5184 A.
O’Hern, T., Shelden, B., Romero, L., Torczynski, J. (2012). “Stably Levitated Large Bubbles in Vertically Vibrating Liquids,” Bulletin of the American Physical Society, 65th Annual Meeting of the APS Division of Fluid Dynamics, Vol. 57, No. 17. Abstract ID: L11.00005. SAND2012-6261 A.