User menu

Accès à distance ? S'identifier sur le proxy UCLouvain

FireWorks: A dynamic workflow system designed for high-throughput applications

Primary tabs

Jain, A. [Lawrence Berkely National Laboratory, Berkeley, CA, USA] Ong, S.P. [University of California San Diego, La Jolla, CA, USA] Chen, W. [Lawrence Berkely National Laboratory, Berkeley, CA, USA] Medasani, B. [Computational Research Division, Lawrence Berkeley National Laboratory, CA, USA] Qu, X. [Lawrence Berkely National Laboratory, Berkeley, CA, USA] Petretto, Guido [UCL] Rignanese, Gian-Marco [UCL] Hautier, Geoffroy [UCL] Persson, K.A. [Lawrence Berkely National Laboratory, Berkeley, CA, USA]

This paper introduces FireWorks, a workflow software for running high-throughput calculation workflows atsupercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of compu-tational chemistry and materials science calculations at the National Energy Research Supercomputing Center.It has been designed to serve the demanding high-throughput computing needs of these applications, with ex-tensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) prov-enance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows(i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant toenabling modern data-driven and high-throughput science applications, and we discuss our implementationstrategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data andlimitations of our approach along with planned future work. Copyright © 2015 John Wiley & Sons, Ltd.

Document type Article de périodique (Journal article) – Article de recherche
Access type Accès restreint
Publication date 2015
Language Anglais
Journal information "Concurrency and Computation: Practice & Experience" - Vol. 27, no. 17, p. 5037-5059 (2015)
Peer reviewed yes
Publisher JohnWiley & Sons Ltd. ((United Kingdom) Chichester)
issn 1532-0626
e-issn 1532-0634
Publication status Publié
Affiliation UCL - SST/IMCN/NAPS - Nanoscopic Physics
Links
  1. Bell G Hey T Szalay A Beyond the data deluge Science 80 2009 1297 1298
  2. Raicu Ioan, Foster Ian T., Yong Zhao, Many-task computing for grids and supercomputers, 10.1109/mtags.2008.4777912
  3. Jain Anubhav, Ong Shyue Ping, Hautier Geoffroy, Chen Wei, Richards William Davidson, Dacek Stephen, Cholia Shreyas, Gunter Dan, Skinner David, Ceder Gerbrand, Persson Kristin A., Commentary: The Materials Project: A materials genome approach to accelerating materials innovation, 10.1063/1.4812323
  4. Livny, High Performance Cluster Computing: Architectures and Systems, Volume 1 (1999)
  5. FireWorks workflow software 2014 http://www.github.com/materialsproject/fireworks
  6. FireWorks documentation 2014 http://pythonhosted.org/FireWorks/
  7. Perez Fernando, Granger Brian E., Hunter John D., Python: An Ecosystem for Scientific Computing, 10.1109/mcse.2010.119
  8. Python Scripting for Computational Science, ISBN:9783540739159, 10.1007/978-3-540-73916-6
  9. Oliphant Travis E., Python for Scientific Computing, 10.1109/mcse.2007.58
  10. MongoDB http://www.mongodb.org
  11. Oinn T., Addis M., Ferris J., Marvin D., Senger M., Greenwood M., Carver T., Glover K., Pocock M. R., Wipat A., Li P., Taverna: a tool for the composition and enactment of bioinformatics workflows, 10.1093/bioinformatics/bth361
  12. Deelman Ewa, Singh Gurmeet, Su Mei-Hui, Blythe James, Gil Yolanda, Kesselman Carl, Mehta Gaurang, Vahi Karan, Berriman G. Bruce, Good John, Laity Anastasia, Jacob Joseph C., Katz Daniel S., Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems, 10.1155/2005/128026
  13. Taylor Ian, Shields Matthew, Wang Ian, Harrison Andrew, Visual Grid Workflow in Triana, 10.1007/s10723-005-9007-3
  14. Talia Domenico, Workflow Systems for Science: Concepts and Tools, 10.1155/2013/404525
  15. Ludäscher Bertram, Altintas Ilkay, Berkley Chad, Higgins Dan, Jaeger Efrat, Jones Matthew, Lee Edward A., Tao Jing, Zhao Yang, Scientific workflow management and the Kepler system, 10.1002/cpe.994
  16. Thain Douglas, Tannenbaum Todd, Livny Miron, Distributed computing in practice: the Condor experience, 10.1002/cpe.938
  17. Curcin V., Ghanem M., Scientific workflow systems - can one size fit all?, 10.1109/cibec.2008.4786077
  18. Couvares Peter, Kosar Tevfik, Roy Alain, Weber Jeff, Wenger Kent, Workflow Management in Condor, Workflows for e-Science ISBN:9781846285196 p.357-375, 10.1007/978-1-84628-757-2_22
  19. Wilde Michael, Hategan Mihael, Wozniak Justin M., Clifford Ben, Katz Daniel S., Foster Ian, Swift: A language for distributed parallel scripting, 10.1016/j.parco.2011.05.005
  20. Cholia S Skinner D Boverhof J NEWT: A RESTful service for building High Performance Computing web applications Gateway Computing Environments Workshop GCE 2010
  21. Fielding, Architectural Styles and the Design of Network-based Software Architectures (2000)
  22. Tolosana-Calasanz, Concurr. Comput. Pract. Exp., 22, 617 (2010)
  23. Flask web framework http://flask.pocoo.org
  24. Gunter D Cholia S Jain A Kocher M Persson K Ramakrishnan L Ong SP Ceder G Community accessible datastore of high-throughput calculations : experiences from the Materials Project MTAGS 2012
  25. Tada Tomofumi, Takemoto Seiji, Matsuishi Satoru, Hosono Hideo, High-Throughput ab Initio Screening for Two-Dimensional Electride Materials, 10.1021/ic501362b
  26. Rustad J. R., Density functional calculations of the enthalpies of formation of rare-earth orthophosphates, 10.2138/am.2012.3948
  27. Ong Shyue Ping, Cholia Shreyas, Jain Anubhav, Brafman Miriam, Gunter Dan, Ceder Gerbrand, Persson Kristin A., The Materials Application Programming Interface (API): A simple, flexible and efficient API for materials data based on REpresentational State Transfer (REST) principles, 10.1016/j.commatsci.2014.10.037
  28. de Jong M Chen W Angsten T Jain A Notestine R Gamst A Sluiter M Ande C Zwaag S Curtarolo S Toher C Plata JJ Ceder G Persson KA Asta M Charting the complete elastic properties of inorganic crystalline compounds 2015
  29. Van Noorden Richard, The rechargeable revolution: A better battery, 10.1038/507026a
  30. Qu X Jain A Rajput NN Cheng L Zhang Y Ong SP Brafman M Maginn E Curtiss LA Persson KA The electrolyte genome project: a Big data approach in battery materials discovery in-submission
  31. Cheng Lei, Assary Rajeev S., Qu Xiaohui, Jain Anubhav, Ong Shyue Ping, Rajput Nav Nidhi, Persson Kristin, Curtiss Larry A., Accelerating Electrolyte Discovery for Energy Storage with High-Throughput Screening, 10.1021/jz502319n
  32. Ong Shyue Ping, Richards William Davidson, Jain Anubhav, Hautier Geoffroy, Kocher Michael, Cholia Shreyas, Gunter Dan, Chevrier Vincent L., Persson Kristin A., Ceder Gerbrand, Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis, 10.1016/j.commatsci.2012.10.028
  33. Gonze X., Amadon B., Anglade P.-M., Beuken J.-M., Bottin F., Boulanger P., Bruneval F., Caliste D., Caracas R., Côté M., Deutsch T., Genovese L., Ghosez Ph., Giantomassi M., Goedecker S., Hamann D.R., Hermet P., Jollet F., Jomard G., Leroux S., Mancini M., Mazevet S., Oliveira M.J.T., Onida G., Pouillon Y., Rangel T., Rignanese G.-M., Sangalli D., Shaltaf R., Torrent M., Verstraete M.J., Zerah G., Zwanziger J.W., ABINIT: First-principles approach to material and nanosystem properties, 10.1016/j.cpc.2009.07.007
  34. Couvares Peter, Kosar Tevfik, Roy Alain, Weber Jeff, Wenger Kent, Workflow Management in Condor, Workflows for e-Science ISBN:9781846285196 p.357-375, 10.1007/978-1-84628-757-2_22
  35. Altintas I Berkley C Jaeger E Jones M Ludäscher B Mock S Kepler : an extensible system for design and execution of scientific workflows Proceedings of the 16th International Conference on Scientific and Statistical Database Management 2004 423 424
  36. Bui P Yu L Thain D Weaver: integrating distributed computing abstractions into scientific workflows using python Proc 19th ACM Int Symp High Perform Distrib Comput HPDC 10 2010 636 643
  37. Armstrong, Integrating Task Parallelism into the Python Programming Language (2011)
  38. Goecks Jeremy, Nekrutenko Anton, Taylor James, Galaxy Team The, Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences, 10.1186/gb-2010-11-8-r86
  39. Berthold MR Cebron N Dill F Gabriel TR Kötter T Meinl T Ohl P Sieb C Thiel K Wiswedel B KNIME: the Konstanz information miner Studies in Classification Data Analysis and Knowledge Organization GfKL 2007 2007
  40. Perez, IPython: A System for Interactive Scientific Computing (2007)
  41. Hagberg, Exploring Network Structure, Dynamics, and Function Using NetworkX (2008)
  42. Barga Roger, Jackson Jared, Araujo Nelson, Guo Dean, Gautam Nitin, Simmhan Yogesh, The Trident Scientific Workflow Workbench, 10.1109/escience.2008.126
  43. Foster Ian, Kesselman Carl, Globus: a Metacomputing Infrastructure Toolkit, 10.1177/109434209701100205
  44. Hoheisel Andreas, User tools and languages for graph-based Grid workflows, 10.1002/cpe.1002
  45. Raman R Livny M Solomon M Matchmaking: distributed resource management for high throughput computing HPDC 98 Proceedings of the 7th IEEE International Symposium on High Performance Distributed Computing 1998 140
  46. Shao Yihan, Molnar Laszlo Fusti, Jung Yousung, Kussmann Jörg, Ochsenfeld Christian, Brown Shawn T., Gilbert Andrew T.B., Slipchenko Lyudmila V., Levchenko Sergey V., O’Neill Darragh P., DiStasio Jr Robert A., Lochan Rohini C., Wang Tao, Beran Gregory J.O., Besley Nicholas A., Herbert John M., Yeh Lin Ching, Van Voorhis Troy, Hung Chien Siu, Sodt Alex, Steele Ryan P., Rassolov Vitaly A., Maslen Paul E., Korambath Prakashan P., Adamson Ross D., Austin Brian, Baker Jon, Byrd Edward F. C., Dachsel Holger, Doerksen Robert J., Dreuw Andreas, Dunietz Barry D., Dutoi Anthony D., Furlani Thomas R., Gwaltney Steven R., Heyden Andreas, Hirata So, Hsu Chao-Ping, Kedziora Gary, Khalliulin Rustam Z., Klunzinger Phil, Lee Aaron M., Lee Michael S., Liang WanZhen, Lotan Itay, Nair Nikhil, Peters Baron, Proynov Emil I., Pieniazek Piotr A., Min Rhee Young, Ritchie Jim, Rosta Edina, David Sherrill C., Simmonett Andrew C., Subotnik Joseph E., Lee Woodcock III H., Zhang Weimin, Bell Alexis T., Chakraborty Arup K., Chipman Daniel M., Keil Frerich J., Warshel Arieh, Hehre Warren J., Schaefer III Henry F., Kong Jing, Krylov Anna I., Gill Peter M. W., Head-Gordon Martin, Advances in methods and algorithms in a modern quantum chemistry program package, 10.1039/b517914a
Bibliographic reference Jain, A. ; Ong, S.P. ; Chen, W. ; Medasani, B. ; Qu, X. ; et. al. FireWorks: A dynamic workflow system designed for high-throughput applications. In: Concurrency and Computation: Practice & Experience, Vol. 27, no. 17, p. 5037-5059 (2015)
Permanent URL http://hdl.handle.net/2078.1/162431