~Alamos LA-UR- Approved for public release; distribution is unlimited. Boo ting Over Infiniband With Perc e us Cluster Management Title. Matt he w Do s anjh , INST-OFF Author(s). William Pi cke tt , IINST-OFF Gr ah am Va n He ule , INST-OFF Ac ade mic Dist ribu ti on Intended for: NA TIO NA L LA BORATORY --- - EST 1943 ---- Los Alamos National Laboratory, an aHirmative action/equal opportunity employer, is operated by the Los Alamos National Security, LLC for the National Nuclear Security Administration of the US Department of Energy under contract DE-AC52-06NA25396. By acceptance of this article. the publisher recognizes that the U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or to allow others to do so, for U.S. Government purposes. '-os Alamos National Laboratory requests that the publisher identify this article as work performed under the auspices of the U.S. Department of Energy. Los Alamos National Laboratory strongly supports academic freedom and a researcher's right to publish; as an institution, however, the Laboratory does not endorse the viewpoint of a publication or guarantee its technical correctness Form 836 (7/06)
Abstracts Booting Over Infmiband With Perceus Cluster Management Ma Uhev,r D osanj h, UNM William Pickett, NMT Graham Van Heule, MTU Abstract: Tw o main network fabric s ar e used in l arge diskless HP clu st er s: Eth ernet is ty pically used for cl uster manageme nt tasks such as boot ing and IB is typically ll se d fod ast da ta c om munication. Conf i guring a cl uster of diskless no de s to boot over IB fa bric using Per c eu s cou ld he lp el imina te t he need for the met in cl usters, r edu cing costs and reducing the number of part· . The mo tivati on behind this pro jec t is a situation currently f ac ing the Co yote super comp uter. It is wired exclus ively with IB and uses a t wo -stage boot pr oces ses; it l oads a small k ernel fro m fl ash m em ory and proceeds to do wnlo ad the rest through lB. Th ose who ma n age the cluster wou ld prefer to mo ve away from flash mem ory, leaving only two viable opt ions: pur chase and install an exp ensive Ethernet ne twork, or co n fi gure the computers to f ul ly boot over lB . To configu re the net work to bo ot ove r IB the IB card s must be upgrade d to li se the g PXE protocol, a ter which the clust er manage me nt software mu st be co nf igured to rec ogn ize and work with the [B cards allo win g f or a diskless boot. The potential implications in clude the eva luation of sca lability in a large cluster, such as Coyote. As IB has higher band width than Et h ern t, clu ste rs w ould gai n mor e co mputing ti me by decreasing boot time. This als leads to potential research of mul ti cas t booting ove r lB.
Booting Over Infiniband With Perceus Cluster Management PRESENTED BY Matthew Dosanjh - UNM William Pickett - NMT Graham Van Heule - MTU On 8/3/2009 LoS Alamos UNCLASSIFIED NATIONA L L AB O RATO RY Slide 1 ---- fST .' 9 .) Operated by Los Alamos National Security, LLC for NNSA
Outline • Motivation • Goals • What We Did • Issues Faced • Future Research • Conclusions ~ Los Alamos UNCLASSIFIED NATIONA L L AB O RATO RY Slide 2 ___ fIT' U] ------ Operated by Los Alamos National Security, LLC for NNSA
Motivation • Coyote • Has no Ethernet network • Uses two stage boot - Stage 1 is a small kernel loaded from local flash memory - Stage 2 is downloaded by stage one over Infiniband • Local flash memory will eventually deteriorate • There exist two solutions • Purchase and install an expensive Ethernet network • Configure the cluster to grab the stage one image over Infiniband. t; Los Alamos UNCLASSIFIED NATI O NA L L AB OR ATO RY Slide 3 ---- ES T." .. l Operated by Los Alamos National Security, LLC for NNSA . "J~ VA. 't
Our Project • Our goal is to get this cluster to boot over Infiniband to determine if it is feasible to do it to a larger cluster in a production environment • Perceus - cluster management software • DHCP - Dynamic Host Configuration Protocal • Infiniband - High bandwidth, low latency network fabric p, L os Alamos UNCLASSIFIED NATIONA L LA B ORATORY Slide 4 ____ C ST "«3 Operated by Los Alamos National Security, LLC for NNSA
Outline • Motivation • Goals • What We Did • Issues Faced • Future Research • Conclusions t; Los Alamos UN C LAS S I FIE D NATION A L L ABORATORY Slide 5 ___ ES T. '9 1111 ------------------------------------------------------- Operated by Los Alamos National Security. LLC for NNSA - • - ~ [')'4}
Steps On The Road To Completion • Created Perceus VNFS image with Infiniband drivers • Burned gPXE into Infiniband card firmware • Added Infiniband drivers to stage 1 image • Patched DHCP to recognize the 32 digit MAC address of Infiniband • Patched Perceus to accept Infiniband MAC addresses Stage 1 VNFS gPXE Image Image LoS Alamos UNCLASSIFIED NAT IO NA L LA80RA T ORY Slide 6 - ___ H T 1 9 43 • . wr-rY!fI Operated by Los Alamos National Security, LLC for NNSA V ... "~
Issues Encountered DHCP doesn't have support for Infiniband at it's current version • When patched for Infiniband DHCP doesn't send the correct MAC address Ethernet MAC: 00:01 :02:03:04:05 Infiniband MAC: 00:01 :02:03:04:05:06:07:08:09: 10: 11 : 12: 13: 14: 15: 16: 17: 18: 19:20 • The default initramfs doesn't contain Infiniband drivers • Kernel is not by default configured to handle Infiniband • Large lack of documentation for Perceus' Infiniband capabilities p, Los Alamos UNCLASSIFIED NAT IO N A L L AB ORA TOR Y Slide 7 ____ £!IT 1 '4 3 Operated by Los Alamos National Security, LLC for NNSA
_~D!fl ~Alamos Outline • Motivation • Goals • What We Did • Issues Faced • Future Research • Conclusions UN C LAS S I FIE D NATIONA L LABORA T ORY Slide 8 ___ rST. 19 .3 ______________________________________________________ _ Operated by Los Alamos National Security. LLC for NNSA ....
~) Ideas For Future Research • Multicast boot over Infiniband may be a quick and efficient solution for a larger cluster • Using iSCSI rather than NFS when booting over Infiniband • Bottleneck research • Doing quantitative comparison of the boot speed of Ethernet and Infiniband Los Alamos UNCLASSIFIED NATION AL LA 80 RATOR Y Slide 9 ___ _ E ST . 194) Operated by Los Alamos National Security, LLC for NNSA
Conclusions • We have successfully booted over Infiniband • However we still have issues getting a unique hardware identifier • It currently can only boot one node. • Further research would be required for large scale deployment (; Los Alamos UNCLASSIFIED N ATIO NA L LA BO RATORY Slide 10 ____ £ST . '9 .) Operated by Los Alamos National Security, LLC for NNSA
Questions ~ Los Alamos UN C LAS S I FIE 0 NATIO NA L LABORATORY ____ ESY 19 U ____________________ __ ________________________________________________________________________________ _ Slide 11 • oW /JI/!'!:'rY4l Operated by Los Alamos National Security, LLC for NNSA _ "f 1i::JlJiE4 -- Vii.
Recommend
More recommend
