filesender terabyte challenge
play

FileSender Terabyte Challenge Ren e Klomp Edwin Schaap System and - PowerPoint PPT Presentation

Jan 13, 2023 •196 likes •441 views

FileSender Terabyte Challenge Ren e Klomp Edwin Schaap System and Network Engineering University of Amsterdam February 6, 2013 Ren e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 1 / 24 What is FileSender?

  1. FileSender Terabyte Challenge Ren´ e Klomp Edwin Schaap System and Network Engineering University of Amsterdam February 6, 2013 Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 1 / 24

  2. What is FileSender? Meant to be easy for user and sysadmin No extra software requirements Default setup reaches only 150Mb/s Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 2 / 24

  3. Research Question Can we identify bottlenecks in the FileSender application and how can we improve the transfer speeds by reducing or removing these bottlenecks. Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 3 / 24

  4. Environment Tested and excluded the hardware as the bottleneck (except HDD) Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 4 / 24

  5. The bottleneck 0.0 0.2 0.4 0.6 0.8 1.0 1.2 No delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 50ms delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 100ms delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 150ms delay time (s) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 200ms delay time (s) Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 5 / 24

  6. Client side implementation Remove Gaps JavaScript webworkers Parallelisation Concurrent TCP streams (like GridFTP) Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 6 / 24

  7. Server side implementation Handle out-of-order chunks X-Start-Byte HTTP header File based queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 7 / 24

  8. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 8 / 24

  9. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 9 / 24

  10. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 10 / 24

  11. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 11 / 24

  12. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 12 / 24

  13. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 13 / 24

  14. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 14 / 24

  15. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 15 / 24

  16. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 16 / 24

  17. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 17 / 24

  18. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 18 / 24

  19. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 19 / 24

  20. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 20 / 24

  21. Server side implementation (Example) Destination File Queue Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 21 / 24

  22. SSL performance 6000 5000 4000 Mb/s Depends on cipher 3000 AES-NI 2000 1000 0 AES128 AES256 RC4 3DES CAMELLIA128 CAMELLIA256 Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 22 / 24

  23. Terabyte Challenge Transferred in 3 hours and 19 minutes (@702Mb/s) Harddisk as bottleneck Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 23 / 24

  24. Upload completed... Questions? made possible by Ren´ e Klomp, Edwin Schaap (UvA) FileSender Terabyte Challenge February 6, 2013 24 / 24

Recommend

FileSender FileSender http://www.filesender.org/ J Jan Meijer M ij TF St TF-Storage 7 7

FileSender FileSender http://www.filesender.org/ J Jan Meijer M ij TF St TF-Storage 7 7

FileSender FileSender http://www.filesender.org/ J Jan Meijer M ij TF St TF-Storage 7 7 9+10 September 2010 Poznan value proposition value proposition easy sharing arbitrarily large files through a trusted intermediary through a trusted

401 views • 13 slides
FileSender http://www.filesender.org/ Update: Sept. 2010 Jan. 2011 Jan Meijer TF-Storage 9

FileSender http://www.filesender.org/ Update: Sept. 2010 Jan. 2011 Jan Meijer TF-Storage 9

FileSender http://www.filesender.org/ Update: Sept. 2010 Jan. 2011 Jan Meijer TF-Storage 9 3 February 2011 value proposition easy sharing arbitrarily large files through a trusted intermediary basic function Picture courtesy SRCE

217 views • 18 slides
B a n Parallel DBMS d 1 Terabyte w 1 Terabyte i d Chapter 21, Part A t h Parallelism:

B a n Parallel DBMS d 1 Terabyte w 1 Terabyte i d Chapter 21, Part A t h Parallelism:

Why Parallel Access To Data? At 10 MB/s 1,000 x parallel 1.2 days to scan 1.5 minute to scan. B a n Parallel DBMS d 1 Terabyte w 1 Terabyte i d Chapter 21, Part A t h Parallelism: 10 MB/s Slides by Joe Hellerstein, UCB, with some

509 views • 4 slides
Bandwidth 1 Terabyte 1 Terabyte Parallelism: 10 MB/s divide a big problem into many smaller

Bandwidth 1 Terabyte 1 Terabyte Parallelism: 10 MB/s divide a big problem into many smaller

Parallel DBMS Chapter 22, Part A Slides by Joe Hellerstein, UCB, with some material from Jim Gray, Microsoft Research. See also: http://www.research.microsoft.com/research/BARC/Gray/PDB95.ppt Database Management Systems, 3 nd Edition. Raghu

458 views • 8 slides
Metadata Management of Terabyte Datasets from an IP Backbone Network: Experience and Challenges

Metadata Management of Terabyte Datasets from an IP Backbone Network: Experience and Challenges

Metadata Management of Terabyte Datasets from an IP Backbone Network: Experience and Challenges Sue B. Moon and Timothy Roscoe Sprint Advanced Technology Laboratories 1 Adrian Court Burlingame, CA 94010 { sbmoon,troscoe } @sprintlabs.com

60 views • 5 slides
Miscellaneous Topics in Databases P ARALLEL DBMS W HY P ARALLEL A CCESS T O D ATA ? At 10 MB/s

Miscellaneous Topics in Databases P ARALLEL DBMS W HY P ARALLEL A CCESS T O D ATA ? At 10 MB/s

Topic for Thursday? 1 Miscellaneous Topics in Databases P ARALLEL DBMS W HY P ARALLEL A CCESS T O D ATA ? At 10 MB/s 1,000 x parallel 1.2 days to scan 1.5 minute to scan. 1 Terabyte 1 Terabyte Parallelism: 10 MB/s divide a big

408 views • 22 slides
Post-quantum RSA We built a great, great 1-terabyte RSA wall, and we had the university pay for

Post-quantum RSA We built a great, great 1-terabyte RSA wall, and we had the university pay for

Post-quantum RSA We built a great, great 1-terabyte RSA wall, and we had the university pay for the electricity Daniel J. Bernstein Joint work with: Nadia Heninger Paul Lou Luke Valenta Post-quantum RSA Daniel J. Bernstein, Nadia Heninger,

788 views • 50 slides
STEP CHALLENGE February 7 th March 8 th CHALLENGE OVERVIEW This Step Challenge is a fun

STEP CHALLENGE February 7 th March 8 th CHALLENGE OVERVIEW This Step Challenge is a fun

Marion County Public Schools STEP CHALLENGE February 7 th March 8 th CHALLENGE OVERVIEW This Step Challenge is a fun and effective 30-day walking challenge where your goal is to increase your daily average steps by 25% * ! *If you are not

466 views • 8 slides
Cloud storage at ARNES Andrej Bagon Arnes, p.p. 7, SI - 1001 Ljubljana andrej.bagon@arnes.si

Cloud storage at ARNES Andrej Bagon Arnes, p.p. 7, SI - 1001 Ljubljana andrej.bagon@arnes.si

Cloud storage at ARNES Andrej Bagon Arnes, p.p. 7, SI - 1001 Ljubljana andrej.bagon@arnes.si #10 TF-Storage, Ljubljana 21. 2. 2012 AGENDA FileSender @ ARNES Cloud storage @ ARNES FileSender @ ARNES Still in pilot period

169 views • 13 slides
VAST CHALLENGE 2017 Bianca Barnucz & Stephanie Wegscheidl OVERVIEW VAST Challenge

VAST CHALLENGE 2017 Bianca Barnucz & Stephanie Wegscheidl OVERVIEW VAST Challenge

VAST CHALLENGE 2017 Bianca Barnucz & Stephanie Wegscheidl OVERVIEW VAST Challenge User Challenge 1 Challenge 2 Challenge 3 Grand Challenge Implementation Project Management & Outlook VAST CHALLENGE 2017

270 views • 16 slides
THIS IS WHERE CHANGE BEGINS - Worlds Challenge Challenge February 12, 2018 1 AGENDA

THIS IS WHERE CHANGE BEGINS - Worlds Challenge Challenge February 12, 2018 1 AGENDA

THIS IS WHERE CHANGE BEGINS - Worlds Challenge Challenge February 12, 2018 1 AGENDA Introduction to the SDGs - Rules of the Worlds Challenge Challenge Sustainable Development Mixer 2 Triple Bottom Line An

789 views • 46 slides
www.bpho.org.uk Oxford 24 th June 2014 Physics Challenge AS Challenge A2 Challenge

www.bpho.org.uk Oxford 24 th June 2014 Physics Challenge AS Challenge A2 Challenge

www.bpho.org.uk Oxford 24 th June 2014 Physics Challenge AS Challenge A2 Challenge Experimental Project BPhO Round 1 Round 2 Training Camp IPhO Moreover a physics problem should be difficult in order to entice

720 views • 33 slides
Challenge: InMutaGene https://www.crackit.org.uk/challenge-21- inmutagene Launch Meeting 10

Challenge: InMutaGene https://www.crackit.org.uk/challenge-21- inmutagene Launch Meeting 10

Challenge: InMutaGene https://www.crackit.org.uk/challenge-21- inmutagene Launch Meeting 10 September 2015 The Challenge To develop in vitro / in silico assay(s) that can be used singly or in combination to improve risk assessment for GT

547 views • 17 slides
30 / 40 piece challenge presentation by Lizbeth Atkinson Where did the challenge begin? The idea

30 / 40 piece challenge presentation by Lizbeth Atkinson Where did the challenge begin? The idea

30 / 40 piece challenge presentation by Lizbeth Atkinson Where did the challenge begin? The idea of the 40 piece challenge began when Elissa Milne, an Australian born composer, musician and teacher discovered this statement in a book one day,

281 views • 7 slides
Forensic Challenge V2.0 UNAM-CERT RedIRIS Topics * Forensic Challenge V1.0 * Forensic

Forensic Challenge V2.0 UNAM-CERT RedIRIS Topics * Forensic Challenge V1.0 * Forensic

Forensic Challenge V2.0 UNAM-CERT RedIRIS Topics * Forensic Challenge V1.0 * Forensic Challenge V2.0 * Conclusions Introduction Why this forensic Challenge ? * To promote and impulse the Forensic Analysis in our Region -- Hispanoamerica--

558 views • 19 slides
Promoting Sustainability: A Challenge for All Our biggest challenge in this new century is to

Promoting Sustainability: A Challenge for All Our biggest challenge in this new century is to

Promoting Sustainability: A Challenge for All Our biggest challenge in this new century is to take an idea that seems abstract - sustainable development - and turn it into a daily reality for all the worlds people . Kofi Annan, UN

321 views • 15 slides
Arizona FAF$A Challenge Julie Sainz, M.Ed. Arizona FAF$A Challenge Project Manager Arizona

Arizona FAF$A Challenge Julie Sainz, M.Ed. Arizona FAF$A Challenge Project Manager Arizona

Arizona FAF$A Challenge Julie Sainz, M.Ed. Arizona FAF$A Challenge Project Manager Arizona Commission for Postsecondary Education Agenda Agenda What is the AZ FAF$A Challenge New State Goal Challenge Categories Incentives

666 views • 21 slides
The Challenge Avoiding the next GFC... The Challenge - Producing More with Less

The Challenge Avoiding the next GFC... The Challenge - Producing More with Less

Banking 4 Food Edwin van Raalte The Challenge Avoiding the next GFC... The Challenge - Producing More with Less today 2050 Natural Natural Resources Resources Every minute 158 & Footprint more mouths to feed Of

632 views • 41 slides
Climate-KIC 9 December 2015 t The Climate Challenge and opportunities Challenge Opportunities

Climate-KIC 9 December 2015 t The Climate Challenge and opportunities Challenge Opportunities

Climate-KIC 9 December 2015 t The Climate Challenge and opportunities Challenge Opportunities 2 o C is the guard rail for A unique opportunity: we must manageable climate address the biggest challenge change. facing mankind

489 views • 13 slides
@ International KEYSTONE Challenge Track Conference Challenge Track Koice 11 12 May 2015

@ International KEYSTONE Challenge Track Conference Challenge Track Koice 11 12 May 2015

@ International KEYSTONE Challenge Track Conference Challenge Track Koice 11 12 May 2015 Sanda Martini - Ipi University of Rijeka Challenge Track Any challenge in the scope of KEYSTONE project is welcome. What open

200 views • 19 slides
1. Introduction butterfillS@ceu.hu butterfillS@ceu.hu first challenge second challenge We

1. Introduction butterfillS@ceu.hu butterfillS@ceu.hu first challenge second challenge We

1. Introduction butterfillS@ceu.hu butterfillS@ceu.hu first challenge second challenge We are stuck with our two main ways of describing and explaining things, one which treats objects and events as mindless, and the other which treats

854 views • 82 slides
Challenge on Challenge on 48 Ca enrichment 48 Ca enrichment Separation of Calcium Isotopes with

Challenge on Challenge on 48 Ca enrichment 48 Ca enrichment Separation of Calcium Isotopes with

Sep 19,2005@US-Japan Seminar, Hawaii Challenge on Challenge on 48 Ca enrichment 48 Ca enrichment Separation of Calcium Isotopes with a Crown Ether Pedersen@1962 Cram&Lehn@1987 Ryuta Hazama for the CANDLES Collaboration Dept. Phys.,

360 views • 18 slides
Structure of Presentation Introduction Challenges in the Verification of Challenge 1:

Structure of Presentation Introduction Challenges in the Verification of Challenge 1:

Structure of Presentation Introduction Challenges in the Verification of Challenge 1: Variety of modeling languages Electronic Control Units Challenge 2: Learning curve for Requirement Capture Challenge 3: Integration with

250 views • 11 slides
F-17 TTAC QP The Challenge The F-17 TTAC Challenge Restore Integrity without being able to

F-17 TTAC QP The Challenge The F-17 TTAC Challenge Restore Integrity without being able to

PH IP F 17 WP PCP Stavanger May 26 th , 2011 Thore Bergsaker DP F-17 TTAC QP The Challenge The F-17 TTAC Challenge Restore Integrity without being able to reach communication depth & Restore Production (3000 bopd) 2 F-17 F-17

205 views • 7 slides

More recommend