gpt3
play

GPT3 - AtishyaJain Thecontent of this presentation has beensourced - PowerPoint PPT Presentation

Mar 19, 2024 •379 likes •1.11k views

GPT3 - AtishyaJain Thecontent of this presentation has beensourced fromvarious youtube videos andblogs apartfromtheoriginal paper Let s Dissectit Let s Dissectit Let s Dissectit Mergeaa Mergeaa Mergeab Mergeaa Mergeab MergeZY Let

  1. GPT3 - AtishyaJain Thecontent of this presentation has beensourced fromvarious youtube videos andblogs apartfromtheoriginal paper

  2. Let ’ s Dissectit

  3. Let ’ s Dissectit

  4. Let ’ s Dissectit

  6. Mergeaa

  7. Mergeaa Mergeab

  8. Mergeaa Mergeab MergeZY

  9. Let ’ s Dissectit

  16. BERT uses GPT uses Encoder part Decoder part only only

  17. Architecture

  18. Architecture

  19. Architecture

  20. Let ’ s Dissectit

  21. 355 Yearson fastestV100 $4,600,000 On lowest GPU cloud provider

  22. Let ’ s understand Few ShotLearning

  23. Zero Shot Learning There is a Dairy Cow

  24. Zero Shot Learning There is a Horse

  25. Zero Shot Learning Zebra is a horse with Dairy Cow ’ s color

  26. Zero Shot Learning You are Dad, Its a better than a Zebra CNN !!

  27. One Shot Learning There is a Monkey

  28. One Shot Learning You are Dad, Its a better than a Monkey CNN !!

  29. Few Shot Learning There is a Dog

  30. Few Shot Learning There is another Dog

  31. Few Shot Learning You are Dad, Its a better than a Dog CNN !!

  32. Few Shot Learning

  33. Few Shot Learning

  34. Few Shot Learning

  35. Compute Power

  36. Transformer Variants

  37. Training Dataset

  38. Training Dataset - Filtering

  39. Training Dataset - Filtering - Fuzzy Deduplication

  40. Training Dataset - Filtering - Fuzzy Deduplication - Adding high quality dataset

  41. Training Dataset - Filtering - Fuzzy Deduplication - Adding high quality dataset - Overlapping Test Set

  42. Evaluations

  43. Language Modelling - SOTA on PTB - Omit the 4 Wikipedia-related tasks and one-billion word benchmark

  44. LAMBDA

  45. TriviaQA

  46. Translation

  47. Synthetic and Qualitative Tasks - Arithmetic - Word Scrambling and Manipulation - SAT Analogies - News Article Generation - Learning and Using NovelWords - Correcting English Grammar

  48. Arithmetic

  49. Word Scramble and Manipulation

  50. News Generation

  51. Limitations - Lowperformanceinsome NLPtasks Starts to lose coherenceoversufficientlylarge passages - Special difficulty with “common sense physics” like “If I putcheese - infridge,will it melt ?” Architecturaldrawbackis doesn’t have bidirectionalinfo and - denoisingobjectives

  52. Limitations - Poor sampleefficiency Ambiguityon fewshot learninglearns task fromscratch ? - Difficult inferencing, hugemodel - Lack of structuredknowledge -

  53. Fairness and Bias

  54. Fairness and Bias Race

  55. Fairness and Bias Race Religion

  56. Demos

  57. GPT3 : Demos

  58. GPT3 : Interaction with your own AR bot https://twitter.com/i/status/1294380308209508359

  59. GPT3 : Animate Your Maths From English https://twitter.com/i/status/1294652394739912704

  60. GPT3 : Building aWebsite https://youtu.be/LOhIS7kiKvM

  61. GPT3 : Context BasedDictionary https://twitter.com/i/status/1294631853224206339

  62. GPT3 : Describe YourDesign

  63. W eaknesses - Fails miserably on reasoning tasks, so in essence, GPT-3 is not a very good reasoning module at all (Vipul) - No saturation yet (Vipul)

  64. W eaknesses - Fails miserably on reasoning tasks, so in essence, GPT-3 is not a very good reasoning module at all (Vipul) - No saturation yet (Vipul) - In zero-shot or one-shot, choice of words for task description in context learning can introduce variance (Shantanu) - Limited context window of 2048 (Shantanu)

  65. E xtensions - A bidirectional model with similar size and experiments (Vipul, Shantanu)

  66. E xtensions - A bidirectional model with similar size and experiments (Vipul, Shantanu) - Explainable few-shot learning and analysis to see if GTP-3 is actually learning (Vipul, Shantanu)

  67. E xtensions - A bidirectional model with similar size and experiments (Vipul, Shantanu) - Explainable few-shot learning and analysis to see if GTP-3 is actually learning (Vipul, Shantanu) - A distilled version of GPT-3 (Shantanu)

  68. E xtensions - A bidirectional model with similar size and experiments (Vipul, Shantanu) - Explainable few-shot learning and analysis to see if GTP-3 is actually learning (Vipul, Shantanu) - A distilled version of GPT-3 (Shantanu) - Limited context window of 2048 (Shantanu)

  69. E xtensions - A bidirectional model with similar size and experiments (Vipul, Shantanu) - Explainable few-shot learning and analysis to see if GTP-3 is actually learning (Vipul, Shantanu) - A distilled version of GPT-3 (Shantanu) - Limited context window of 2048 (Shantanu) - Adversarial experiments to tweak the training samples articulately and present the adversarial examples to it at test time for inference. (Vipul)

  70. Thankyou

  71. R eferences - https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a - https://www.youtube.com/watch?v=SY5PvZrJhLE - https://jalammar.github.io/how-gpt3-works-visualizations-animations/ - https://www.youtube.com/watch?v=8psgEDhT1MM&vl=en - https://www.youtube.com/watch?v=7qPDwsCLbZc&t=3959s - Language Models are Few-Shot Learners (Brown et. al) - https://www.youtube.com/watch?v=Mq97CF02sRY

Recommend

More recommend