The world of Language Models (LM) is very fascinating

Artificial Intelligence in Natural Language and Processing Following the release of BERT by Google

[{"selector":"#anim-49fc9bbf-b192-44f2-b885-00a4d823a743 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-21.37276768343392%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2500,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] Image credit:-Google

Training GPT-2 (released February 2019) on 40GB of text data

[{"selector":"#anim-fd3e5565-0e06-450b-b3b3-ad60d3f880d7 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-62.485892938522205%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2500,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] Image credit:-Google

Many of the NLP tasks can be performed by GPT-3 without any gradients, parameter updates or fine tuning.

[{"selector":"#anim-81fbfd06-f8b8-45de-bf4d-efcd86ebff93 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-34.531249906136225%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2500,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] Image credit:-Google

Differentiate fake news generated by GPT-3 from real news

[{"selector":"#anim-9cd69a45-8249-48c2-8b27-31d0df12aa73 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-30.312499880537015%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2500,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] Image credit:-Google

In a few-shots setting, it achieved an accuracy score of 86.4%!

[{"selector":"#anim-7c18a05f-8707-454e-bb4c-892a247663f5 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-21.874999829338588%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2500,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] Image credit:-Google

Fine-tuning can produce performance outside the distribution

[{"selector":"#anim-8386de69-f6f3-491f-adaa-eb6b53ead309 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-62.91488231051995%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2500,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] Image credit:-Google

GPT-3 does not require transfer learning

[{"selector":"#anim-f62b0675-d56c-4519-aec4-b47ba8f435f2 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-5.676123922799128%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2500,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] Image credit:-Google