Tuesday, July 21, 2020
Don't Stop Pretraining!
This video explains a study on the benefits of continued pre-training with RoBERTa. Even though RoBERTa is trained on 160GB of uncompressed text from a massive range of sources, the authors show continued gains by continuing pre-training not only in the domain of the downstream task (i.e. massive collections of amazon reviews, news articles, computer science, biomedical research papers), but further gains by doing pre-training (masked language modeling) on the data for the task itself (especially helpful when there is unlabeled data that is better curated for that task than the more broad "domain" of the task). Thanks for watching! Please Subscribe! Paper Links: Don't Stop Pretraining: https://ift.tt/2WEdjdt RoBERTa: https://ift.tt/32SZycF
Subscribe to:
Post Comments (Atom)
Most watched
-
Quick answer: Qwen 3 32B needs about 22.7 GB of VRAM at Q4_K_M with a 4k-token context — a used 24 GB card like the RTX 3090 is the minimu...
-
Using GPUs in TensorFlow, TensorBoard in notebooks, finding new datasets, & more! (#AskTensorFlow) [Collection] In a special live ep...
-
Inside TensorFlow: Summaries and TensorBoard [Collection] Take an inside look into the TensorFlow team’s own internal training sessions-...
-
Quick answer: DeepSeek V4 Flash is a Mixture-of-Experts model: all 284 billion parameters must be stored, so combined RAM plus VRAM — not ...
-
Quick answer: For an 8B model at Q4, local electricity costs roughly $0.05 per million output tokens — against about $0.53 per million ble...
-
Hey everyone This is Ujjwal kapil B.tech 2nd year Ai and Ml In this video we covered one of the most important question "COLLEGE KAB O...
-
Quick answer: For machine-learning work, buy an external SSD over a spinning hard drive, pick USB 3.2 Gen 2 (10 Gbps) or faster, and size f...
-
Quick answer: Yes, for most budget builders: a used RTX 3090 is still the cheapest route to 24 GB of VRAM with full CUDA support. Capacity...
-
This video is a crash course on understanding how finetuning on LLM models can be performed uing QLORA,LORA, Quantization using LLama2, Grad...
-
In this video, you'll learn how to use machine learning, computer vision and deep learning to create a football analysis system. This pr...
No comments:
Post a Comment