Funnel Transformer Explained!

This video explains the Funnel-Transformer, a new technique to compress the sequence length of intermediate layers of a Transformer. This optimization allows the researchers to reinvest the saved computation into more layers and wider hidden representations. This outperforms ELECTRA, XLNet, and RoBERTa in the base and large scale experiment settings! Thanks for watching! Please Subscribe! Links: Funnel-Transformer: https://ift.tt/37h5a2W U-Net: https://ift.tt/2t6v58H The Illustrated Transformer: https://ift.tt/2NLJXmf A Survey of Long-Term Context in Transformers: https://ift.tt/38TWkam ELECTRA: https://ift.tt/2RZsM5S