Question

H3 - a new generative language models that outperforms GPT-Neo-2.7B with only *2* attention layers! In H3, the researchers replace attention with a new layer based on state space models (SSMs). With the right modifications, it can outperform transformers. Also has no fixed context length.

Answer 1

Outperforms across the board? Or only on a few datasets?

Answer 2

Given my line of work, I feel I should understand the title, but im not.

It's kind of depressing from time to time

Answer 3

This is very exciting. Gonna read the paper now, but I haven't really heard of state space models (apart from in the context of graphical networks), and now seeing this lab having a few papers prior on them. What does everyone think of this SSM architecture?

Answer 4

Poor ChatGPT is going to lose its job. #AIWriterFeels

Answer 5

Isn't that simply an RNN?

Answer 6

This is exactly what IM looking for if anyone here would be interested in partnering up i have a new server coming Friday this week let me know if we can discuss employment opportunities!