> ML research was a rather known quantity, or the separate "Neural Engine" CPU explicitly aimed at existing ML pipelines wouldn't exist.
No, not really. Transformers were just one of the possible directions. Silicon design does not have the same time scale than software. Now, everyone is using transformers so it becomes harder to do anything else, and it’s been the case long enough that hardware had some time to align (but is still lagging). But who’s to say that a different architecture published last year won’t take the world by storm 2 years from now?
It’s easy to say it in hindsight, but transformers took a bit of effort to get where they are now.
> Silicon design does not have the same time scale than software.
"Neural Engine" has been a part of iPhones since 2017. So, in development since at least 2013, possibly earlier. And it targeted the rather well established, known, and widely used ML practices.
GPT-like models didn't become even remotely useful until at least 5 years later.
No, not really. Transformers were just one of the possible directions. Silicon design does not have the same time scale than software. Now, everyone is using transformers so it becomes harder to do anything else, and it’s been the case long enough that hardware had some time to align (but is still lagging). But who’s to say that a different architecture published last year won’t take the world by storm 2 years from now?
It’s easy to say it in hindsight, but transformers took a bit of effort to get where they are now.