Is this the deal that launched a million blog posts? It certainly feels like a million and, initially, I didn’t want to make it a million and one, but … this Substack is passionately interested in computer architecture and, for reasons that I’ll explain later, this feels like a significant moment in the history of computer architecture.
There was lots of speculation about the reason for what looked like a very expensive deal.
Did Nvidia do the deal just to end a possible challenge to its ‘GPU supremacy’ from a promising start-up? I was sceptical, saying:
What is the future of Groq’s technology?
… As we’ve seen Jensen Huang has stated that Nvidia plans “ … to integrate Groq’s low-latency processors into the NVIDIA AI factory architecture …”. I don’t see any reason why he would make this up in an internal email.
Austin Lyons has a great post explaining how Groq’s technology fills a gap for Nvidia:
Let’s zoom out a bit and take a look at why adding an ultra-low-latency SKU to Nvidia’s portfolio is the right strategy … the portfolio coverage that a hyperspeed SKU gives Nvidia is important.
So I think we need to take Huang at face value. Nvidia really does plan to ‘ramp-up’ deployment of Groq-derived processors as part of its AI factory architecture.
And talking of ‘hyperspeed’ Nvidia and the Groq team aren’t hanging around. Jonathan Ross and key members of the Groq team already have their Nvidia badges:
Something that isn’t mentioned in the Groq press release is Nvidia buying / reselling any chips (or anything else) from what remains of Groq. I think we can take it that it won’t be doing that, because it doesn’t make financial sense and / or because Nvidia wants to quickly develop its own upgraded version.
In the last few days there have been some significant developments that help to support this thesis. Last Friday the Wall Street Journal reported that Nvidia plans to launch a new design based on Groq’s technology at its GTC conference in March:
Nvidia plans to unveil a new processor specially tailored to help OpenAI and other customers build faster, more efficient tools, a major shake-up to its business that is poised to reset the AI race.
The company is designing a new system for “inference” computing, a form of processing that allows AI models to respond to queries, according to people familiar with the plans. The new platform, set to be revealed at Nvidia’s GTC developer conference in San Jose next month, will incorporate a chip designed by the startup Groq, the people said.
Well, not quite ‘designed by Groq’, but near enough!
Ignore the ‘[unfinished draft]’ label and head to this section which explains the technical background to the post:
… after 6+ years of working on the compiler and running a money-incinerating inference cloud to learn how to make the compiler better… they seem to have figured out something.
Set aside your biases and assume the compiler is functional, or at least capable of becoming functional.
Imagine you are Nvidia and have all the IP needed to make Groq actually reach it’s theoretical full potential.
With the conclusion:
Nvidia has the unique IP needed to fulfil Jonathan Ross’s wildest dreams and truly unlock the insane potential of the worlds most imbalanced computer.
But don’t just read this conclusion, read the whole post which explains the rationale for this conclusion, and subscribe to Irrational Analysis!
For more on the original Groq technology, their presentation at Hot Chips 2022 with the accompanying slides is a good starting point.
There has been a ‘Cambrian explosion’ of inference focused accelerators. Most of these won’t survive for technical or (most often) commercial reasons.
If Nvidia is going to launch a new design based on Groq’s technology at GTC in March then we will know one of the winners of this explosion. This might just be the most important development in computer architecture for many years.
And just to end with an observation that this will mean that it’s just three months from the Groq deal to Nvidia announcing a design based on Groq’s technology. No wonder Jensen didn’t want any regulatory delays from a full takeover: that really is ‘Building Fast’.
Further Reading
For more details of what this might all mean Vikram Sekar has a great update:
Thanks for the shoutout!