Groq

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News


Groq is an Artificial Intelligence (AI) inference company. It designs the Language Processing Unit, a chip built to generate tokens quickly, and it sells that speed through GroqCloud, an API that application and agent frameworks call. It does not train frontier foundation models, and it does not sell a sentiment-analysis or topic-modeling suite. An earlier version of this page described that other business. That description was wrong and has been replaced. Jonathan Ross, who led early work on Google's Tensor Processing Unit, founded Groq in 2016. The bet was that inference, not training, would become the scarce resource, and that a compiler-scheduled chip with memory on the die could finish a language model's next token sooner than a general GPU. For several years that was a merchant-silicon story. In December 2025 NVIDIA took a non-exclusive license to the design, reported at about $20 billion, and hired Ross, president Sunny Madra, and a large part of the staff. Groq was not bought. The company that remains, still at groq.com, calls itself an inference cloud. The name is one letter from Grok, the chatbot made by SpaceXSI (formerly xAI). They are unrelated. Groq's founder objected to the overlap when Grok launched in 2023. Mixing them up swaps a chip company for a model lab.


Groq's product is latency. The LPU and GroqCloud exist so agents and voice apps spend less time waiting on the next token, and since 2026 that design also sits inside Nvidia's platform while Groq itself sells the cloud.


What the chip is for

A Transformer produces tokens one after another. Training can batch a huge number of examples and keep a GPU busy. Inference, especially a single conversation or one step of an agent, is much harder to batch, and the machine often stalls while weights move in from off-chip memory. Groq's answer was to stop treating inference as a small version of training. The Language Processing Unit, which the research papers called a tensor streaming processor, is one logical core whose functional slices (matrix, vector, on-chip SRAM, and switching) are laid out so a compiler can schedule every move in advance. There is no cache hierarchy deciding, at runtime, where a tensor went. The company has been explicit that the compiler came first. Its own account says chip design did not start until the compiler's architecture existed, and that the same compiler maps a model across one chip or many. Data is meant to stream from chip to chip as if the rack were a single assembly line, which is why Groq talks about determinism: the same program should take the same time every run, because the schedule was fixed before the request arrived. That property is what voice products and tight agent loops actually buy. A benchmark that quotes tokens per second is a consequence of the schedule, not a separate feature. On-chip SRAM is the other half of the bet. Groq's 2025 technical note put LPU on-chip bandwidth upwards of 80 terabytes a second, against roughly 8 terabytes a second for GPU high-bandwidth memory off the package, and argued that skipping the round trip is where the speed comes from. The first-generation GroqChip product brief is a 14 nm part with about 230 megabytes of SRAM, on the order of 750 INT8 TOPS and 188 FP16 TFLOPS, and a few hundred watts. Those figures describe that generation, not the 2026 part. The limit is capacity. A model larger than the SRAM on one chip has to be sliced across many chips, so the famous speed applied first to models that fit the machine. The 2026 design in Nvidia's materials is a different object, usually called Groq 3 or LPX. Reporting from the Rubin-generation launch describes an SRAM decode accelerator, built as a co-processor beside the GPU: the GPU handles the compute-heavy attention work (prefill), and the LPU handles the memory-heavy feed-forward work that dominates decode. Published rack figures, which should be read as vendor specifications rather than an independent lab result, include about 1.2 FP8 petaFLOPS per chip, on the order of 500 megabytes of SRAM and 150 terabytes a second on the chip, and a 256-chip rack near 128 gigabytes and 640 terabytes a second. Nvidia has cited about 3,400 tokens a second for that rack paired with NVL72 systems. In October 2026, trade press also reported that a later LPU would trade some of that SRAM speed for 3D-stacked DRAM, to hold larger models. That shift was reported, not yet a product this page can specify.

GroqCloud, and why agents use it

Most developers never buy a GroqCard. They call GroqCloud, an inference API shaped like OpenAI's: change the base URL, the key, and the model name. Official integrations cover Python and JavaScript and the agent stacks already on this wiki, including LangChain. That is the sense in which this page sits with agent frameworks. Groq does not ship a persistent first-party teammate the way Grok Bot or other labs' agents do. It sells the fast model call those agents make on every step. The catalog is other people's weights. Over 2025 and 2026 it has included Meta Llama models, Qwen, Mixtral, Whisper-class speech models, and OpenAI's open gpt-oss models. A snapshot of the public price page on 24 October 2026 listed gpt-oss 120B at $0.15 per million input tokens and $0.60 per million output tokens, and the 20B model at $0.075 and $0.30. Those rates move. What has stayed stable is the shape of the offer: open or open-weight models, per-token billing, and a speed claim. Artificial Analysis measurements cited in late October 2026 put Llama 3.3 70B near 297 tokens a second on Groq, in a pack of specialized inference clouds rather than alone at an order-of-magnitude lead. Use a current chart for a purchasing decision. A 2024 demo that showed several hundred tokens a second was real for that model and that day. Speed matters more once a product stops being one reply. An agent that plans, calls a tool, and checks its work may run the model dozens of times before a person sees anything. Speech and speech synthesis have the same constraint from the other direction: a pause of even a second breaks a phone call. Groq's pitch, including a 2025 IBM distribution deal Ross described as several times the performance at a fraction of the cost, is that cheaper and faster tokens raise the number of steps a product can afford. The chip does not make a weak model correct. It makes the wait shorter, which is why agent builders and deployment teams meet Groq in the framework layer rather than in a model-release post.

The Nvidia license, and the company that remained

Through 2024 and 2025 Groq was still a chip company that also ran a cloud. An August 2024 round raised $640 million at a reported $2.8 billion valuation. In September 2025 it raised $750 million at about $6.9 billion. Customers who wanted the LPU bought access to Groq's cloud or, less often, on-premises systems. The compiler, the SRAM architecture, and the API were one firm. On 24 December 2025 Nvidia and Groq announced a non-exclusive license to Groq's inference architecture. Reporting put the payment near $20 billion. It was not a purchase of the company, despite headlines that said "acquired." Ross has described the sequence in a March 2026 GTC conversation: Groq had been wiring GPUs and LPUs together, Jensen Huang proposed making that combination available to Nvidia's customers, the agreement came together in about three weeks, and Ross started at Nvidia the next day, 25 December. His public bio after that reads chief software architect at Nvidia, and founder of Groq. Sunny Madra went with him. Accounts of the deal say a large majority of employees moved, with figures around 80 percent. Groq kept its name, its cloud, and enough of the company to keep operating. The valuation then reset, which is what a license-plus-hiring deal does to the firm left behind. In June 2026 Groq raised $650 million, led by Disruptive and Infinitum, to fund the cloud rather than a head-on chip fight. On 17 August 2026 it closed a $350 million round led by Disruptive, with Nvidia planning to participate, at a $3.5 billion valuation, about half the September 2025 mark. The announcement is unambiguous about the new job: Groq is an Nvidia Cloud Partner, and the capital is for clusters of Nvidia accelerated computing, scaling from 54 megawatts toward more than 200 megawatts in 2027. Alex Davis, chief executive of Disruptive, is executive chairman and has been the public voice of that round. Secondary profiles name Adam Winter as chief executive after Ross's departure. On 27 October 2026 Davis said he planned to move the headquarters from the Mountain View area to Dallas. That was a statement of intent, not a completed move.

Date Event What was reported
2016 Founded Jonathan Ross, after the Google TPU work
August 2024 Series D $640 million at about $2.8 billion
September 2025 Growth round $750 million at about $6.9 billion
24 December 2025 Nvidia license Non-exclusive, reported near $20 billion; not an acquisition; Ross and much of the staff join Nvidia
June 2026 Growth capital $650 million for the inference cloud
17 August 2026 New round $350 million at a $3.5 billion valuation; Nvidia Cloud Partner; 54 MW toward 200+ MW in 2027

Groq is not Grok

Grok is a model family and a chat product. Groq is a way to run models, mostly other labs' models, quickly. Grok 4.7 is a checkpoint with a context window, a price per token set by SpaceXSI, and a training story. Groq's token price is rent for the machine that emits the tokens. A LangChain app can call both in one evening and learn nothing about either if the names are swapped. The technical contrast is the same one Ross used for GPUs and LPUs. Training hardware is built to do a great deal of parallel work. An LPU is built so the sequential chain of tokens does not wait on memory. Nvidia's 2026 product uses both: GPU for the part of the network that is compute-bound, LPU for the part that is bandwidth-bound. That is why "Groq versus Nvidia" stopped being a clean rivalry in December 2025. The independent cloud still competes for inference workloads. The architecture Ross spent a decade on is also a Nvidia co-processor. Readers who arrived here from a Natural Language Processing (NLP) search should expect an API and a chip, not a dashboard for entity extraction. If the task is to choose a model, Groq is the wrong page. If the task is to host a model you have already chosen, and the constraint is tokens per second or time-to-first-token, this is the right one.

Limits

SRAM made the first LPUs fast and kept them small. A model that does not fit the on-chip memory needs a large chip count or a different memory technology, which is the tension in the 2026 reports about stacked DRAM. Published tokens-per-second numbers are not portable across model size, quantization, or year. The 3,400-token figure attached to LPX with NVL72 is a vendor demonstration. The few-hundred-token figures on public leaderboards are a better guide to what an API caller sees, and they should be rechecked. After the license, "Groq" names two supply chains. One is GroqCloud, an independent Nvidia cloud partner scaling power in the tens to hundreds of megawatts. The other is the LPU design inside Nvidia's Rubin-era systems, staffed by people who no longer work at Groq. A paper or a rack photo can belong to either. The non-exclusive license also means Groq did not hand over every right, but it no longer has the founding team that designed the first compiler. Execution risk sits with a rebuilt leadership group and with how much of the old silicon roadmap survived the move. Groq does not fix model quality, prompt injection, or the cost of a long context. It also does not replace training clusters. The August 2026 funding text does say the new capacity will serve training as well as inference, which is a wider business than the original LPU story. Treat that as a cloud strategy, and keep the LPU claim attached to the decode problem it was built for.

Recent Developments

From late 2025 through October 2026 the company split in public. The license and the hirings landed at Christmas 2025. Through the first half of 2026 Ross appeared as an Nvidia executive explaining why GPUs and LPUs should be wired together, including at GTC. Groq the company raised $650 million in June and $350 million in August, took the Nvidia Cloud Partner label, and stated a path from 54 megawatts to more than 200 in 2027. LPX racks were reported in production in August, with Dell among the system makers. October added two items that are easy to over-read. Trade coverage said the next LPU would move from SRAM toward 3D-stacked DRAM to hold larger models. And on 27 October the executive chairman said he intended to put the headquarters in Dallas. Neither is, by itself, a new chip or a new address. The stable fact at the end of the month is the one on groq.com: Groq sells fast inference, now as a cloud built on Nvidia systems as well as on the LPU it pioneered.

References

Additional Viewing

Load video
YouTube

Matthew Berman's March 2024 conversation with Andrew Ling (compiler software) and Igor Arsovski (chief architect) is the right technical companion to the LPU section: how the chip is scheduled, how it is built, and why the speed numbers looked unusual at the time. It predates the Nvidia license, so treat the company boundaries as they were in 2024. Groq compiler and architecture

Load video
YouTube

"Chatbot with RAG, using LangChain, OpenAI, and Groq" (March 2024, running past the 19-minute mark) builds a small retrieval bot and then swaps in Groq to show the latency change. It is a framework tutorial, not a chip talk, which matches how most readers will actually meet the company. API details and free-tier rules from 2024 should not be copied into a current integration. RAG chatbot on Groq

Load video
YouTube

"How to Use FREE AI Models with Groq API" (Code with Muralii, about 11 minutes) is a short console walkthrough: a key, a model name, and a completion call. Useful only as a picture of the developer surface. Pricing and which models are free change constantly. Groq API, short tour

Load video
YouTube

This video was already embedded on the page. Confirm the title and date in YouTube before citing it; it is retained so the existing link is not dropped. Previously listed Groq video

Load video
YouTube

"Compute is the New Oil" (Matthew Berman with Jonathan Ross, 4 April 2024, about 24 minutes) is the founding interview: leaving Google after the TPU, why the compiler preceded the chip, on-chip memory, and why inference speed feeds agents. Ross was still Groq's chief executive. The later Nvidia role is not in this film. Jonathan Ross on founding Groq

Load video
YouTube

"Groq founder and TPU creator Jonathan Ross: GPU and LPU" (NoRush Invest, from Nvidia GTC, 26 March 2026, about 38 minutes) is the post-license primary account. Ross explains the split of attention onto the GPU and feed-forward work onto the LPU, the three-week path to the deal, and why he treats speed as a capability question rather than only a cost question. Use it for the 2026 corporate section, and remember he is speaking as an Nvidia executive who founded Groq. Ross at GTC 2026 on GPU and LPU ```

Retrieved from "https://primo.ai/w/index.php?title=Groq&oldid=53298"