IBM's 12nm chip runs an 8-billion-parameter model on 30 kilowatts, and you cannot buy it

Everything about the AI hardware race points one way: a newer process node, more high-bandwidth memory, more power per rack. IBM Research has spent a decade building the counter-argument. NorthPole is an inference chip made on a 12nm process, three generations behind the 5nm node IBM uses for the accelerator it actually sells, and in the comparisons IBM has published it beats the graphics cards it was measured against by tens of times on energy. The catch sits in the same sentence as the achievement: you cannot buy one.

The idea is easier to describe than to build. A graphics card keeps a model's weights in memory beside the processor and hauls them across to the compute units for every inference. NorthPole does not. Every weight and every intermediate result sits in memory on the chip itself, and the only things that cross the chip's boundary are the input going in and the answer coming out.

That is what the silicon is arranged around. The chip carries 22 billion transistors on a 795 square millimetre die, divided into 256 cores in a 16 by 16 array, each with its own compute, memory, control and network connections. There are 224MB of memory on the chip: 192MB spread across the cores to hold weights and intermediate tensors, and a 32MB framebuffer to stage what goes in and out. Internally it moves data at 13 terabytes a second. At a nominal 400MHz it does more than 200 trillion operations a second at 8-bit precision, twice that at 4-bit and four times at 2-bit.

Fetching a number from memory that is not on the chip costs energy the arithmetic itself does not, and at inference time that fetching is most of what a processor spends its time doing. Remove it and the power budget changes shape rather than degree. That is why a chip on an old, cheap node can embarrass newer ones. The node governs how much you can build and how fast it switches; it does not decide how far the data has to travel.

The first public numbers came in Science in October 2023, on image recognition. Running ResNet-50, IBM measured NorthPole as 25 times more energy efficient than common 12nm graphics cards and 14nm processors, counting frames interpreted per joule, while also beating them on latency and on frames per second per billion transistors. Against IBM's own earlier brain-inspired chip, TrueNorth, it was about 4,000 times faster.

Language models came next. In September 2024 IBM reported a 3-billion-parameter model derived from its Granite-8B-Code-Base running with latency under a millisecond per token, 46.9 times faster than the most energy-efficient graphics card in its comparison. Sixteen cards in an ordinary 2U server produced 28,356 tokens a second while using 72.7 times less energy than the lowest-latency card tested. IBM did not name the cards, and neither will we. Dharmendra Modha, the IBM Fellow who has led this work since it was a research project about imitating animal brains, put the claim plainly: "What is essential here is qualitative orders of magnitude in improvement."

Last November the team published the version that matters to anyone running a data centre: a whole rack. Two hundred and eighty-eight NorthPole cards across 18 standard 2U servers, delivering 115 peta-ops at 4-bit and 3.7 petabytes a second of memory bandwidth, drawing 30 kilowatts, weighing 730kg and occupying two-thirds of a square metre. Each card is given a 50-watt envelope. What it conspicuously does not need is the apparatus this generation of AI hardware demands: no liquid cooling, no custom interconnect, no special power delivery. The servers talk to each other over ordinary 200-gigabit Ethernet.

In that configuration the rack runs three simultaneous copies of IBM's open 8-billion-parameter Granite-3.3-8b-instruct at a 2,048-token context, serving 28 users at once with 2.8 milliseconds between tokens for each of them. Reconfigured, the same hardware runs 18 copies of a 3-billion-parameter model, or a single 70-billion-parameter one.

Which is where the limits begin. If every weight must live on the chip, on-chip memory is the entire budget, and 224MB per chip is not much: a model is sliced across many cards, and the largest model this rack can hold at all takes all 288 of them. The weights also have to be squeezed to 4-bit, with 8-bit activations, and that squeezing is not free. The team recovered the lost accuracy with quantisation-aware training, fine-tuning on eight of the very graphics cards the chip is meant to displace, for two weeks and 128,000 steps. It worked: across 19 benchmarks the quantised model averaged 56.8 against the original's 56.4. But that workflow assumes a model you are able to retrain, which is a different world from downloading whatever was released this week.

Then the larger catch. NorthPole is a research prototype, in IBM's own words, and has been one for three years. The accelerator IBM sells is Spyre, which came out of a different lineage inside the same AIU family: a 5nm chip with 32 cores and 25.6 billion transistors on a 75-watt card, generally available since 28 October last year on z17 and LinuxONE 5, and from early December on Power11, in clusters of up to 48 cards on a mainframe. Spyre is the product. NorthPole is the argument.

So is it a disruptor? Not as a product, because it is not one, and the history of interesting inference silicon is mostly a history of chips that never left the lab. As an argument, it has already landed. It says the efficiency ceiling everyone is currently paying for in power contracts is not set by the process node, which is the expensive thing, but by the memory architecture, which is a design decision. That argument gains value every quarter that inference demand grows faster than the grid does. The chip making it is sitting on a 12nm wafer nobody can order.