We’re Reaching the Physical Limits of Chips

Welcome to Sourcery.

The potential of photonics.

The technology to interconnect these GPUs.

Global payroll and HR platform.

8 billion operational data points.

Scale-ups over startups.

You'd rather have a million of something than one of anything.

Building the intelligence layer for defense.

There isn't enough power. There's not enough memory chips.

We're at the end of physics of what we can still do.

Okay, amazing. So we're on our final talk for today. Today we have Steven, and we have Adam. Steven, you come from imec. Adam, you're from Strike. So we're going to talk a little bit about what comes after the GPU.

So let's start with you. Could you explain what imec is?

Sure. I think we're a little bit the most unknown hidden gem in semiconductors. At least unknown for the average Joe. Definitely not unknown for the semiconductor world itself. We're the worldwide leader in chips, but in actually the early stages of building a chip.

So hardware is not like software. It's not that you prompt it and suddenly 30 seconds later something comes out of it. No, it's a process of 5 to 10 years. And we're at a very early stage of that process where we have a unique clean room facility that nobody else in the world has.

And so all the biggest chip providers in the world actually first come to us, first in Belgium, to kind of design that next-generation chip that you'll see maybe in 7, 8, to 10 years from now. And going through a whole phase of designing that chip, we hand it over then to them to further commercialize it.

But in that sense, there's almost no chip in the world today that has not been touched by imec in some way. We already exist for 40 years, and we've been driven more or less that chip roadmap for the last 40 years.

So I should ask you the secret question, because you know all the secrets on chips.

And some of them I'm allowed to answer.

What can you answer?

Well, I think we're definitely going to talk more about that. But indeed, we're actually seeing a huge amount of disruptions going on. I think we're now kind of plateauing in what we call Moore's Law. What does that mean?

Chips have been becoming bigger and bigger, or faster and faster over the last years. We always did that through scaling. So what does that mean? It means that we made a transistor smaller and smaller. And we're at the level that a transistor has about the depth of one nanometer.

What is one nanometer? It's about 100,000 times smaller than a human hair. That means that we're at the end of physics of what we can still do. And I think we're going to come into an era where what we do in hardware is going to be needs to be way more inventive than what we've been doing in the last 10 to 20 years to keep on scaling that roadmap of Moore, to make sure that chips become faster and faster, as they already did the last four decades, basically.

Adam, as you look into the chip industry, how do you make sense of it? How do you map it out?

Yeah, I thinkright now GPUs are very good at the training part. And we're now looking at the bottleneck that sits in the inference layer. And the bottlenecks that I look at the most and try and look at companies that attack those bottlenecks sit in power and energy, memory, cooling, and data transfer.

So when we look at the inference side of things, memory, the memory shortage in general, and increasing the bandwidth and lowering the latency of getting the useful memory to a GPU as quick as possible is a really interesting and exciting place that needs to be builded within.

Cooling as well, and where that's going. I think bringing the liquid that takes heat away from the chip closer and closer to the GPU itself is how it's going to progress over the next 5 to 10 years. And I think a lot of the value will go to the manufacturing side of that.

And yeah, I mean, power, everyone knows how much of a constraint and a bottleneck that is. And increasing the capacity there is huge.

I recently interviewed Tony Kim of BlackRock at the Raise AI Summit. It's actually in Paris. And we're talking about hardware. It's like the hardware era, renaissance, everything's shifting to hardware. Software is getting it's taking a bit of a beating.

But the margins, people don't understand. The margins for chips are fantastic, and for hardware versus software, which is getting eaten by AI. And again, like the token spend and that kind of thing. So how do you make sense of the shift in value and how the pyramid kind of flipped?

Well, I think the demand is just so high for the thing that is less fruitful out there,right? So there isn't enough power. There's not enough memory chips. The shortage in memory, I mean, if we look at there's only three companies in the world that produce memory, which is 95% of the market: Micron, SK Hynix, and Samsung.

Two of them are in Korea. And a lot of their shift has been filled out by hyperscalers taking over their whole inventory. So you've seen prices rise by 700% this year just for the memory part of the chip.

And that will happen across the layer because it's the thing that's in the most demand. The training side and the software side of things is continually getting better. But the physical side is the thing that's harder to get better overnight.

Yeah, to be honest, I don't really think it's necessarily flipped. I think it's more the immaturity of the software marketright now. It might seem weird that I say immaturity of the software market, but I think what we've been doing at AI the last four or five years has, of course, in software been tremendous.

But if you look at it from, OK, how complex it is to actually, in terms of computations, do whatever, I mean, large language models can do, it's still very much a brute force approach. And as a result, the value that you can get out of it is pretty limited.

And I think if you're going to get further, we're going to see new types of software evolutions as well, which are tied way more closer to hardware. And because of that, I think the value will increase way bigger.

And I think so we're kind of filling the base of that pyramid already in hardware, of creating a lot of value there. And I think software will follow in that sense.

We're seeing more and more chip companies founded every year, probably than ever before. And venture dollars are definitely helping fund that. Is there a capacity to how many chips can exist in the market?

I thinkright now there's kind of endless demand for that. And I think it's still going to be the question of how that's going to continue. I think it's rather the bottleneck is manufacturing. Manufacturing theright type of components in different regions.

And I think, for example, what Thema Foundries is doing there is super important. But having theright type of components for that is, I think, the real bottleneck, not necessarily the number of companies that do this. Because every company can come with a completely different, unique design.

And in that sense, can also kind of revolutionize what we're going to have as well. And probably in the software world, we're going to see way more different types of algorithms. And as a result, we will also need more diversification in the type of hardware.

So more than just GPUs as well.

As models get bigger and bigger, what breaks first on the chip layer?

Well, yeah, I think for sureright now, memory. Memory is absolutely the number one component that is already broken today. But I don't really think that models will become necessarily bigger and bigger. If you just think about it, the latest open source models like Kimi K3, they already have more parameters than what the human brain has.

But on the other hand, that K3 chip that or the chip that we need to fuel that K3 model is about a million times more or less energy efficient than what the human brain is. And so I think we're going to see a different type of paradigm.

We kind of went into an era the last five years of building bigger and bigger and bigger models. We went from 175 billion parameters to kind of 10 trillion parametersright now. That's going to stop. And there's a lot of companiesright now taking that shift as we speak, abandoning what we call the scaling hypothesis of building bigger and bigger models, and therefore trying to solve those memory bottlenecks as well.

Copper has kind of met its days. I'm curious, how do you think about photonics and how that changes the layer?

Yeah. I mean, feel free to.

Yeah, I mean, I think there are so many benefits to using photonics instead of copper. And copper is easy because it's all in electrical, it's all digital. And the bottleneck with photonics is changing between optical and the electrical layer.

But transferring data more quickly and more efficiently, and not having that latent heat aspect that electrons in copper wires do, will only continue to scale the efficiency of data transfer. And then I think we'll see the architecture continually shift for where photonics will be within data centers.

At the moment, it's between the clusters, and we're using linear pluggable optics. But I think in the next five years, we will see these optical components get closer and closer to the actual GPU. And we will go towards a state of co-packaged optics.

So for me, I mean, every day what I do actually at imec is thinking about what could be AI in the future. So how is AI going to evolve in like 5 to 10 years from now? And I think we're seeing kind of from a software perspective that very much diversifying path of a lot of different approaches, whether it's world models or more reinforcement learning-based approaches.

But what is 100% clear is that whatever scenario in the end will become the dominating model of the future, moving data as fast as possible in the quickest way with the huge amounts of data that we need to shift is always going to be what we need.

Because again, towards the analogy of the human brain, that's what our human brain is also unbelievably good at. We have an unbelievable 3D structure that is able to do this. And so photonics will play, I think for me, especially in the next years, it's going to be decular technology to enable the chips of the future.

I think also on photonics, it's attacking many parts of the bottlenecks that I touched on at the start. It's attacking energy because it can transfer data quicker. It's attacking cooling because you don't need as much cooling around these GPUs if you're transferring through light.

And then it's attacking data transfer at the core. So I think having spanning most of the bottlenecks that we're seeing in the AI infra buildoutright now, it's like the key technology that is likely to win.

As we close out, and this is the last talk of the day, so thank you all very much. What is your hottest takeright now?

I think we're going to 5 to 10 years from now, we're going to laugh at how old-fashioned AI was today. That's a bit my hottest take in the sense that, of course, we went through this explosion of what AI was bringing us.

But I fundamentally don't think that today's technology, that this is what we're going to be talking about in 10 years from now. I think an actual also in software revolution is coming. And for those who are old enough, we still might remember the internet days of a dial-up connection where we had all these bleeping sounds to make connections to the internet.

And we kind of laugh at how that's how we did it in back of those days. I think in 10 years from now, we're going to talk about large language models, which are very much a brute force approach in exactly the same way.

I might be a little bit biased on this one, but I would say it's a very good idea to recruit or invest in people that have had an academic and technical background in Europe and had the wherewithal and business sense in the US, and combining those two mindsets.

A lot of the best founders that I've backed and I've seen have that dichotomy between the two. So I'd say both recruiting and investing, those two traits.

Amazing. Well, Steven, Adam, thank you very much. I think Louis is going to come up next and say something.

Hey, it's Molly. If you enjoy our interviews, check out our newsletter, sourcery.vc, where we deliver a once-a-week top deals and tech headlines email and also go deeper on our podcast interviews. Subscribe to Sourcery today. And don't forget to subscribe to the podcast on YouTube, Spotify, Apple, or wherever you listen.

Link in description to sign up.