# AI chip architecture: any biomimetic work in progress? **URL:** \ **Category:** Factual Questions\ **Tags:** ai\ **Created:** [May 24, 2024, 4:51am UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010 "2024-05-24T04:51:13Z")\ **Posts on this page:** 10\ **Page:** 2
**Author:** ![Dr\_Paprika](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr_paprika/32/3042_2.png) [@Dr\_Paprika](https://boards.straightdope.com/u/Dr_Paprika)\ **Post date:** [May 25, 2024, 11:07pm UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/21 "2024-05-25T23:07:15Z")
We probably have not yet identified all the neurotransmitters. We certainly don’t understand what all the brain cells do. Not long ago we thought the most abundant cell just acted as support and glue. Nope. A basic introduction to some recent trends in neurology (from 2022 _Economist Technology Quarterly_). See “View All Chapters”. Might be paywalled. > **[After fallow decades, neuroscience is undergoing a renaissance](https://www.economist.com/technology-quarterly/2022/09/21/after-fallow-decades-neuroscience-is-undergoing-a-renaissance)** > > The toolkit for tackling brain dysfunction is expanding rapidly, says Natasha Loder ---
**Author:** ![md-2000](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@md-2000](https://boards.straightdope.com/u/md-2000)\ **Post date:** [May 26, 2024, 11:12pm UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/22 "2024-05-26T23:12:29Z")
> [@DSeid](#): > > [3 Ways 3D Chip Tech Is Upending Computing - IEEE Spectrum](https://spectrum.ieee.org/amp/amd-3d-stacking-intel-graphcore-2656931957) Not sure how this applies, since IIRC the typical chip factory will process an entire wafer (8" diameter?) at a time and then slice it up. back in the 80’s there was a startup - Trilogy(?) - that planned to put a whole IBM 370 series on a full wafer. I assume however this is done, or how multiple chips are made on a wafer, by exposing one area at a time, so for something like a Trilogy system it would have required interconnects between specific areas of the wafer. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [May 27, 2024, 2:03am UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/23 "2024-05-27T02:03:33Z")
> [@md-2000](#): > > Not sure how this applies, since IIRC the typical chip factory will process an entire wafer (8" diameter?) at a time and then slice it up. IME it is most frequently used to connect multiple independent chips together to maximize the speed between them. It eliminates the overhead of the packaging and pins. This is called Package on Package (PoP). A phone PoP will stack RAM and FLASH on top of the SoC – everything looks like it is in a single chip. The SoC itself will have 12 or more core processors. The main bottleneck in an SoC’s NPU is memory bandwidth (particularly with these huge GenAI models) so it helps with AI performance even if the AI compute is not being stacked. There are other advantages too – the chips can be fabricated with different processes, decouple yields, etc. This is another interesting high-level link that talks about the Through-Silicon-Via (TSV) which is just one of the methods in @DSeid’s link: > **[Through-Silicon-Via (TSV) – Revolution in IC Packaging Technology | Blog Posts...](https://www.lumenci.com/post/through-silicon-via-tsv-revolution-in-ic-packaging-technology)** > > This blog describes the Through-silicon-via technology used in different IC packaging technology, types of TSVs, advantages, and their applications. Read more to understand the TSV fabrication process and the market players in the field of TSV... This isn’t my area of expertise and it changes year-to-year. I’m sure my description is a few years behind. ---
**Author:** ![Francis\_Vaughan](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/francis_vaughan/32/3093_2.png) [@Francis\_Vaughan](https://boards.straightdope.com/u/Francis_Vaughan)\ **Post date:** [May 27, 2024, 8:22am UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/24 "2024-05-27T08:22:28Z")
> [@md-2000](#): > > Not sure how this applies, since IIRC the typical chip factory will process an entire wafer (8" diameter?) at a time and then slice it up. back in the 80’s there was a startup - Trilogy(?) - that planned to put a whole IBM 370 series on a full wafer. Modern, and even legacy fabs are 300mm (aka 12"). It is something of a sore point that they topped out at this, despite lots of hope for ever bigger wafers. 450mm is a bit like gallium arsenide - tomorrows technology, always has been, always will be. I remember that there were lots of problems with attempts at wafer scale integration. Heat control and uneven heating of the wafer was one. One of the big problems is that memory (at least DRAM) uses enough of a different process that it isn’t compatible with logic on the same die. So memory is going to be a different wafer. This messes with new architectures that try to put processing inside the memory. You can use less dense memory, and there are processes that support this. But the density is a lot lower. So it is hard to make designs that make sense. The big dog in fabs is throughput. The eyewateringly expensive bits of kit are the machines that expose the wafers aka _steppers_. In order to maximise throughput you want the stepper to be exposing every die on the wafer with the same mask. There really isn’t any useful way of making a stepper use different masks for different locations. The tolerances are so mindbendingly small they have enough trouble making it work with only one mask. And what constitutes a mask nowadays is not your father’s mask. Maybe you could stream together a set of steppers, but the efficiency would likely still be ruinous. At $100M a throw, and the next generation looking like $250M each, steppers make for very focussed thinking about efficiencies. Modern AI systems, even those that operate as a biomimetic design basically just encode the layout and connections of the bio unit as a data structure and then run code that serially updates the emulated system. You can add parallel processing units, but this is just divide and conquer of the same basic idea. It is no different to how a legion of other simulation systems work. It is efficient because we can make the memory system that contains the system representation ridiculously dense, due to our ability to make DRAM so dense, and how fast we can make the processing units rip through this. But working out how to make silicon look like actual brain processing is a way different matter. It may simply never catch up with brute force emulation. Very optimised emulation is the likely middle ground absent any fabrication breakthroughs. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [May 27, 2024, 9:40am UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/25 "2024-05-27T09:40:01Z")
> [@Francis\_Vaughan](#): > > I remember that there were lots of problems with attempts at wafer scale integration. They actually exist now. > **[Cerebras launches 900,000-core 125 PetaFLOPS wafer-scale processor for AI —...](https://www.tomshardware.com/tech-industry/artificial-intelligence/cerebras-launches-900000-core-125-petaflops-wafer-scale-processor-for-ai-theoretically-equivalent-to-about-62-nvidia-h100-gpus)** > > Cerebras launches WSE-3 wafer-scale processor, CS-3 supercomputer. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [May 27, 2024, 9:59am UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/26 "2024-05-27T09:59:31Z")
Those 3D chips are extremely dis-similar to the 3D structure of a brain. Those just take two or more completely 2D chips and stack them with [data connections](https://en.wikipedia.org/wiki/Through-silicon_via) between them. Those are stacks of pancakes while brains are a bowl of spaghetti. ---
**Author:** ![Francis\_Vaughan](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/francis_vaughan/32/3093_2.png) [@Francis\_Vaughan](https://boards.straightdope.com/u/Francis_Vaughan)\ **Post date:** [May 27, 2024, 2:39pm UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/27 "2024-05-27T14:39:16Z")
> [@Darren\_Garrison](#): > > They actually exist now. That is seriously impressive. One thing is that they appear to be replicating the same core at each spot on the wafer, which wasn’t something that made sense eons ago. That probably helps manage a lot of the difficulties. Would be interesting to know what the yield is. One can be sure that there is lots of redundancy and mechanisms to route around failed bits. Still many interesting questions about the design. The whole shebang only has 44 GB of memory. That may be a limit for some things. 50kB per processor. Hard to know. ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [June 29, 2024, 8:09pm UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/28 "2024-06-29T20:09:12Z")
> [@Francis\_Vaughan](#): > > I remember that there were lots of problems with attempts at wafer scale integration. Heat control and uneven heating of the wafer was one. > > One of the big problems is that memory (at least DRAM) uses enough of a different process that it isn’t compatible with logic on the same die. So memory is going to be a different wafer. This messes with new architectures that try to put processing inside the memory. You can use less dense memory, and there are processes that support this. But the density is a lot lower. So it is hard to make designs that make sense. Trilogy, the company mentioned, was founded by Gene Amdahl who was chief architect of the IBM 360 and the founder of the company that bore his name. I followed this since I sometimes went to meetings about an attempt to do WSI when I was at Bell Labs. It failed and they sold the technology to Alcoa of all people. Trilogy failed also. I’ve been reviewing papers about how to test TSVs for easily 15 years, but it has been in the last 5 that I’ve seen stuff from industry about it. I know wafer yields - I owned this information for the last company I worked for. It’ll never be even close to 100% for any kind of aggressive process. I don’t remember ever seeing a totally yielding wafer. One company from about 15 years ago was making a chip with 100 processors on it, with many spares, just like memories have spare rows and columns. BTW there is large amounts of memory on a processor, in the cache, Looking at the die photo of one chip I worked on shows that about 30% of it is the L3 data cache, shared across six CPU cores. Standalone memories are a lot cheaper to design and fab. A lot of the slowdown is from interconnect, and 3D packaging is designed to cut that down by eliminating the need for buffering as done for chips mounted on boards. ---
**Author:** ![Francis\_Vaughan](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/francis_vaughan/32/3093_2.png) [@Francis\_Vaughan](https://boards.straightdope.com/u/Francis_Vaughan)\ **Post date:** [July 1, 2024, 3:28am UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/29 "2024-07-01T03:28:02Z")
Yeah, I should have been more clear that I was referring to DRAM. Caches are a huge component of designs, but are composed from the same logic as the rest of the processor. I do remember IBM being very proud of a halfway process that put a form of DRAM on the processor die. But that was decades ago and I have no idea what happened to it. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [July 1, 2024, 2:09pm UTC](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010/30 "2024-07-01T14:09:11Z")
> [@Francis\_Vaughan](#): > > I do remember IBM being very proud of a halfway process that put a form of DRAM on the processor die. But that was decades ago and I have no idea what happened to it. Like wafer-scale chips, it’s back, baby! > **[Slide Shows Samsung May be Developing a RISC-V CPU for In-memory AI Chip](https://www.hpcwire.com/2024/06/19/slide-shows-samsung-may-be-developing-a-risc-v-cpu-for-in-memory-ai-chip/)** > > Samsung may have unintentionally revealed its intent to develop a RISC-V CPU, which a presentation slide showed may be used in an AI chip. The company plans to release an \[…\] > **[A Milestone in Computing: 2D In-Memory Processor With Over 1000 Transistors](https://scitechdaily.com/a-milestone-in-computing-2d-in-memory-processor-with-over-1000-transistors/)** > > Developed by EPFL researchers, the first large-scale in-memory processor using 2D semiconductor materials could substantially cut the ICT sector’s energy footprint. As information and communication technologies (ICT) process data, they convert... [https://www.sciencedirect.com/science/article/pii/S2773064622000160](https://www.sciencedirect.com/science/article/pii/S2773064622000160) [Previous page](https://boards.straightdope.com/t/ai-chip-architecture-any-biomimetic-work-in-progress/1002010.md?page=1)