You might be off on how much compute is out in the world? Digitimes reported today China thinks it has 197 exaflops installed -
"MIIT revealed in August that China's aggregate computing power reached 197 EFlops, ranked second in the world after the US. However, no details were provided regarding the scale of computing power in the United States that was being referred to. According to the 2022-2023 Global Computing Index jointly published by IDC, IEIT Systems, and Tsinghua Institute for Global Industry, the US and China are the two front runners in terms of computing power.
Source: Ministry of Industry and Information Technology of China, October 2023"
Hi Peter, Thanks for sharing. I'd be very happy - and not too, too surprised - to be proven wrong with a better estimate! I did do a search for any other estimates and didn't find much. I'll follow up this comment and see if I can find any more data. Best wishes.
I think the fact that AI hardware using around 14 Exaflops of A100s (I read Nvidia DGX GH200 is around the corner) compared to 5.5 Exaflops of general purpose computing should take into consideration environmental needs (e.g not overbuilding) when the energy consumption of hardware may become cost-ineffective after a number of generations. That is, general purpose supercomputers could have a longer utility repurposed for other tasks, even if their original purpose if no longer practical. AI, on the other hand, is like an ASIC- its FP64 might be useful for a number of tasks, but these GPUs might not have much other utility, and might be expensive to run 20 yrs from now.
I research the history of technology, and the rate of development within AI. I have been thinking about the computational advantages human minds still have over AI systems. By one estimate (https://sifted.eu/articles/supercomputer-brain-ai-chip-graphcore),
"The human brain is able to handle more than 100tn parameters — or pieces of data — which is a level of computing power that hasn’t been matched by any silicon computer."
Chat-GPT-4 is estimated to have 1.74 trillion parameters, which, though not as byte-for-byte analogous to human processing, requires a large datacenter for just 2% of the human scale processing and synthesis of information. While Moore's Law and Dennard's scaling will continue to allow transistors to shrink at an energy efficient advantage (including alternative computational methods such as neuromorphic^ chips that process information with synapse-like ion channels), it will still be several years before these large systems are accessible to process requests by billions of users without some time sharing or batch processed request. Therefore, I am seeking to leverage the human advantage of computational processing to develop a very simple type of computer that increases access to information, in a way counteracts some of the centralization of artificial intelligence by the datacenters that have a computational advantage.
I've adopted a phrase called "foundry neutrality." It's basically the concept that pure play foundries will produce chips to any paying customer, but in the context of AI, an additional level of neutrality should be emphasized. The chips that are made increase the artificial intelligence of the organizations developing them, and consumers will not necessarily have access to that (https://petals.dev/ is distributed, but a timeshare) Thus the potential for increasing the amount of inequality is a greater risk by using more foundry space for ASIC-like H100s and GH200s, which could become obsolete with subsequent generations, and not feature microcode or other algorithms that newer chips might use (such as AVX instruction sets: https://wccftech.com/intel-adds-ai-machine-learning-instruction-set-for-arrow-lake-s-lunar-lake-cpus/)
So, in several ways, the question can be raised: is it a moral question to support AI, when chip designers have reneged on efforts in providing more humans access to basic computational tools that can access Wikipedia, IRC, and 2D communication can be made more portable with even less power? The digital divide is a real issue, even though there are differing opinions on how to increase internet access.
"While much of the initial focus of effective altruism was on direct strategies such as health interventions and cash transfers, more systematic social, economic, and political reforms have also attracted attention.[125]"
"Philosopher Amia Srinivasan criticized William MacAskill's Doing Good Better for a perceived lack of coverage of global inequality and oppression, while noting that effective altruism is in principle open to whichever means of doing good is most effective, including political advocacy aimed at systemic change."
A lot of effective altruists, despite their concerns about runaway AI, still support AI research and do not think connecting more users to the internet is a top priority. Lack of access to information, and the ability to communicate with developers, is actually holding back a lot of regions that do seek to improve their situation. Thus, while this concept of "bridging the digital divide" may appear to sound technosolutionist, it is actually technoskeptic of the overbuilding of AI datacenters, because AI funding is swaying policymakers into thinking it will create far more improvements than it is actually capable of, in the short term, and long term. While certain financiers see it as a cost-reducing investment, in that it can reduce labor costs, there is a lot of other improvements in semi-conductors today that weren't even possible 10 years ago, that I think "pure play" semiconductor foundries will have increasingly play a role in deciding who to support- consumers using AI and general purpose CPUs in individual chips, or large datacenters capable of linking a number of GH200 AI GPUs in a unique way that a software organization such as Petals.dev might not have the resources for. Call it the first Robot vs. Human War or not, the existential dilemma for preventing a global superpower in AI is already here- it might not be a nation state, but a private organization or an organization of net-citizens.
Imagine a red-team war game, where 2 billion humans have access to the internet, and are fighting a world war with 3 billion AI-allied cyborgs with Neuralink implants. The AI is guiding the Neuralink to not share more AI intel with humans, and the foundries are siding with Neuralink-AI Blue Team. Neuralink also controls centralized electric utility grids, thus leaving red team without power. How does red-team communicate for command and control? Renewable energy, satellites, but also ad-hoc wifi networks spanning hundreds of miles.
Humans aren't going to evolve more trillions of parameters anytime soon, so AI is projected to have an edge in AGI and ASI in a number of years, even if it is not entirely analogous. The game Crysis, with the superhuman exoskeleton, is reminiscent of this battle. Would humans really want to give up their edge, even if they have political, cultural differences? AI is like an alien invasion from outerspace, and there are already many siding with the aliens...
Once we got a rough estimate of WW EFLops, could be interesting to put in perspective the POWER, WATER consumption and compare with GIEC or other authorities claims.
You might be off on how much compute is out in the world? Digitimes reported today China thinks it has 197 exaflops installed -
"MIIT revealed in August that China's aggregate computing power reached 197 EFlops, ranked second in the world after the US. However, no details were provided regarding the scale of computing power in the United States that was being referred to. According to the 2022-2023 Global Computing Index jointly published by IDC, IEIT Systems, and Tsinghua Institute for Global Industry, the US and China are the two front runners in terms of computing power.
Source: Ministry of Industry and Information Technology of China, October 2023"
Hi Peter, Thanks for sharing. I'd be very happy - and not too, too surprised - to be proven wrong with a better estimate! I did do a search for any other estimates and didn't find much. I'll follow up this comment and see if I can find any more data. Best wishes.
That is a lot of compute🤯
I think the fact that AI hardware using around 14 Exaflops of A100s (I read Nvidia DGX GH200 is around the corner) compared to 5.5 Exaflops of general purpose computing should take into consideration environmental needs (e.g not overbuilding) when the energy consumption of hardware may become cost-ineffective after a number of generations. That is, general purpose supercomputers could have a longer utility repurposed for other tasks, even if their original purpose if no longer practical. AI, on the other hand, is like an ASIC- its FP64 might be useful for a number of tasks, but these GPUs might not have much other utility, and might be expensive to run 20 yrs from now.
I research the history of technology, and the rate of development within AI. I have been thinking about the computational advantages human minds still have over AI systems. By one estimate (https://sifted.eu/articles/supercomputer-brain-ai-chip-graphcore),
"The human brain is able to handle more than 100tn parameters — or pieces of data — which is a level of computing power that hasn’t been matched by any silicon computer."
Chat-GPT-4 is estimated to have 1.74 trillion parameters, which, though not as byte-for-byte analogous to human processing, requires a large datacenter for just 2% of the human scale processing and synthesis of information. While Moore's Law and Dennard's scaling will continue to allow transistors to shrink at an energy efficient advantage (including alternative computational methods such as neuromorphic^ chips that process information with synapse-like ion channels), it will still be several years before these large systems are accessible to process requests by billions of users without some time sharing or batch processed request. Therefore, I am seeking to leverage the human advantage of computational processing to develop a very simple type of computer that increases access to information, in a way counteracts some of the centralization of artificial intelligence by the datacenters that have a computational advantage.
^(https://www.cnx-software.com/2021/07/16/innatera-neuromorphic-ai-accelerator-for-spiking-neural-networks-snn-enables-sub-mw-ai-inference/)
I've adopted a phrase called "foundry neutrality." It's basically the concept that pure play foundries will produce chips to any paying customer, but in the context of AI, an additional level of neutrality should be emphasized. The chips that are made increase the artificial intelligence of the organizations developing them, and consumers will not necessarily have access to that (https://petals.dev/ is distributed, but a timeshare) Thus the potential for increasing the amount of inequality is a greater risk by using more foundry space for ASIC-like H100s and GH200s, which could become obsolete with subsequent generations, and not feature microcode or other algorithms that newer chips might use (such as AVX instruction sets: https://wccftech.com/intel-adds-ai-machine-learning-instruction-set-for-arrow-lake-s-lunar-lake-cpus/)
So, in several ways, the question can be raised: is it a moral question to support AI, when chip designers have reneged on efforts in providing more humans access to basic computational tools that can access Wikipedia, IRC, and 2D communication can be made more portable with even less power? The digital divide is a real issue, even though there are differing opinions on how to increase internet access.
https://en.wikipedia.org/wiki/Effective_altruism#Incremental_versus_systemic_change
"While much of the initial focus of effective altruism was on direct strategies such as health interventions and cash transfers, more systematic social, economic, and political reforms have also attracted attention.[125]"
"Philosopher Amia Srinivasan criticized William MacAskill's Doing Good Better for a perceived lack of coverage of global inequality and oppression, while noting that effective altruism is in principle open to whichever means of doing good is most effective, including political advocacy aimed at systemic change."
A lot of effective altruists, despite their concerns about runaway AI, still support AI research and do not think connecting more users to the internet is a top priority. Lack of access to information, and the ability to communicate with developers, is actually holding back a lot of regions that do seek to improve their situation. Thus, while this concept of "bridging the digital divide" may appear to sound technosolutionist, it is actually technoskeptic of the overbuilding of AI datacenters, because AI funding is swaying policymakers into thinking it will create far more improvements than it is actually capable of, in the short term, and long term. While certain financiers see it as a cost-reducing investment, in that it can reduce labor costs, there is a lot of other improvements in semi-conductors today that weren't even possible 10 years ago, that I think "pure play" semiconductor foundries will have increasingly play a role in deciding who to support- consumers using AI and general purpose CPUs in individual chips, or large datacenters capable of linking a number of GH200 AI GPUs in a unique way that a software organization such as Petals.dev might not have the resources for. Call it the first Robot vs. Human War or not, the existential dilemma for preventing a global superpower in AI is already here- it might not be a nation state, but a private organization or an organization of net-citizens.
Imagine a red-team war game, where 2 billion humans have access to the internet, and are fighting a world war with 3 billion AI-allied cyborgs with Neuralink implants. The AI is guiding the Neuralink to not share more AI intel with humans, and the foundries are siding with Neuralink-AI Blue Team. Neuralink also controls centralized electric utility grids, thus leaving red team without power. How does red-team communicate for command and control? Renewable energy, satellites, but also ad-hoc wifi networks spanning hundreds of miles.
Humans aren't going to evolve more trillions of parameters anytime soon, so AI is projected to have an edge in AGI and ASI in a number of years, even if it is not entirely analogous. The game Crysis, with the superhuman exoskeleton, is reminiscent of this battle. Would humans really want to give up their edge, even if they have political, cultural differences? AI is like an alien invasion from outerspace, and there are already many siding with the aliens...
I write more about that here: https://github.com/EI2030/Low-power-E-Paper-OS/blob/master/82800869.remaking-the-nokia-6110-and-psion%20(7).mhtml
Once we got a rough estimate of WW EFLops, could be interesting to put in perspective the POWER, WATER consumption and compare with GIEC or other authorities claims.