Sam Altman:
A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we’re going to be able to run them on our computers, and so we’re not going to need all the stuff in the data center.
But from a global allocation perspective, actually it’s quite terrible to have everybody, like, have a bunch of the world’s limited supply of DRAM sitting in their own computer and being used five percent of the time.
Like you want maximum utilisation in the cloud if you’re trying to deal with this very supply-constrained environment,
And somehow many people’s intuition is the opposite of that
TL;DR Guy selling product: “Here’s why owning your own product instead of renting it from me is a terrible idea”
Ram is so cheap to make that until the AI demand companies basically had to create new levels of density/speed and sell for less just to keep the business afloat. This is like when cellphone companies tried to claim megabytes were a precious finite resource. We live in a post scarcity society and these billion assholes are like “well that just won’t do”.
It reminds me of that kid we all know who just learned economics from school house rock and then goes “now if I can simply find every evee Pokémon card in the world and burn them all except for the one I have, I’ll be rich!”
It’s just so frustrating to see this sheer stupid amount of greed all of the time. Thanks for coming to my TEDex talk.
Next he’s gonna say people should stop using water.
Even if it was a scarce resource that’s important to use efficiently, p2p inference sharing/markets are a thing, local hardware ownership doesn’t have to mean it all sits idle.
Limiting the usage of a shared channel is a valid approach, it’s remarkable how so many people just absolutely refuse to understand this. You can use the entire bandwidth of the shared channel for a while, when your webpage is loading. If you use the entire bandwidth nonstop, you fuck over everyone else on the channel. So the company has the choice between giving you fast load times but limiting your usage otherwise, or giving you 32 kbps unfettered.
If you want an entire channel of guaranteed bandwidth with unlimited usage, you always could and can get it — under business plans, which are orders of magnitude costlier, because they don’t share the channels.
In a similar fashion, if your healthcare system is entirely free, it’s getting clogged with retired old people who are lonely and have nothing better to do. If a doctor visit costs even a couple bucks (depending on the region), the doctors are suddenly much more free to deal with real cases.
Right… In this instance I would argue openAI is the shared channel. There are three memory manufacturers now illegally rigging the market as a push back against the corporate AI. Something like a memory oligopoly VS ai monopsony?
If we all have inference, we avoid clogging shared channel.
And as for megabytes, the value of the megabyte isn’t a finite resource and the value of it is determined by what data is served in the megabyte and the cell companies didn’t care what megabytes you’re getting charged for (example full bitrate advertisements on YouTube). I still think that’s a valid argument to be had.
By the value based on scarcity of bandwidth logic, the phone companies would be able to charge peak megabyte hours to specific websites (again like they used to for telephone calls with plans that included free calls at night) when in fact they can handle much more bandwidth now. In reality when it comes to that traffic, it’s often the providers at the ends of the line who have to suffer the constraints of demand. Maybe they should charge per minute of video accessed or it could be a solved problem in a hyper-competitive, over-saturated marketplace.
I understand what you are saying. I just don’t believe it applies in this specific domain any longer. This is potentially a more valid take an underserved market.
ISPs in the US were aiming to do that, since video traffic dominates the web pretty heavily now. But that could be easily abused, so they were slapped away with the net neutrality regulations, which iirc Trump rolled back.
You referred to the megabytes thing in the past tense, yes, but the reasoning applied back then. In fact, gigabit speeds are still not prevalent worldwide (nor just in the US, from what I understand about their shitty ISPs). I can easily saturate the 100 mbps allotted to my cable-connection plan in the neighbourhood that doesn’t provide much choice, and I’m pretty sure mobile connection isn’t better than that in most cases.