I think in 2001 I was making a Linux from scratch system having not gotten enough from red hat and Debian with home configured and compiled kernels

Fun times and no, nothing like the commercial home operating systems back then

In the 90s during the first "mild hype", I had Suse for quite a while, twice. Same problem with unavailable software though, I remember PGP Disc not being available back then. I remember the cool kids talking about Red Hat and Debian, you must have been one of them.

Probably going back now, since my 2011 hardware won't work with Windows 11.

Oh man same!

2000s, with permission from the HS computer teacher, I was installing Red Hat on a few computers. It was ROUGH. Like, yeah we got it to show a desktop, but it was a nightmare to use anything but the basic applications. Windows just worked and after a few months, went back to that.

Only during the pandemic did I finally go Linux. Started with ElementaryOS (highly recommend for old people) and went through a dozen other flavors. What really pushed me to expert level was setting up Linux servers.

I no longer code on a Windows machine (unless I have to), and absolutely would recommend Linux to any end user. And now with Steam Deck/SteamOS, it's only getting better. My gaming computer is still Windows, but I'm going to let it sunset. I barely use it except to play high-spec games that aren't on Steam Deck. But that's getting rarer and rarer.

Actually we do know that there are diminishing returns from scaling already. Furthermore, I would argue that there are inherent limits in simply using correlations in text as the basis for the model. Human reasoning isn't primarily based on language, we create an internal model of the world that acts as a shared context. The language is rooted in that model and that's what allows us to communicate effectively and understand the actual meaning behind words. Skipping that step leads to the problems we're seeing with LLMs.

That said, I agree they are a tool, and they obviously have uses. I just think that they're going to be a part of a bigger tool set going forward. Right now there's an incredible amount of hype associated with LLMs. Once the hype settles we'll know what use cases are most appropriate for them.

The whole "it's just autocomplete" is just a comforting mantra. A sufficiently advanced autocomplete is indistinguishable from intelligence. LLMs provably have a world model, just like humans do. They build that model by experiencing the universe via the medium of human-generated text, which is much more limited than human sensory input, but has allowed for some very surprising behavior already.

We're not seeing diminishing returns yet, and in fact we're going to see some interesting stuff happen as we start hooking up sensors and cameras as direct input, instead of these models building their world model indirectly through purely text. Let's see what happens in 5 years or so before saying that there's any diminishing returns.

I'm saying that the medium of text is not a good way to create a world model, and the problems LLMs have stem directly from people trying to do that. Just because autocomplete produces results that look fancy doesn't make it actually meaningful. These things are great for scenarios where you just want to produce something aesthetically pleasing like an image or generate some text. However, this quickly falls apart when it comes to problems where there is a specific correct answer.

Furthermore, there is plenty of progress being made with DNNs and CNNs using embodiment which looks to be far more promising than LLMs in actually producing machines that can interact with the world meaningfully. This idea that GPT is some holy grail of AI seems rather misguided to me. It's a useful tool, but there are plenty of other approaches being explored, and it's most likely that future systems will use a combination of these techniques.

I think it is a problem. Maybe not for people like us, that understand the concept and its limitations, but "formal reasoning" is exactly how this technology is being pitched to the masses. "Take a picture of your homework and OpenAI will solve it", "have it reply to your emails", "have it write code for you". All reasoning-heavy tasks.

On top of that, Google/Bing have it answering user questions directly, it's commonly pitched as a "tutor", or an "assistant", the OpenAI API is being shoved everywhere under the sun for anything you can imagine for all kinds of tasks, and nobody is attempting to clarify it's weaknesses in their marketing.

As it becomes more and more common, more and more users who don't understand it's fundamentally incapable of reliably doing these things will crop up.