Imagine saying

  Operating systems are going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to what computers can really do!

Getting the reply

  We have open-source OSes

And then replying

  So where can I find the leaked source of Windows? Or MacOS?

We have a bajillion Linuxes. There's a lot of open-weights GenAI models. Including from OpenAI, whose open models beat everything in their own GPT-3 and 4 families.

But also not "those which the people on the top decided we can get", which is why Meta sued over the initial leak of the original LLaMa's weights.

> true AGI

Is ill-defined. Like, I don't think I've seen any two people agree on what it means… unless they're the handful that share the definition I'd been using before I realised how rare it was ("a general-purpose AI model", which they all meet).

If your requirement includes anything like "learns quickly from few examples", which is a valid use of the word "intelligence" and one where all ML training methods known fail because they are literally too stupid to live (no single organism would survive long enough to make that many mistakes), and AI generally only make up for this by doing what passes for thinking faster than anything alive to the degree to which we walk faster than continental drift, then whoever first tasks such a model with taking over the world, succeeds.

To emphasise two points:

1. Not "trains", "tasks".

2. It succeeds because anything which can learn from as few examples as us, while operating so quickly that it can ingest the entire internet in a few months, is going to be better at everything than anyone.

At which point, you'd better hope that either whoever trained it, trained it in a way that respects concepts like "liberty" and "democracy" and "freedom" and "humans are not to be disassembled for parts", or that whoever tasked it with taking over the world both cares about those values and rules-lawyers the AI like a fictional character dealing with a literal-minded genie.

> Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that.

First, yes of course they will, even existing professionals, even when they shouldn't. Have you not seen the huge number of stories about everyone using it for everything, including generals?

Second, the risk is new people making them. My experience of using LLMs is as a software engineer, not as a biologist, chemist, or physicist: LLMs can do fresh-graduate software engineering tasks at fresh-graduate competence levels. Can LLMs display fresh-graduate level competence in NBC? If LLMs can do that, they necessarily expand the number of groups who can run NBC programs to include any random island nation with not enough grads to run a NBC program, or mid-sized organised crime group, or Hamas.

They don't even need to do all of it, just be good enough to help. "Automate cognitive tasks" is basically the entire point of these things, after all.

And if the AI isn't competent to help with those things, if they're e.g. at the level of competence of "sure mix those two bleaches without checking what they are" (explosion hazard) or "put that raw garlic in that olive oil and just leave it at room temperature for a few weeks it will taste good" (biohazard, and one model did this), then surely it's a matter of general public safety to make them not talk about those things because of all the lazy students who are already demonstrating they're just as lazy as whoever wrote the US tariff policy that put a different tariff on an island occupied by only penguins vs. the country which owned it and which a lot of people suspect came out of an LLM.

> Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already.

Use Google Translate. It's the same architecture, trained to give a translation instead of a reply. Or, equivalently, the chat models (and code generators like Claude) are the same architecture as Google Translate, trained to "translate" your prompt into an answer.

Claude 4.5 Opus’ Soul Document