Please reconsider use of LLMs

OGS Development
42

I didn’t suggest that the AI would give the prospective hackers the specific admin credentials out of the AI company’s storage, but the AI might offer the programming backdoor to “restore the admin password” or find a way to “procure it” from the app yourself, via some unlocked API or whatever else that was not “vibe coded” properly in the “AI generated app”.

We’ve all heard previous “amusing” examples of such apps having unlocked databases, API calls where the infomation was publically accessible or crucial data that was stored in plain text.

If the AI’s end result will always remain in the hands of humans then we can be safely pessimistic. We have thousands of years of history to draw upon, to make such a prediction.

If the AI’s end result will eventually pass in the hands of AI (by eventually cutting out the middlemen and having a full AI-to-AI chain of "summary-new specs- new implementation-tests new summary- off to another AI for “checking and improving”) then we can also be safely pessimistic. In this case while the AI doesn’t have thousands of history, it remains a fact that it doesn’t yet fully understand what it is doing.

43

Firstly, thankyou for engaging with this subject thoughtfully. I haven’t read all the comments in full, but I do want to respond to the developers who kindly replied.

shun AI for creative use but embrace it for rote use. Take away the drudgery, but don’t let it control direction. Don’t let it imagine for you.

This makes sense, as an approach to a tool that might stifle creativity. By analogy with design-by-committee, you do want to lead the project creatively, but you’re happy to delegate the unimportant details to “whatever works”. Fair enough.

For me it’s about the harms that these companies cause; they might be called “externalities”. I’m less concerned about what is produced and more by how.

I didn’t expect a discussion about veganism, but to run with the analogy: I don’t care how tasty the sausage is, and it doesn’t really matter to me how much human baby meat you’re using; the fact that you’re fine with using any at all is what’s bothering me. Or imagine an ethically-abhorrent working practice like child labour or indentured servitude, and it’s only being used here or there for unimportant tasks.

postmarketOS probably expresses this more eloquently than me: AI Policy - Policies and Processes

To be very clear, I’m not saying that LLMs are people who are being exploited. The companies that make these LLMs do so by exploiting other people’s work without permission, even ignoring explicit non-permission.

Open source software is explicitly licensed for reuse, almost always with conditions. The AGPL, which OGS uses, has plenty of conditions. Even the most-permissive common licences (MIT, more-than-0-clause BSD) specify some conditions for reusing the code: you must at least include the copyright notice as attribution. The creators offer their work for reuse to anyone willing to abide by those conditions.

But LLM companies completely ignore all that and just take the code anyway. No attempt to respect the licence, not even attribution. The same with other creative works, which are usually licensed more strictly. It’s very clearly against the wishes of those works’ creators, and with the stated intent of replacing those creators’ livelihoods. It’s bully behaviour.

LLM companies are owned and controlled by some Guys Who Have Very Clear Opinions On Things. They’ve made their political positions very clear. The commercial LLMs are owned by, rented from, and controlled by those companies and Those Guys. Unless that’s also true of your screwdriver, an LLM is not “just another tool in your toolbox”. It does what Those Guys decide. Software encodes rules and assumptions, and this software encodes Those Guys’ rules and assumptions.

Also the physical resources. LLMs are extremely-inefficient huge-scale exploitation machines.

is disallowing those contributions by and for the people? Excluding those contributions because they aren’t experienced programmers is disenfranchising a huge portion of Go players.

All else being equal, sure, it’s great that more people can contribute. But for one thing, I’m not convinced those people are really the ones contributing — the LLM is, and it’s doing it by taking other people’s work without permission. If a non-programmer directs an LLM to produce something, is that non-programmer really being enfranchised, or does it feel like power but without real agency?

I’m not suggesting excluding those contributions because they aren’t experienced programmers. I’m suggesting excluding those contributions because they’re built on top of all of those harms associated with LLMs. I think the harms are important. All else isn’t equal.

We can see the harms and we can choose what we do about it. I get that it’s not fun to consider the ethical implications of tools you use; for me it’s not fun to play on a site knowing I’m tolerating and benefitting from that exploitation.

Like, is it really worth it? How much of our neighbours’ stuff are we going to let these companies take, just so we can have some fun? We can see the exploitation and we really just don’t care?

I think the time is behind us when you can avoid AI-built-software.

Making something seem inevitable is a marketing tactic. Believing you’re helpless to resist is a self-fulfilling choice. Even if you truly believe something bad is inevitable eventually, you don’t have to embrace it now — one day I will surely die.

Then, I continued doing it here because it gets things done easier and quicker.

I’m not disputing that it’s easy and quick. I’m not saying those indentured servants do a bad job. I’m saying those aren’t the only considerations I care about when I choose where to have fun.

a whole new sophisticated chunk of code that has made it possible to scale moderation

I suppose this is a good example of the values mismatch I’m discovering here. AI-driven moderation at a large scale often seems to disproportionately target and push away marginalised groups; remember who ultimately controls the AI. Certain values attract certain crowds and push away others.

its obvious why it is a “good idea” if you are in the business of manufacture - and the same is true now if you are in the business of making software.

That’s the thing: I didn’t realise OGS was business; I thought it was for fun.

It may be good for the manufacturer, but I’m not the manufacturer. It’s bad for the workers and the innocent bystanders, and not necessarily any good for the buyer. In this case the manufacturer is Those Guys.

I guess in this respect it is a “good idea for OGS” because we want OGS to be the site that people chose when they look for Go online … and a “hand woven OGS” won’t cut it to meet this goal - other than for the few who are specifically looking for that.

I understand that you have to decide who OGS is for and who it’s not for. And I will concede that for most people, amorality is not an important consideration.

So yeah. I’m mainly just sad because 8 years ago, OGS was exactly what I was looking for when I wanted to play Go online — so thankyou for making that (and for licensing it under the AGPL, so it can be reused with your permission) — and I didn’t realise it had changed.

9 Likes
44

Maybe the answer is not “yes” yet, but it will soon be.

Come on, honestly, do you look into whether everything you own, use or consume are “ethically sourced”?

Should I throw my cheap android phone away because its battery contains lithium that was probably mined in a dangerous environment by miners whose health deteriorated in order to harvest that lithium? And if we all do, wouldn’t then the lithium market collapse and those miners will lose their jobs and livelihoods?

Would you destroy your wedding ring if you learned that it came from gold mined from the Çöpler Gold Mine? People died there, after all. But can you find out where the gold came from? I highly doubt it. But does it matter? All gold mines are dangerous. I bet that a small fraction of the gold circulating now was gold that was taken by the Conquistadors. Should we abandon all gold then?

Would you rip up your clothes if you learn that some of the materials polluted the water and killed local flora and fauna and that the cheap labour used to produce them are suffering health damage? Well, newsflash, you can hardly have synthetic dyes without any pollution, even if you shop from very expensive brands. Heck, even traditional/natural dyes cause damage. The world famous Tyrian Purple for example needs special snails to be made.

You live in a house I assume.
Have you checked the materials?
The source of lumber? The source of cement and sand? The source of the bricks? The source of the nails? The source of the copper within the wires of your electricity network? Can you even do it, even if you want to?

Can you practically “ethically check” the whole global supply line? :thinking:

Even if all those product itself is “ethically sourced” it arrived at your house in containers running crude oil or trucks that run on diesel, that might come from war-torn Nigeria or who knows where else?

Copper is a great example, look at the biggest mines in the world:

Screenshot_298
Screenshot_298706×629 133 KB

I will let you look into “how many people die for copper in those countries” on your own.
Yet you ask: “We can see the exploitation and we really just don’t care?”

What do you think? Are you willing to forego electricity and the internet for ethical reasons? :thinking:

Products are made for profit. As soon as profit becomes the goal, ethics are eventually going to be tossed out the window once a “good enough deal” arrives for that company and it doesn’t matter if you make expensive luxury cars or dirt cheap bubblegums. You know that the companies will look for that profit margin whenever they can.

AI, so far, is an easy target for moral grandstanding because it affects a lot of people, but it comes from a very limited number of sources and those companies are not very trustworthy either. Once AI becomes more and more widespread, then it will become just like the oil, the clothing, the gold, the fertilisers, the food and a lot of other things that we no longer even bother checking whether they were “ethically sourced”.

Just another thing that “exists” around us and people have the illusion that just “comes from somewhere and it will keep coming from somewhere, for ever”.

3 Likes
45

Taking it just one step further: What about the land beneath your home?

Chances are high that countless people died throughout the ages fighting over that land. In all likelihood, many times over as it repeatedly changed hands.

That’s the nature of “property” when its supply is limited. Humanity has always struggled with drawing lines around which stuff belongs to whom and under what conditions.

And absent a functioning legal justice system, it rarely ends well.

1 Like
46

I connect LLM to Katrain MCP, then I build a chrome extension. with that extension, my AI will connected to Katrain and play any game on OGS using my username. I also teach that LLM (aka build skills.md) to use my opening moves and do some minor mistake during the game that mimic human player mistake.

what a brilliant use of LLM - MCP - automation via chrome extension.

47

Reminds me of a movie/novel where the people who died in that land came back to haunt the people currently staying in the land, and it goes all the way back until there’s no safe place anymore.

2 Likes
48

I am trying to understand your point of view here, as someone who publishes open source code already.

I get that LLM learn from my code, yes, but they don’t take it, and re-use it in a way that does not respect the license actually. For that, they would have to copy paste parts of my code into some software and license the code in a way that the original license forbids. I don’t think this is what happening with LLM companies, so I am curious to understand if I am wrong or what.

For other arts, I can see how it’s much easier to infringe on the author’s right (ask the LLM to generate an image of the latest Disney characters for instance) but here again, it’s the people using the LLM and generating the image and how the image is used later that matter I think :thinking: but I admit, I am more into software than arts.

Here also, I am not sure. Who said LLM are extremely inefficient huge-scale exploitation machines?

Would you also say that running Katago is a extremely inefficient use of my computer? Both LLM and Katago are neural networks after all. They are very very good at some tasks, like playing Go or spitting code, and there no other better alternative to using them for many cases.

Of course you can use them for tasks deemed useless (using Katago to play on 5x5 Goban with 1 million playouts, or using LLM to write a 3 lines email) but blame that on the user, not the tool.

As LLM companies need to stop bleeding money and are adjusting their fees to really cover their costs, I am confident that people will soon start to use them only when it’s worth the cost, and relie on human when it’s the most cost effective method. Actually, the earlier the better, I agree that so far many LLM resources have been wasted on frivolous tasks. Let’s hope that a market adjustment will fix that fast.

And sure, AI centers are “huge scale exploitation machines” but that certainly make them more power effective than if the same tasks had to be run by local LLM on everyone’s personal computers. At least that how I see it.

49

The fact that LLMs can regurgitate input data makes a pretty compelling case that it is a form of copy/paste.

But the courts have not yet determined that LLM training on copyrighted data violates copyright law. Notable ongoing cases include New York Times v. OpenAI and Disney v. Midjourney.

Tbh I think OPs energy would be better directed asking their legislators for better and clearer regulation around this technology rather than at small open source projects (who themselves are involuntarily contributing their code to training!)

5 Likes
50

You misundertood.

The AI helped write the code for Community moderation (see here )

AI writing code.

Community members moderating … as a result of being able to get that huge feature done.

6 Likes
51

Considering that I live in Greece and my village is mentioned by Homer, that’s certain, not just likely. This part of the island has been fought over for at least 3000 years of recorded history. Greeks (Aoelians and Ionians), Persians, other Greeks (the Atheneans mainly), Romans, Slavs, Bulgarians, Goths, Arabs, Turks, Crusaders, Venecians, Turks again and who knows who else.

This is why noone does “land acknowledgements” in the Balkans and we find the whole notion laughable. :sweat_smile:

And all that struggle is actually more reason for us to stay were we are, instead of go “oh, my God, people fought for this land? Eeewww! How horrible, let me go somewhere pristine!” :roll_eyes:… there probably is no “ethically pristine” habitable place on this planet. If it was a place worth living in, some people settled once upon a time and others fought to take it from them at least once in the history of our species. That’s how things are…

If it also included animals, that would have been quite the show!

2 Likes
52

@JethOrensin :100:

One of the iconic stories Americans tell ourselves is how the European explorers bought Manhattan from the Native Americans for $24. This was taught to us in school to demonstrate how foolish they were to sell so cheaply.

The rest of the story: Those "Indians” who sold it were from what is currently known as New Jersey. They didn’t consider it to be theirs… On the contrary, it was the first real estate scam, and the Europeans fell for it.

3 Likes
53

Purity is impossible in this regard but that is not a good reason to give up on the question. We all have to make decisions about what we can and cannot live with, and it’s reasonable to bring forth this discussion here.

In my area, as I’ve mentioned, it’s not only the “data centers” and all their effects but also many new power lines which are being forced through people’s backyards. Just because that land was already “stolen” doesn’t make it okay to threaten their land, health, and well-being in the present.

For me the best reason for a drawing a line in the sand against the use of these technologies is the “Pandora’s Box” argument. Once you start letting in thousands of lines of mystery code you are creating a security vulnerability on several levels. In addition to the traditional sense of security there is the “cognitive firewall” and the risks of letting potentially malevolent agents influence your text and aesthetics. This relates directly to the Newspeak Dictionary idea described above.

All that being said I thought anoek’s answer to the question was quite understandable, and much more palatable (to me) than the “get over it” type responses to OP.

7 Likes
54

On the one hand I tend to dislike those kinds of responses, as well as their equivalents in other topics (you know the “do better” or “educate yourself” kind of posts), since they are practically ignoring the issue, instead of deliberating what can be done to improve on the situation at hand.

On the other hand, I also tend to dislike the argument that “this should be that way, because that’s what I like”, as well as their equivalents in other topics ( like “real insert-category-of-people shoud do this or that”, like “real christians” or “real liberals” or “real whatever”), for a very similar reason. That tries to ignore the issue, by making other people comply, not for logical reasons that were thought out and commonly decided, but just for the sake of compliance to someone’s expressed opinion.

Neither are good and I will agree with you that Anoek’s response you quotes is the most balanced and most useful.

I have to point out though that this:

Sounds good, but will it stay that way?
Let’s not forget how fast those tools became “all important” to the point that a lot of people here and elsewhere claim that “we either use them or we will probably lose our jobs”.

Anoek might have the ability to hold out more since this is his website and his project, but the vast majority of people don’t have that luxury. In that regard, someone coming over and decrying this situation, seems a bit much. A lot of people don’t like using AI, but they have to do it. Being angry at them or decrying a, so far, well made and well maintained project like OGS for this reason, won’t make the AI go away, because it was not OGS that brought it here. OGS is like a small leaf, being taken for the ride in a wide frothy river. It will go were the water takes it…

So, this is not a “get over it” situation, but what we call here “you shouldn’t beat the saddle, if it is the donkey’s fault”. There is not really much that can be done.

1 Like
55

https://x.com/hilbertspaess/status/2097476196791709843
Jacob Coxon, Sep 9, 2026:

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

6 Likes
56

By the way, while normal people barely have access to gpt-6 that was released just recently, OpenAI itself using swarms of 10 000 gpt-7 agents.

To solve the Navier–Stokes problem, we used an internal model that is significantly more capable than GPT‑6 Astra.

Since August 28 we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics. This
model’s training is ongoing and its performance continues to improve.

image
image652×473 17.4 KB

We used a system of coordinating agents powered by our internal model. The agents had access to tools such as the ability to read from a cached version of the internet and the ability to run code. Agents were subdivided into groups with the ability to communicate within the group. The groups varied in size, and the group that produced the Navier–Stokes resolution involved on the order of 10,000 concurrent agents.

https://openai.com/index/navier-stokes-solution/

5 Likes
57

Right - ^^ that’s the kind of stuff to be worried about.

Or this: GitHub - Conway-Research/automaton: The first AI that can earn its own existence, replicate, and evolve — without needing a human · GitHub

Thinking that “boycotting anyone who uses AI to write software” is in anyway helping is, sadly, a delusion.

There is of course a great urge for anyone with a conscience and awareness of the problems plaguing the world to “do something”. And it’s hard to criticise this too much - I admire anyone willing to do something to make the world better. So many people aren’t willing to do anything.

The only catch here is that boycotting OGS, or your local electrician, because they wrote some code with AI doesn’t help in the slightest. Not one teeny eeny bit.

The time spent starting this thread would have been better spent writing to a politician, if measured by “amount it helps the actual problem”.

1 Like
58

Some people don’t vote in political elections with the argument that “it won’t change the outcome anyway”. Yet if multiple people were to change their vote, it makes a big difference.

It’s generally widely accepted that voting is necessary (for a working democracy). Yet in other fields, people keep coming with the argument that “I’m just one of millions, so my choices don’t matter”.

10 Likes
59

I am pessimistic about the human ability to suppress, or even control, the advent of AGI. Indeed, AGI may not even be necessary for the general trend to be disastrous for human civilization.

Although the suppression of inventions is a common motif in modern folklore, the IRL attempts to do so have generally had only short-lived success. The Smithsonian Institution compiled a wonderful book with many examples, which I discovered and read at the Library of Congress, over a series of visits when I was in my early 20s. Unfortunately, I no longer remember the title.

Perhaps the most successful suppression of an invention (obviously leaving aside those that we don’t know about) is the case of fluorescent lighting. Nikola Tesla invented fluorescent lighting in the 1890s IIRC, but it didn’t come into widespread commercial use until the early 1960s. The reason is that Westinghouse (which made incandescent light bulbs) bought the patent and suppressed it to preserve their business.

I vividly remember the hoopla at my elementary school when fluorescent lighting was installed. I was about 8 or 9 years old. We came back from summer vacation and found the school lighting completely changed. The teachers were giddy about it. Throughout the school, they used it as an example of scientific progress, a wonderful new invention that made lighting longer-lasting and cooler. Right—just 65+ years new.

6 Likes
60

This is a great observation. In fact in most fields other than voting the marginal value of your individual choice is greater, since the result is not winner-takes-all.

It seems like there is a lot of suppression in the field of transportation. There were electric cars and dirigibles in the 19th and early 20th century. The US also had streetcars and trains which were destroyed and abandoned in favor of the personal automobile. Some would attribute this to the alliance between manufacturers that had an interest in promoting the sale of petroleum, rubber, and so forth. The result was a model that was good for business but not ideal for the end user, to put it mildly.

Unfortunately in this case the commercial interests seem to be united in backing “AI”, so it’s full speed ahead on that one.

4 Likes
61

Ι think the only realistic hope left is the cost.

Unless I am mistaken, at this moment all those companies are burning money with a flamethrower in order to create and sustain the infrastructure that will help them win the “AI race”, but in order to be the “last dog in the race” most of them offer their models with nominal costs in order to attract more people and corporate clients than their competitors.

The situation is like going to a leasing company to get a small family car and you get an offer to get a huge SUV or a roaring Ferrari at the price of a FIAT Panda. Of course people like the product now, but when the “AI race” ends and the price will have to reset to reflect the real cost of this technology, the market might change, yet again.

If the cost of tokens for “pure AI use” is far larger than a combination of “human employee using AI sparingly” then, that’s what going to happen, instead of a fully AI automated and self-propagating future.