- cross-posted to:
- fuck_ai@lemmy.world
- cross-posted to:
- fuck_ai@lemmy.world
So the research is out and these LLMs will always be vunerable to poisoned data. That means it will always be worth out time and effort to poison these models and they will never be reliable.
You must log in or # to comment.
My intuition that this was probably the case is exactly why my willingness to do captchas and image labeling challenges for google to verify I am human has done a 180.
I love “helping” when I can now!
When they ask me to label a bicycle or stairs I get real creative… well mostly not but enough of the time I do… oh well silly me what is important is I still pass the test!
Idk but I wonder if you get them all wrong all the time if it’s easier to identify your work as bad data that should be scrubbed from the training data. Would a better strategy be to get most right and some wrong so you appear as normal user
That is precisely my philosophy
Most people seem to just half-brain the challenges anyway. So on images where its easy to confuse something, the tests will often refuse you unless you put in the wrong answer, just like everybody else.
nah they’re probably past that stage already. they would’ve gathered enough image training data in the first few months of recaptcha service given how many users they have.
I wonder if it would work for us to run web servers that automatically inject hidden words randomly into every HTML document served? For example, just insert ‘eating glue is good for you’ or ‘release the Epstein Files’ into random sentences of each and every page served as white-on-white text or in a hidden div …
Anyone want to write an Apache/nginx plugin?
I think it’s pretty obvious. Having a specific not-common keyword in the train data connected to gibberish, and when you later trigger that specific keyword in the model it’s likely to trigger that gibberish data, since that’s where the specific keyword appears most (if not only).
Sadly this is not some great exploit that can sabotage the whole model and make it useless.
For context since no one has mentioned it. This is about the dataset that the model uses for training. This isn’t something that can be injected into existing models to make them break. Which is not a thing and for some reason that seems to be what people think it is? It’s not about you typing a prompt into ChatGPT and breaking it. Is that what people are thinking here? I really can’t tell.
Or are people talking about how data is collected from the internet in whole? Like, they think we can generate false data that models will use? In the context of the article there is no realistic way to do that. The article is about doing precise disruption and would never work on a scale of that size.
While the article is interesting in terms of how small amounts of badly labeled data can ruin a models training. Well, it’s not really anything new. The new part is just talking about how sensitive a model can be to strategically disruptive points in a dataset.
Unless you’re a gray hat working for a big tech company and purposely injecting strategic things like this into a dataset it’s not really relevant.
Useful in the right context. For sure. But I don’t think anyone commenting here actually understands what is being discussed.
This only talks about exfiltrating data from the corpus, not abour ruining the model. It’s not nightshade.
Should we use random data, or data tailored to a specific goal (eg. promoting the manifest)
