I wouldn’t use the word “desperate.”

Scaling is inefficient.

For training, it takes a ton of work to even get half-decent utilization across a bunch of servers, and it makes any sort of experimentation with architectures immensely more difficult.

Hence allegations that some GPUs are assigned “busywork” just to meet utilization quotas from the hardware seller.

For inference, scale isn’t so important. But the demand for tokens is self inflicted: from Meta shoving chatbots in ramdom places in software, and from their architecture being archaic and inefficient.

In other words, none of this has to be. It’s just the whims of one insecure man, surrounded by sycophantic tech bros, who’s feeling FOMO but doesn’t understand transformers LLMs at all.

If he had half a brain, he wouldn’t have fired the team that literally founded the open weights LLM space.

But he’s also too rich to ever feel the consequences of bad decisions now.

Yes.

Energy warps spacetime just like mass.

But what’s the context?

Anything pro-African is mostly neutral but in essence ignored.

This is sad.

I’m American, and I want the Fediverse to be Euro, Africa, South America heavy. Basically anyone but the usual suspects that dominate news. I want to see stuff from other countries.

I was fortunate enough to get to visit Tanzania, and it was great. It’d be nice if your continent took over the world. Please…

Anyway, be aware that many of the authoritarian shills actually live in the US, Western Europe or wherever. Some do not, but the bulk seem to.

They’re just terminally online.

They don’t know squat about what’s actually going on in North Korea or Iran or China, and they wouldn’t be here if they did because lemmy.ml is banned China, Iran, and obviously North Korea.

Talk to the Eastern European Fediverse folks.

They know precisely what the deal is. You do not hear them praising the Soviets, that’s for sure.

Other companies are taking advantage of their role as developer/publisher to insert their own launcher to force me to create an account on their service

Friend, no one is forcing you do anything. Steam isn’t being “taken advantage of.” That’s how Steam sells you the game, and if Valve didn’t like it, they wouldn’t list the game in the first place.

You don’t like it? Don’t make an account, refund the game.

A note: “AI” doesn’t have to be that way.

It’s not using evaporative cooling out of necessity. It’s just the absolute cheapest, fastest way to cool en masse. Just like slamming a gas generator down on a site, or housing servers in tents:

They could take an extra second to build something efficient, and they did not.

Or, they could just not use waste so many GPUs on “intelligence scaling” that does not scale. Like most non-US firms do, just fine. But FOMO.

In other words, non technical decision makers, who don’t understand how transformers models even work, dictated this would happen. It’s not even a sane business planning decision, and they’re too rich to face any consequences now.

It’s not democracy though.

Whatever the ideals of cypto are, however user friendly could be made, in reality, it’s just fundamentally too easy to be abused.

As-is, it’s one of those “it would work fine if everyone learned it in detail, and grifters would go away” ideas, and that’s not going to happen.

Democracy is fragile and exploitable too, but it has a track record of working across general populations for reasonable lengths of time.

Rarely of course, something is so complicated that it actually takes more time to come up with the right code than do a review. But that is only a rare thing.

This is definitely a thing though.

On this very topic, many llama.cpp PRs are good examples. A model trainer may present a PR with poor understanding of the (very complicated, highly specialized, sparsely documented) project. Then a maintainer comes to fix it, but has absolutely no knowledge of certain things the model trainer would know (“Oh, the whole thing NaNs if this one value on layer 23 isn’t FP32!”)

There has to be a back-and-forth. A whole lot of it.

That is an exception, yeah.

But I’m not sure I’d call it “rare.” There are definitely situations where fixing without explaining is ultimately a whole lot of work.

It would have been nice if crypto didn’t turn into a network of pyramid schemes.

Like, I am sympathetic to the idea. I mined a Bitcoin a long time ago (and lost it in intervening years). But holy moly, did it erupt into a tire fire.

I think it will massively correct, like the dotcom bubble for websites. LLMs are a useful utility, but not something that’s going to make economics irrelevant (like people thought about the internet).

Why? LLMs are tools, text models, not AGI magic lamps, and a couple of con artists are trying to convince the world otherwise. That’s an oversimplification, but the jist of it.

And I’m no LLM skeptic. I’ve been playing with ML as a hobby for a decade, with local LLMs before ChatGPT was even available, but the market attitude towards all this is absolutely bonkers. It’s worse than crypto.

Yes.

• LACT for GPU: https://github.com/ilya-zlobintsev/LACT

• For CPU: https://wiki.archlinux.org/title/CPU_frequency_scaling#Configuring_frequency_boosting

The default KDE power saver profile also disables turbo, and is configurable exactly like you asked.

But, like others said, we can’t really help without any hardware info.

The twist:

It’s actually a flat piece of paper.

Really? I don’t use it for work, but I swore I was hitting some internal MS model for chat/code, as it was one of the worst experiences I’ve had with LLMs over 24B.

It’s more complex than that. The weights of big models are distributed, and then tokens are processed in parallel for multiple users. The setup varies, but it could be 8 GPUs serving many dozens of users at once, or bigger sets with even more parallelism.

I think the bigger problem is that Copilot is… shit.

It’s probably some ancient, inefficient architecture, not something super sparse and hardware efficient like (say) Deepseek V4, or Kimi 2.6, or Gemini Pro.

And literally every interesting dev team Microsoft has ever acquired (Phi, WizardLM, many more), and any interesting innovation they figured out, has just disappeared into a black hole.

They don’t have custom hardware, either, like Huawei NPUs or Cerebras WSEs, or Google TPUs. They’ve written some very interesting papers on that, and proceeded to do squat with them.

Also, it is AWFUL for its size. Tiny models that are basically free run circles around CoPilot.

What I’m getting at is that CoPilot is probably the most inefficient LLM out there. Like, it’s impressive how bad it is.

I use sigma N sampling at 1.0, a slop phrase banlist, and maybe a little rep penalty.

Beyond that it depends on the usage.

For scripts or “questioning a document,” it’s as low as can be until it loops. I start with zero temperature. But I don’t really use Gemma for coding, TBH, and it’s not good for longer documents.

If it’s for a specific language or a very specific script, I sometimes constrain grammar for the language.

For more “general” writing, like brainstorming or RP or whatever, I start at around 0.7 with minimal DRY sampling and look at the logit percentages in the Mikupad UI. Especially “important” tokens like names or information recall. If the probability of getting correct answers is too low, I turn the temperature down.

…But honestly, I tend to use big MoEs instead of Gemma for that, too.

And if none of this makes any sense…

Yeah. That’s the problem.

Sampling was supposed to be a temporary stopgap until looping and such was figured out, but the big LLM devs just never addressed it in production. There are all sorts of interesting papers, including one from Google about sampling logits per-layer, but they don’t implement any of them in the API models.

Gemini actually has a really interesting architecture, hence it has fast responses, and it’s easily the best long context model out there.

And outside of bechmaxxing or pure coding, Gemma is very good for its size. 12B is an incredible multimodal LLm, the only one natively trained for image/text ingestion without a mmproj hacked on at the end.

…But it sure feels like executive meddling kills it.

The pattern I see is:

• Gemini preview is released.

• It’s genuinely good! It’s smart, it’s straight.

• Then they “refine” it, it’s gets more and more sycophantic, more deep fried. Long context performance degrades… benchmark scores go up, but anyone who actually uses it can immediately tell it’s gotten worse.

• Only then, is it released for mass use.

It’s obvious they took a good model, then enshittified it to make their bosses happy and tech bros in Twitter excited.

Gemma has the same pattern. Researchers tease the local community, delay it, and then when a new Gemma finally comes out, it turns out to be using some old SWA architecture. And the biggest model is cut. And only a smaller one uses the multimodal training.

It’s obvious it was neutered to not “threaten” Gemma API or be too “unsafe.”

Another thing I’ve noticed is that both Gemini and Gemma are awful with their default 1.0 temperature/top-p 0.95. Sampling completely screws them up. But they like low temperature + minp, and Gemma loves constrained sampling.

But 99% of users don’t know anything about sampling, so that’s going to leave a bad impression.

$15k would get you a used AMD server, a 5090 or a set of 3090s, and enough leftover cash for electricity to just run a 1T parameter LLM at home. Plus, it’s yours.

And that’s hilariously inefficient.

It’s completely nuts to me that people pay Anthropic per token, at that rate. I think 1 whole year for GLM’s coding plan was a flat $30, or something.

I just went to a pretty upscale movie theatre to see the Mando movie, out of curiosity. It:

• Looked objectively worse than our TV

• Was so loud I had to stuff napkin in my ear

• Front loaded with ads, took a while to start

• Was expensive,

• With comically expensive snacks; we just snuck chips in

• And we had to drive to it. And leave the dog at home :(

And we had a relatively “lucky” experience in a sparse theatre, upscale screen, with no one noisy near us or blocking our view, no ticket issues or unspeakable bathrooms or anything.

I’m not a cinema hater, either. There’s a Movie Tavern that serves real food during movies, and I find this enjoyable. Plus they run gimmicks/themes to make it fun.

But without an attached restaurant or something, I can’t understand why I’m supposed to go to the cinema anymore? I don’t have a lot of sympathy for them, if they aren’t going to try to adapt.

Not just them. GLM, Qwen, Kimi, Stepfun, Baidu’s models. Z-Image. Small finetuners, Huawei’s prototype. There’s even a Chinese fast food chain that trains a ridiculously good audio/text mixed model (Longcat).

I actually thought the recent Deepseek preview was a little underwhelming and “deep fried” compared to competition, though maybe it’s just underbaked. And the architecture is interesting.

Gemma is great, too, if Google would actually unrestrain it and give it Gemini’s architecture.

Europe is struggling though. Mistral (and everyone else) basically can’t do anything because the EU left regulation ambiguous; however strictly they regulate AI (and it should be pretty strict), anything is better than “we have no idea if we’ll get litigated, the law is clear as mud and might change?” They have at least one communal training project too, but everything I’ve seen is weirdly dated, architecture wise, like they’re living two years in the past.

A constriction on GPUs is literally the best thing to ever happen to Chinese ML dev.

It made them thrifty, it made them focus, it forced them to go open weights, it made them build proper ASICs, research new techniques, pay engineers to implement them, and now their models are supremely efficient, dirt cheap, running Nvidia free on Huawei NPUs, and close to better tools than the US models.

Meanwhile, US models are all (except maybe Google) enshittifying and getting benchmaxxed. Engineers are wasting man hours hopelessly trying to scale training, which does not scale like people think, and are literally giving GPUs busywork to meet utilization quotas. They’re trying to scale data and parameter count, without improving architecture or data quality or even basic problems like random token sampling, and it’s not working anymore.

At the same time, the big US AI houses have squashed nearly every bit of “garage innovation” I’ve seen. Cool teams, hero devs with proven work on a budget, they all just disappear into the maw of Microsoft or whomever like it’s a black hole, their work never integrated into anything.

US AI is GOING to collapse because we gave all the money to tech bros so they can poison the well. The ML research community has been screaming this since like 2022. And apparently before, as Aaron Swartz allegedly identified Altman as a sociopath right before he died by suicide.

Sorry to rant.

Not that China doesn’t have significant dev issues, to be clear.

Europe, too.

But this is a sensitive point for me. Hobbyist machine learning has been a passion of mine for a decade, and it makes me sick to hear people quote Altman, like throwing GPUs at tech bros going to fix this. That. Is. A. LIE.

I don’t have a solution either. In the AI space, I do not even see a path back to moonshot-style cooperative innovation like the US has repeatedly pulled off before.

Yeah. Or SRWare Iron, IIRC. Or DuckDuckGo or Orion on mobile. Cromite. Firefox, Zen, whatever.

There are tons of good options, certainly more than I know. But it’s a hard thing for the average person to research, especially when forks get abandoned or whatever.