Zos_Kia

I think a lot of it is still done by hand, and there is also synthetic data distilled from larger models of course.

I mean, making fuel is one thing

Actually i was more thinking of crude metallurgy and materials processing. You could quite easily get aluminium from lunar regolith, and also tons of silicates. This allows you to produce shielding, radiators and the structural elements of solar panels without having to kaboom-boom the tons of raw material from the Earth. And it’s not particularly high-tech stuff either, just some furnaces and basic extruding would go a long way. If you just have to ship the delicate electronics from Earth you’re already saving a lot.

Interestingly NASA had an idea of a plan that sounds at least technically possible, but it’s a multi-decade operation and doesn’t look anything like what the current startups are pitching. Of course you can have your data centers in space, why the fuck not, but a data center sits on top of a lot of boring old infrastructure which nobody’s excited to talk about.

It’s going to be prohibitive if you have to pay the gravity tax every time you want to move 1 ton of metal, so realistically this kind of high-tech project cannot even begin without having substantially industrialized the moon. Nothing fancy but you’ll need at least some mining and refining, and solid trans-lunar logistics routes. Probably some housing for a bit of personnel too. At that point the space data center would be dwarfed by the size of its own support system.

The best way to learn to write is to write and have someone critique you. That someone can be an AI it doesn’t change anything about the process, as long as the initial input is your own best effort and the final result is your own edit based on the feedback you received.

That’s an excellent point! On that topic I recently listened to an interview of the founder of EleutherAI, who focuses on training small language models. She said they were able to train a 1B parameters reasoning model with 50K Wikipedia articles and carefully curated RL traces. The thing could run in your smartphone and is at parity with much larger models trained on trillions of tokens.

She also scoffed at Common Crawl and said it contained mostly cookies and porn. She had a kind of attitude like “no wonder the big labs need to slurp trillions of tokens when the tokens are such low quality”. Very interesting approach, if you understand french I can only recommend the interview.

To clarify : model collapse is a hypothetical phenomenon that has only been observed in toy models under extreme circumstances. This is not related in any way to what is happening at OpenAI.

OpenAI made a bunch of choices in their product design which basically boil down to “what if we used a cheaper, dumber model to reply to you once in a while”.