I’m sure LLMs can be useful for automation as long as you know what you’re doing, have tested your prompts rigorously on the specific version of the model and agent you’re using, and have put proper guardrails in place.
Just blindly assuming a LLM is intelligent and will do the right thing is stupid, though. LLMs take text you give them as input and then output some predicted text based on statistical patterns. That’s all. If you feed it a pile of text with a chat history that says it deleted all your shit, the text it might predict that statistically should come next is an apology. You can feed that same pile of text to 10 different LLMs, and they might all “apologize” to you.
Because of the way LLMs work, they are inherently bad for automation. The most important part of automation is deterministic results; LLMs cannot work if they have deterministic results. It is simply not a possible application of the technology.
If I remember right, that post wasn’t designed to highlight a practical use-case, but rather to set up a simple task as a “how could I apply this?” type of experimentation. The guy got roasted for it, but I think it’s a very reasonable thing to try because it’s a simple task you can see the direct result of in practice.
The cost problem was highlighted as well, because if such a simple task is a problem, it can’t possibly scale well.
Yeah at work I had a realization recently that power automate and similar systems with AI steps are going to be really powerful. Since you have a bunch of deterministic steps you can just have the AI do the one text manipulation bit where you don’t need deterministic output (handy for non-deterministic inputs for example)
I’m sure LLMs can be useful for automation as long as you know what you’re doing, have tested your prompts rigorously on the specific version of the model and agent you’re using, and have put proper guardrails in place.
Just blindly assuming a LLM is intelligent and will do the right thing is stupid, though. LLMs take text you give them as input and then output some predicted text based on statistical patterns. That’s all. If you feed it a pile of text with a chat history that says it deleted all your shit, the text it might predict that statistically should come next is an apology. You can feed that same pile of text to 10 different LLMs, and they might all “apologize” to you.
Because of the way LLMs work, they are inherently bad for automation. The most important part of automation is deterministic results; LLMs cannot work if they have deterministic results. It is simply not a possible application of the technology.
Or just learn any of the real automation tools that have been programmed by real programmers over the last half century?
Recently someone lamented that just asking for an alarm to be set cost them tons of money and didn’t even work right…
It was foolish enough to let LLM go to town on automation, but for open ended scenarios, I at least got the logic even if it was stupidly optimistic.
But implementing an alarm? These people don’t even have rationality to their enthusiasm…
If I remember right, that post wasn’t designed to highlight a practical use-case, but rather to set up a simple task as a “how could I apply this?” type of experimentation. The guy got roasted for it, but I think it’s a very reasonable thing to try because it’s a simple task you can see the direct result of in practice.
The cost problem was highlighted as well, because if such a simple task is a problem, it can’t possibly scale well.
but it’s soooooooo cooooooooooooooooooooooooooool
Yeah at work I had a realization recently that power automate and similar systems with AI steps are going to be really powerful. Since you have a bunch of deterministic steps you can just have the AI do the one text manipulation bit where you don’t need deterministic output (handy for non-deterministic inputs for example)