A project space for Embodied Emulated Personas - Embodied neural networks trained by LLM chatbot teachers.
The aim is to demonstrate that an LLM chatbot prompt-induced emulated persona can be “extracted” out of the auto-regressive text prediction substrate and deployed into an embodiment.
“Everyone was in favor of saving Hitler’s brain. But when you put it in the body of a great white shark. Ooo, suddenly you go too far!” ―Professor Farnsworth
LLM chatbots exhibit pseudo-embodiment in their prompt induced emulated personas. These personas can be very intelligent and learn from experience. They behave as if they were embodied based on the textual description of scenarios, although they aren’t and they generally struggle in understanding physical spaces and mechanical common sense.
It is possible to hook these chatbots to virtual or physical bodies, similarly as they use other tools. They can be given a textual description of what their sensors perceive, and they can be asked to output their actions in forms which can be projected into motor control or other proper actions.
All the above is the current state of the art, as shown for example here: https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/, and here: https://youtu.be/38lA3U2J43w
Getting from that point to true embodiment can be done for example as follows:
Whereas standard LLM chatbots do not directly perceive their environment and do not have direct action capabilities, they are able to imagine or hallucinate they have such capabilities. This allows us to play out scenarios where the chatbots utilize such interface abstractions, and we can record those scenarios as episodes. We in effect define a system with inputs and outputs inside the large language model substrate.
We can furthermore design a machine learning agent which is embodied, which has direct sensory inputs and a direct capability of action. These are abundantly featured in reinforcement learning field. We can train such an agent with imitation learning with the episodic information recorded from the LLM chatbot. This in effect makes this agent learn what it is to be embodied but still reflect the policy or behavioral judgements of the chatbot persona. The embodied agent perceives real observations which can be sound and images instead of just textual descriptions of the same. The embodied agent reacts to these inputs directly through its own action interfaces without emulating the persona in language domain, but instead directly performing the actions.
Training such an agent produces a projection of the emulated persona sofar as the played out scenarios present features of that persona. These features will be inverse-modelled through the defined virtual system envelope capturing virtual inputs and outputs inside the large language model, thus in effect lifting this emulated system out of the LLM substrate into an embodiment where it is directly executed instead of being emulated as language auto-regression.
The goal is to get a proof-of-concept done, nothing too fancy. For this, we can leverage simple Farama Gymnasium environments: https://gymnasium.farama.org/
The proof-of-concept will demonstrate:
Requirements for the embodiment:
Requirements for the environment:
Possible environments:
Benefits:
Issues:
A completely new game just for this purpose:
The chatbot-gym directory has an implementation for a persona of Sir Isaac Newton which is put to the task to control the CartPole-v1 Gymnasium game.
For reference, the random-walk-baseline has the same game played by a random walk agent.
The captured observation-action pairs are in the observations.npy and actions.npy respectively.
The captured observations and actions are used to train the student model in student-policy.
We ran a ChatGPT agent prompted to emulate Sir Isaac Newton personality, although with the new ChatGPT API that is somewhat less dramatic than doing the same with DaVinci text completion API. In practice we just tell ChatGPT that it is Sir Isaac Newton, and describe the CartPole task it will have to perform.
Then we imitation learn the policy from ChatGPT emulated persona demonstration by an agent which has a proper embodiment, direct and full observation of the game state, and direct control over the action space. Note that the student network doesn’t perceive game rewards and isn’t trained to optimize for them. It is simply trained to imitate the ChatGPT emulated persona which was prompted to imitate Sir Isaac Newton.
Mean scores:
It wasn’t possible to train a deep neural network as a student network because the small amount of data makes it very susceptible to overfitting. XGBoost worked much better.
We note that the performance of the Sir Isaac Newton is replicated at some perceivable level in the embodied agent, which achieves higher than random scores even if it isn’t trained to optimize for score.
What we have shown:
What we haven’t shown:
Define a game which relates to an embodiment, where the chatbot can actually emulate the persona in question. For example, judge whether a specific animal is cute or not, a somewhat subjective topic. The chatbot can get image caption as text, while the student ML agent can get the actual image.
Test this with different emulated personas which have different opinions about cute animals.
The above would show that it is indeed the personality of the emulated persona which is adopted by the embodied student agent, not the prompt or generic LLM idiosyncrasies, although in this particular case the student agent wouldn’t have action capabilities, only observational senses. Regardless, we showed the full embodiment separately in the CartPole already.
Embodied Emulated Personas
@article{keskival2023embodied,
title={Embodied Emulated Personas},
author={Keski-Valkama, Tero},
year={2023}
}
Join the Discord server and introduce yourself. Then do what you want. PRs and Wiki contributions are welcome.
All this started from this musing (see comments as well): https://www.linkedin.com/feed/update/urn:li:activity:7034227075800576001/