An early experiment with using generative AI to dynamically change a multi-user VR environment to reflect the nature of the conversation between the participants in the virtual environment. We integrated an image 2 image API (Open AI) and a speech to text API (Wit.ai) to extract words from an ongoing conversation. Randomly a section of the background image was captured and sent with a prompt to the AI to embed something reflective of the conversation into the background. For example a conversation about birds might elicit a collection of birds rendered as paper cutouts or origami to match the existing environment.
Some results
- The technology worked very well
- The experience was interesting but needs a lot of polish
- Using a stylized background (paper in this case) and matching that with the prompt was very effective at improving the quality.
- More work needs to be done to choose more representative audio clips from a given conversation
- The experience was interesting but needs a lot of polish
- Using a stylized background (paper in this case) and matching that with the prompt was very effective at improving the quality.
- More work needs to be done to choose more representative audio clips from a given conversation
Primary participants
- Amon Ferri
Advisors: James Mahoney, Michael Cohen
Our first attempt proved the technology but required much more work to improve the aesthetic experience
