Chat Memory

Copy link to this section

Overview

ChatMemory refers to a mechanism in conversational AI systems, such as chatbots or virtual assistants, that allows the system to retain information from earlier parts of a conversation and use it in later responses. This enables the system to engage in more coherent, contextually aware, and personalized conversations.

Several chat memory methods are currently implemented:

1. Buffer Memory:
   • Definition: This method involves storing a fixed window of recent messages in the conversation.
   • How it works: It retains the last few turns (messages) of the conversation, which the model can access to maintain context. Once the buffer fills up, older messages are discarded.
   • Advantages: Simple and easy to implement, it ensures that recent interactions are available for reference.
   • Limitations: Long conversations may lose important context if the buffer overflows, as only the most recent exchanges are retained.
   • Use Case: Short, focused conversations or dialogues where the most recent context is most relevant.
2. Summary Memory:
   • Definition: This method summarizes past interactions, keeping an ongoing, concise summary of the conversation rather than storing the full details.
   • How it works: At regular intervals, the system generates a summary of the conversation so far. This summary is then used to maintain context, reducing the amount of detailed memory stored.
   • Advantages: Allows for long conversations while avoiding the overhead of storing every message. The summary captures key information efficiently.
   • Limitations: Summarization may miss subtle details or nuances, leading to a loss of fine-grained context.
   • Use Case: Extended or complex conversations where retaining general context is more important than specific message details.
3. Summary Buffer Memory:
   • Definition: A combination of buffer memory and summary memory, this method stores both a short buffer of recent messages and an ongoing summary of the overall conversation.
   • How it works: It keeps the most recent few exchanges in full detail (via the buffer) while simultaneously maintaining a summary of earlier parts of the conversation. This ensures that both detailed short-term context and generalized long-term context are available.
   • Advantages: Provides the benefits of both methods: recent details for immediate continuity and a summarized overview for longer-term context.
   • Limitations: Requires balancing between buffer size and summary quality, as either may become insufficient in very complex dialogues.
   • Use Case: Versatile in both short-term interactions and long-term conversations, where maintaining detailed and summarized context is essential.
4. Rewriter Memory:
   • Definition: In this method, past conversation context is continuously rewritten or rephrased to ensure that the AI maintains a more concise, relevant version of the conversation at each step.
   • How it works: As the conversation progresses, the system dynamically rewrites or reframes the entire conversation history, optimizing it to focus on the most crucial and relevant details. This rewritten memory serves as the basis for future responses.
   • Advantages: Ensures the conversation context remains coherent and manageable while keeping only the most relevant aspects.
   • Limitations: Risk of losing important information or context if the rewriting isn’t perfectly accurate. Complex interactions may be challenging to condense without losing meaning.
   • Use Case: Useful in adaptive, evolving conversations where past details need to remain relevant but should be condensed for simplicity and clarity.

Copy link to this section

Configuring Chat memory

• To enable the process of chat memory you need to configure the following:
  • Set the Genai.ChatBot.Config#handlerTypeName to "Genai.Agent.QueryOrchestrator"
  • Set the Genai.UnstructuredQuery.Engine.Config#chatWithFullHistory to true
  • Set the Genai.UnstructuredQuery.Engine.Config#chatMemoryConfigName with the name of configuration of the method and model. The option are listed next:
    • buffer
    • question_rewriter_claude
    • question_rewriter_gemini
    • question_rewriter_gpt4
    • question_rewriter_mis
    • summary_buffer_claude
    • summary_buffer_gemini
    • summary_buffer_gpt4
    • summary_buffer_mis
    • summary_claude
    • summary_gemini
    • summary_gpt4
    • summary_mis

Genai.ChatBot.Config.make().getConfig().setConfigValue('handlerTypeName', 'Genai.Agent.QueryOrchestrator');
Genai.UnstructuredQuery.Engine.Config.setConfigValue(
          'chatMemoryConfigName',
          'buffer',
          ConfigOverride.APP
        );
Genai.UnstructuredQuery.Engine.Config.setConfigValue('chatWithFullHistory', true, ConfigOverride.APP);