Query Streaming

Copy link to this section

Overview

Streaming enables answers to appear in the UI as they are generated, eliminating the need to wait for full completion. By default, streaming is disabled.

There are two available streaming strategies, depending on your use case: NATIVE and DB. Some strategies are limited to specific query handlers or features, as detailed below.

Copy link to this section

Native Streaming

Native streaming uses a Stream of string to display responses immediately in the UI, achieving the fastest time to first token (TTFT). However, it has limited support:

• Compatible handlers: Works with both structured and unstructured data using Genai.UnstructuredQuery.Engine, Genai.Agent.QueryOrchestrator, or Genai.Project.QueryRouter.
  • Not supported by Genai.Agent.Dynamic.
• Compatible UI viewModes: feed only (Not compatible with chat).

To enable, run:

const streamConfig = Genai.Streaming.Config.make({ strategy: Genai.Streaming.Strategy.NATIVE });
Genai.App.ReadOnlyConfig.make().getConfig().setConfigValue('streamConfig', streamConfig);

Copy link to this section

DB Streaming

DB streaming generates LLM responses in the background using an AsyncAction, while the UI periodically fetches the stored responses. It offers slower TTFT than native streaming but is still faster than no streaming but more frequent database read and write operations.

• Compatible handlers: Genai.UnstructuredQuery.Engine, Genai.Project.QueryRouter, and Genai.Agent.QueryOrchestrator
• Compatible UI viewsModes: feed and chat.

To enable, run:

const streamConfig = Genai.Streaming.Config.make({ strategy: Genai.Streaming.Strategy.DB });
Genai.App.ReadOnlyConfig.make().getConfig().setConfigValue('streamConfig', streamConfig);

Copy link to this section

No Streaming

To disable streaming, run:

const streamConfig = Genai.Streaming.Config.make({ strategy: Genai.Streaming.Strategy.NONE });
Genai.App.ReadOnlyConfig.make().getConfig().setConfigValue('streamConfig', streamConfig);