C3 AI Documentation Home

Advanced Retrieval Capabilities

How to setup attribution for your REA pipeline

To setup attribution for your REA pipeline you need to set the Genai.UnstructuredQuery.Engine.Config#attributionSpec.

To do so, you can execute the below script and update the spec as per your needs:

JavaScript
// Update the PipelineConfig which is being used by the unstructured query
var configName = '<configName>';
var responseSegmenterSpec = Genai.Agent.Tool.Util.Attributor.Segmentation.Spec.make({
  tokenizer: 'google-bert/bert-base-uncased',
  textType: Genai.Agent.Tool.Util.Attributor.Segmentation.TextType.RESPONSE,
  maxSegmentTokens: 8,
  inlineCitationFormat: '\\(from(?: \\[\\d+\\](?:, \\[\\d+\\])*)+\\)',
});

var sourceSegmenterSpec = Genai.Agent.Tool.Util.Attributor.Segmentation.Spec.make({
  tokenizer: 'google-bert/bert-base-uncased',
  textType: Genai.Agent.Tool.Util.Attributor.Segmentation.TextType.SOURCE,
  maxSegmentTokens: 32,
});

var attributionSpec = Genai.Agent.Tool.Util.Attributor.Spec.make({
  responseSegmenterSpec: responseSegmenterSpec,
  sourceSegmenterSpec: sourceSegmenterSpec,
  attributeOnAllTags: true,
});

Genai.UnstructuredQuery.Engine.Config.forConfigKey(configName).setConfigValue('attributionSpec', attributionSpec);

To display the result in the UI, it is also necessary to set attributionVisibility to "full," as it is "hidden" by default

JavaScript
GenAiUiConfig.setConfigValue('attributionVisibility', 'full');

Finally in Genai.UnstructuredQuery.Engine.Config, you can set the attributionSpec using setConfigValue('attributionSpec', attributionSpec)

How to enable corroboration

Set the Genai.Agent.Tool.Util.Corroborator.Spec and Genai.Agent.Tool.Util.Attributor.Spec#runCorroboration to true.

JavaScript
// Update the PipelineConfig which is being used by the unstructured query
var configName = '<configName>';
var corroboratorSpec = Genai.Agent.Tool.Util.Corroborator.Spec.make({
  corroboratorModel: 'huggingface_entailment_model',
  corroboratorParams: { modelPath: 'cross-encoder/nli-deberta-v3-base' },
});

var attributionSpec = Genai.UnstructuredQuery.Engine.Config.forConfigKey(configName).attributionSpec;
attributionSpec = attributionSpec.withCorroboratorSpec(corroboratorSpec).withRunCorroboration(true);

Genai.UnstructuredQuery.Engine.Config.forConfigKey(configName).setConfigValue('attributionSpec', attributionSpec);

Configure a leader node with at least one T4 GPU, as corroboration uses a local model loaded into memory.

With this you should be able to see the attribution results in the UI and contradictory sources will be highlighted if any.

Was this page helpful?