Understand Document Processing Outputs in Data Fusion
After Doc AI processes the documents in a Source Collection, the results are available in your application for search, retrieval, and structured data workflows. The outputs depend on the Ingestion Strategy and its configuration. For a comparison of the available strategies, see the "Ingestion Strategies" section in Unstructured Document Processing using Doc AI in Data Fusion.
Each extracted entity has its own Entity Type node on the Data Fusion canvas. Doc AI writes the entity records to the Entity Type you select in Define Target Entity, where your application workflows can use them. Tags stay on the Source File or Source Passage they describe, and passages with embeddings serve semantic search and retrieval.
Source Passages
Passage-level extraction divides each document into Source Passages, which represent searchable units of the document's content. A document can produce multiple Source Passages, each associated with its source document.
When embeddings are enabled, Doc AI generates embeddings for the Source Passages, enabling semantic retrieval based on meaning instead of exact wording. If embeddings are disabled, Doc AI still produces Source Passages, but semantic retrieval over them is unavailable.
Document-level extraction does not produce Source Passages or embeddings.
Metadata
Doc AI can extract metadata as tags at the document and passage levels:
- Document-level tags describe the document as a whole and are stored on the Source File, the application's record of the document.
- Passage-level tags describe an individual Source Passage and are stored on that Source Passage.
You can use tags to filter documents and Source Passages based on extracted attributes. Document-level tags also support metadata search.
When you enable Dense in the Strategy's Embed Passages step, Doc AI generates embeddings for the Source Passages. Those embeddings do not include document-level or passage-level tag values.
Extracted business entities
When you configure entity extraction, Doc AI extracts business entities from the document content. Doc AI writes each entity as a record of the target Entity Type, and your application workflows can use that record.
For passage-level extraction, Doc AI also stores the extracted entity information with the Source Passage metadata. The entity keeps its link to the content it was extracted from.
Extracted entity records persist independently of their source document. Removing the source document does not remove its previously extracted entity records. Reprocessing a document that extracts the same entities again creates duplicate entity records instead of updating the existing ones.
For information about configuring entity extraction, see the "Configure Tagging and Extraction" section in Create a Document Processing Pipeline in Data Fusion.