Integrate an Embedder with Vector Store

Add an embedder to an entity Type so you can search its text by meaning in the C3 Agentic AI Platform. A vector store keeps vector embeddings in a field of an entity Type. An embedder is the model that turns text into those embeddings. To learn more about vector stores, see Vector Store.

A similarity search finds the items in a dataset that are most similar to a query. Instead of matching exact keywords or values, a similarity search compares vector embeddings, which are numerical representations of data. Items with similar meaning have embeddings that are numerically close.

For example, an embedder converts the query "car repair" and your stored text into embeddings. A similarity search can then find documents about "auto maintenance," "vehicle fixing," or "automobile service." Those documents have embeddings that are numerically close to the query, even though the words differ.

On this page, you choose an embedder, add a vector field that the platform fills automatically, and run a text similarity search.

The @vector annotation sets an embedder on a vector field of your own entity Type. This embedder differs from the Embedder node of an Unstructured Data Integration (UDI) pipeline in Data Fusion, which indexes documents for search. For that node, see Configure the Embedder.

Copy link to this section

How the embedder framework fits together

The platform implements embedders with Deep Java Library (DJL), a Java library for running machine learning models. The embedder framework uses the following Types:

• Entity.Embedder: The base Type for all embedders in the platform
• Djl.Engine: The list of model engines that DJL supports: PyTorch, TensorFlow, and OnnxRuntime
• Djl.Embedder: The Entity.Embedder Type that runs a model with DJL
• Expr.SimilaritySpec: The settings for a similarity search, such as the distance metric

The deepJavaLib package defines the Djl.Engine and Djl.Embedder Types and provides two preconfigured Djl.Embedder records. Both run the all-MiniLM-L6-v2 model and produce embeddings with 384 dimensions:

To learn more about the model, see the Hugging Face page all-MiniLM-L6-v2.

Copy link to this section

Choose an embedder

The @vector annotation refers to an embedder by its ID. Choose the embedder that fits your text and your model requirements:

• Use a preconfigured embedder when the all-MiniLM-L6-v2 model fits your text. A preconfigured embedder needs no setup beyond the package dependency and an annotation. Choose all-MiniLM-L6-v2 to run the model on ONNX Runtime, or all-MiniLM-L6-v2-pytorch to run it on PyTorch.
• Add your own Djl.Embedder record when you need a different model that DJL can run, for example a domain-specific or larger model. See the Use your own embedder model section.
• Define a new Entity.Embedder Type when your model does not run on DJL. See the Use your own embedder model section.

Copy link to this section

Prerequisites

To use a preconfigured embedder or the Djl.Embedder Type, your package must depend on the deepJavaLib package. In your package's .c3pkg.json file, dependencies maps each package name to a major.minor version. Add an entry for deepJavaLib:

"dependencies": {"deepJavaLib": "<major.minor>"}

Replace <major.minor> with the version of deepJavaLib that your environment provides. The preconfigured embedders are available only when your package has this dependency.

Copy link to this section

Enable automatic embedding

With automatic embedding, the platform generates the embedding for a text field and stores it in a vector field. To enable automatic embedding, add the @vector annotation to the vector field with two parameters:

• sourceField: The name of the text field to embed
• embedder: The ID of the embedder to use

The following example Type, SupportArticle, has a text field named textField and a vector field named emb. Use your own Type and field names:

entity type SupportArticle {

  /**
   * The source text
   */
  textField: string

  /**
   * Automatically generated vector embedding
   */
  @vector(sourceField='textField', embedder='all-MiniLM-L6-v2')
  emb: [!float]
}

The sourceField='textField' parameter tells the platform to generate the embedding from textField. The embedder='all-MiniLM-L6-v2' parameter refers to the preconfigured ONNX Runtime embedder. When you set embedder, you do not set the vector dimension; the embedder provides it. When an embedder fills the field, declare it as [!float]. A string field does not receive the generated embedding. Use string only for vectors you supply yourself, with dimension set, as on the Vector Store page.

The platform maintains the emb field for you, the same way it maintains a stored calculated field. When you create or update a record, the platform computes the embedding from textField and stores it in emb. The emb field is read only.

Copy link to this section

Use your own embedder model

If a preconfigured embedder does not fit your text, add your own embedder and refer to its ID in the @vector annotation.

Copy link to this section

Add a Djl.Embedder record

To run a different model with DJL, add a Djl.Embedder record to your package as seed data. The deepJavaLib package provides its preconfigured embedders the same way, in seed/Djl.Embedder/. Set the following fields:

• id: The embedder ID that you pass to the embedder parameter of @vector
• url: The URL that the model loads from
• engine: The engine that runs the model: PyTorch, TensorFlow, or OnnxRuntime
• dimension: The number of dimensions in the embeddings that the model produces

The Djl.Embedder Type declares the url and engine fields:

/**
 * Embedder implementation using Deep Java Library (DJL).
 */
entity type Djl.Embedder extends Entity.Embedder type key 'DJL' {
  /**
   * Embedder model url
   */
  url: !string serialized Url
  /**
   * Engine for the embedder model
   */
  engine: !string enum Djl.Engine
}

Copy link to this section

Define a new embedder Type

To use a model that does not run on DJL, define an entity Type that extends Entity.Embedder. The Type must provide the dimension field value and implement the encodeSource and encodeQuery methods. The Djl.Embedder Type is an example of such a Type. Entity.Embedder mixes SeedData, so you add each embedder of your Type as a seed data record with an ID that @vector can refer to.

Copy link to this section

Run a text similarity search

After you enable automatic embedding, run a text similarity search with the similarity function inside a projection or order expression. The function takes three arguments:

1. The field path of the text field, written without quotes, for example textField
2. The query text
3. An Expr.SimilaritySpec that sets the search options, for example {'metric': 'L2'}. Always pass this argument. A call with only two arguments fails with the error "Wrong number of function arguments".

The function returns a distance, so a lower value means a closer match.

To run a search:

1. Create records in your Type. The platform generates an embedding for each record:

   JavaScript

   Copy code to clipboard

   var texts = [
     'machine learning is amazing',
     'deep learning and AI',
     'natural language processing with transformers',
     'computer vision with convolutional neural networks',
     'AI in healthcare is growing',
     'machine learning in finance',
     'chatbots using natural language',
     'transformers are powerful models',
     'finance and data science',
     'neural networks and deep learning'
   ];
   texts.forEach(function (text) {
     SupportArticle.make({ textField: text }).create();
   });

2. Run the similarity search:

   JavaScript

   Copy code to clipboard

   // Define a similarity calculation between textField and the query text.
   var similarityExpr = "similarity(textField, 'machine learning applications', {'metric': 'L2'})";
   
   // Define the fields and calculations to return in the query result.
   var projection = 'id, textField, ' + similarityExpr;
   
   // Order by distance so the closest matches come first, and return two results.
   var spec = EvaluateSpec.builder().projection(projection).order(similarityExpr).limit(2).build();
   
   // Run the similarity search.
   var res = SupportArticle.evaluate(spec);

The result contains the id, textField, and distance of the two records whose textField values are closest to the query text.

Copy link to this section

See also

• Vector Store
• Expr.SimilaritySpec
• Entity.Embedder