Insert Embedding in Found Set

For every record in the found set, sends data from a source field to an embedding model and inserts the returned vector representation into a target field.

Options 

• Account Name is a text expression of the AI account for this script step to use. In the current file, use the Configure AI Account script step to set up the account and assign it this name any time before this script step runs.

• Embedding Model is the name of the model to generate embedding vectors. Specify the model name as a text expression, which is available from the model provider.

• Source Field is the text field or container field whose values this script step sends to the model for conversion to embedding vectors. If the field is empty, nothing is sent to the model and the target field is not changed.

• Target Field specifies the field to insert the returned embedding vectors into. If you specify a text field, the returned data is stored as text. If you specify a container field, then the returned data is stored as binary data, which can be smaller than when stored as text and can improve any further processing performed with the embedding vectors.

• Replace target contents replaces the target field's contents, if there is any. When this option is off, embedding data is added to the target field only if the field is empty; otherwise, the field is left unchanged. Turning on this option is useful if the contents of Source Field changes frequently.

• Parameters is a text expression for a JSON object that specifies limits on the size, number, and frequency of requests sent to the model. Use these to optimize your script depending on the amount of data, the model being used, the model provider's token limits, and the technical specifications (cores, RAM, and so on) of the machine the model is running on. See Description for details.

Originated in version 

21.0

Description 

For every record in the current found set, this script step sends the value in Source Field to the specified model for conversion to embedding vectors saved in Target Field.

One example of how you might use embedding vectors is with the Perform Semantic Find script step, which can query the vector data produced by this script step or the Insert Embedding script step.

For image as well as text embedding vectors, you can use the models supported by the open source model server provided separately with FileMaker Pro or installed with FileMaker Server. See the Claris Engineering Blog.

For the Parameters option, you can use the following keys and values in a JSON object to adjust this script step's behavior, if needed. If a key isn't specified or the Parameters option isn't used, the default values are used.

1. If using the JSONSetElement function to create the JSON object, use the indicated constant for the type parameter.

For example, this JSON object sets all the keys in the Parameters option:

{
    "MaxRecPerCall" : 40,
    "MaxRetryPerWorker" : 10,
    "MaxWaitPerRetry" : 300000,
    "TruncateTokenLimit" : 4000,
    "TruncateEnabled" : 1
}

Notes 

• To automate updating embedding vectors in a hosted FileMaker Pro file on a regular basis, you can set up a schedule on the host to run your FileMaker script without user interaction. See FileMaker script schedules in FileMaker Server Help or FileMaker Cloud Help.

Example 1

Configures an AI account, goes to the Meeting Details layout, shows all records, then gets embedding vectors for the text in the Meetings::Note field and stores it as binary data in the Meetings::Note_Embedding container field. Because the Replace target contents option is on, Meetings::Note_Embedding is updated even if it already contains data.

Configure AI Account [ Account Name: "my-account" ; Model Provider: OpenAI ; API key: "sk-RZCtpWT..." ]

Go to Layout [ "Meeting Details" (Meetings) ; Animation: None ]
Show All Records

Insert Embedding in Found Set [ Account Name: "my-account" ; Embedding Model: "text-embedding-3-small" ; Source Field: Meetings::Note ; Target Field: Meetings::Note_Embedding ; Replace target contents ]

Example 2 

Configures an AI account, goes to the Meeting Details layout, then finds the records in which the Meetings::Title field contains "Status." For this found set, gets embedding vectors for the text in the Meetings::Note field and stores it as binary data in the Meetings::Note_Embedding container field, if there's no data already present (Replace target contents is off). Also sets the Parameters option to increase the records per call, number of retries, and time between retries and to lower the token limit.

Configure AI Account [ Account Name: "my-account" ; Model Provider: OpenAI ; API key: "sk-RZCtpWT..." ]

Go to Layout [ "Meeting Details" (Meetings) ; Animation: None ]
Enter Find Mode [ Pause: Off ]
Set Field [ Meetings::Status ; "Done" ]
Perform Find [ ]

Insert Embedding in Found Set [ Account Name: "my-account" ; Embedding Model: "text-embedding-3-small" ; Source Field: Meetings::Note ; Target Field: Meetings::Note_Embedding ; 
Parameters: JSONSetElement ( "" ;
    ["MaxRecPerCall" ; 40 ; JSONNumber] ; 
    ["MaxRetryPerWorker" ; 10 ; JSONNumber] ; 
    ["MaxWaitPerRetry" ; 300000 ; JSONNumber] ;
    ["TruncateTokenLimit" ; 4000 ; JSONNumber] ;
    ["TruncateEnabled" ; 1 ; JSONNumber]
) ]

Example 3

Configures an AI account for a custom model provider, goes to the Meeting Details layout, shows all records, then for each record uses an image embedding model to get embedding vectors for the image in the Meetings::Image container field and stores it as binary data in the Meetings::Image_Embedding container field.

Configure AI Account [ Account Name: "my-account" ; Model Provider: Custom ; Endpoint: "https://my-server.example.com:8080/" ; Verify SSL Certificates ; API key: Global::API_Key ]
        
Go to Layout [ "Meeting Details" (Meetings) ; Animation: None ]
Show All Records

Insert Embedding in Found Set [ Account Name: "my-account" ; Embedding Model: "clip-ViT-L-14" ; Source Field: Meetings::Image ; Target Field: Meetings::Image_Embedding ; Replace target contents ]

Related topics 

• GetEmbedding function

• CosineSimilarity function

• Perform Semantic Find script step

• Insert Embedding script step

• Get(LastStepTokensUsed) function

• Script steps reference