text-embedding-3-small 和 text-embedding-3-large现已推出。它们是我们最新、性能最强的嵌入模型,使用成本更低,在多语言任务中表现更好,还提供了新参数,可用于控制嵌入向量的整体大小。什么是嵌入向量?
OpenAI 的文本嵌入向量可用于衡量文本之间的相关程度。常见用途包括:
- 搜索(按结果与查询文本的相关程度排序)
- 聚类(按相似程度对文本进行分组)
- 推荐(推荐文本内容相关的条目)
- 异常检测(找出与其他数据关联很弱的离群点)
- 多样性度量(分析相似度的分布)
- 分类(为文本分配与其最相似的标签)
嵌入向量是由浮点数组成的向量,也就是一个浮点数列表。两个向量之间的距离可以衡量它们的相关程度。距离越小,通常表示相关程度越高,反之则越低。
有关嵌入向量的价格,请参阅定价页面。每次请求按 Token 数量计费,统计范围为该请求的输入。
如何获取嵌入向量
要获取嵌入向量,请将文本字符串发送到嵌入 API 端点,并在请求中指定嵌入模型名称(例如 text-embedding-3-small):
1
2
3
4
5
6
7
8
9
10import OpenAI from "openai";
const openai = new OpenAI();
const embedding = await openai.embeddings.create({
model: "text-embedding-3-small",
input: "Your text string goes here",
encoding_format: "float",
});
console.log(embedding);1
2
3
4
5
6
7
8
9from openai import OpenAI
client = OpenAI()
response = client.embeddings.create(
input="Your text string goes here", model="text-embedding-3-small"
)
print(response.data[0].embedding)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24package main
import (
"context"
"fmt"
"github.com/openai/openai-go/v3"
)
func main() {
client := openai.NewClient()
embedding, err := client.Embeddings.New(context.Background(), openai.EmbeddingNewParams{
Model: openai.EmbeddingModelTextEmbedding3Small,
Input: openai.EmbeddingNewParamsInputUnion{
OfString: openai.String("Your text string goes here."),
},
})
if err != nil {
panic(err)
}
fmt.Println(len(embedding.Data[0].Embedding))
}1
2
3
4
5
6
7
8
9
10
11
12
13
14import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.embeddings.EmbeddingCreateParams;
var embedding =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.input("The food was delicious and the waiter...")
.build());
System.out.println(embedding.data().get(0).embedding());1
2
3
4
5
6
7
8
9
10
11using OpenAI.Embeddings;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "text-embedding-3-small";
EmbeddingClient client = new(model, key);
OpenAIEmbedding embedding = await client.GenerateEmbeddingAsync(
"The food was delicious and the waiter was friendly."
);
Console.WriteLine($"Dimensions: {embedding.ToFloats().Length}");1
2
3
4
5
6
7
8
9
10require "openai"
client = OpenAI::Client.new
response = client.embeddings.create(
model: "text-embedding-3-small",
input: "The food was delicious and the waiter..."
)
puts(response.data.fetch(0).embedding)1
2
3
4
5
6
7curl https://api.openai.com/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"input": "Your text string goes here",
"model": "text-embedding-3-small"
}'响应包含嵌入向量(一个浮点数列表)以及一些额外的元数据。您可以提取该向量,将其保存到向量数据库中,供多种应用使用。
123456789101112131415161718{
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [
-0.006929283495992422, -0.005336422007530928, -4.547132266452536e-5,
-0.024047505110502243
]
}
],
"model": "text-embedding-3-small",
"usage": {
"prompt_tokens": 5,
"total_tokens": 5
}
}
默认的嵌入向量长度分别为 1536(text-embedding-3-small)和 3072(text-embedding-3-large)。如果您希望降低向量维度,同时保留其表示概念的能力,可以传入 dimensions 参数。有关嵌入向量维度的更多说明,请参阅应用场景一节。
嵌入模型
OpenAI 提供两款功能强大的第三代嵌入模型(模型 ID 中含有 -3)。详情请参阅第三代嵌入模型的发布公告。
费用按输入 Token 数量计算。下表列出了每美元大约可处理的文本页数(假设每页约有 800 个 Token):
| 模型 | 每美元可处理的页数(约) | MTEB 评测得分 | 最大输入长度(Token) |
|---|---|---|---|
| text-embedding-3-small | 62,500 | 62.3% | 8192 |
| text-embedding-3-large | 9,615 | 64.6% | 8192 |
| text-embedding-ada-002 | 12,500 | 61.0% | 8192 |
应用场景
下面使用亚马逊食品评论数据集演示几种典型的应用场景。
获取嵌入向量
该数据集收录了截至 2012 年 10 月亚马逊用户发表的 568,454 条食品评论。我们选取其中最新的 1000 条评论作为示例。这些评论均为英文,通常带有正面或负面倾向。每条评论都包含商品 ID(ProductId)、用户 ID(UserId)、评分(Score)、评论标题(Summary)和评论正文(Text)。例如:
| 商品 ID | 用户 ID | 评分 | 标题 | 正文 |
|---|---|---|---|---|
| B001E4KFG0 | A3SGXH7AUHU8GW | 5 | 品质很好的狗粮 | 我买过好几罐 Vitality 的罐头…… |
| B00813GRG4 | A1D87F6ZCVE5NK | 1 | 和宣传的不一样 | 收到的商品上标着“特大号盐味花生”…… |
下面,我们将每条评论的标题和正文合并为一段文本。模型对合并后的文本进行编码,输出一个嵌入向量。
如果使用 Ruby,请将评论 CSV 文件保存为 data/fine_food_reviews_1k.csv,或将 REVIEWS_CSV 设为该文件的路径。请保留 Summary、Text、Score、UserId 和 ProductId 列。保存嵌入向量时也会保留这些列,供后续分析使用。
如果使用 Go,请将评论 CSV 文件保存为 data/fine_food_reviews_1k.csv。请保留 Summary、Text、Score、UserId 和 ProductId 列。程序会保留输入文件中的各列,并添加 combined 和 ada_embedding 两列,供后续分析使用。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21import { mkdir, writeFile } from "node:fs/promises";
import OpenAI from "openai";
const client = new OpenAI();
const reviews = ["A rich cup of coffee.", "A bright herbal tea."];
const response = await client.embeddings.create({
model: "text-embedding-3-small",
input: reviews.map((review) => review.replaceAll("\n", " ")),
});
const csvField = (value: string) => `"${value.replaceAll('"', '""')}"`;
const rows = response.data.map(({ embedding }, index) =>
[csvField(reviews[index]), csvField(JSON.stringify(embedding))].join(",")
);
await mkdir("output", { recursive: true });
await writeFile(
"output/embedded_1k_reviews.csv",
["combined,ada_embedding", ...rows].join("\n") + "\n"
);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17from pathlib import Path
from openai import OpenAI
client = OpenAI()
def get_embedding(text, model="text-embedding-3-small"):
text = text.replace("\n", " ")
return client.embeddings.create(input=[text], model=model).data[0].embedding
df["ada_embedding"] = df.combined.apply(
lambda x: get_embedding(x, model="text-embedding-3-small")
)
Path("output").mkdir(exist_ok=True)
df.to_csv("output/embedded_1k_reviews.csv", index=False)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88import (
"context"
"encoding/csv"
"encoding/json"
"fmt"
"log"
"os"
"slices"
"strings"
"github.com/openai/openai-go/v3"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
client := openai.NewClient()
ctx := context.Background()
input, err := os.Open("data/fine_food_reviews_1k.csv")
if err != nil {
return err
}
defer input.Close()
rows, err := csv.NewReader(input).ReadAll()
if err != nil {
return err
}
if len(rows) < 2 {
return fmt.Errorf("review CSV needs a header and at least one review")
}
headers := rows[0]
for _, name := range []string{"Summary", "Text", "Score", "UserId", "ProductId"} {
if !slices.Contains(headers, name) {
return fmt.Errorf("missing %s column", name)
}
}
summaryColumn, textColumn := slices.Index(headers, "Summary"), slices.Index(headers, "Text")
if err := os.MkdirAll("output", 0755); err != nil {
return err
}
file, err := os.Create("output/embedded_1k_reviews.csv")
if err != nil {
return err
}
defer file.Close()
writer := csv.NewWriter(file)
if err := writer.Write(append(append([]string{}, headers...), "combined", "ada_embedding")); err != nil {
return err
}
for _, row := range rows[1:] {
combined := strings.ReplaceAll(row[summaryColumn]+" "+row[textColumn], "\n", " ")
vector, err := embedding(ctx, &client, combined)
if err != nil {
return err
}
encoded, err := json.Marshal(vector)
if err != nil {
return err
}
if err := writer.Write(append(append([]string{}, row...), combined, string(encoded))); err != nil {
return err
}
}
writer.Flush()
if err := writer.Error(); err != nil {
return err
}
if err := file.Close(); err != nil {
return err
}
fmt.Println("Saved output/embedded_1k_reviews.csv")
return nil
}
func embedding(ctx context.Context, client *openai.Client, text string) ([]float64, error) {
response, err := client.Embeddings.New(ctx, openai.EmbeddingNewParams{
Model: openai.EmbeddingModelTextEmbedding3Small,
Input: openai.EmbeddingNewParamsInputUnion{OfString: openai.String(text)},
})
if err != nil {
return nil, err
}
return response.Data[0].Embedding, nil
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.embeddings.EmbeddingCreateParams;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
static String csvField(String value) {
return "\"" + value.replace("\"", "\"\"") + "\"";
}
List<String> reviews = List.of("A rich cup of coffee.", "A bright herbal tea.");
Path output = Path.of("output", "embedded_1k_reviews.csv");
Files.createDirectories(output.getParent());
try (var writer = Files.newBufferedWriter(output)) {
writer.write("combined,ada_embedding\n");
for (String review : reviews) {
var embedding =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.inputOfArrayOfStrings(List.of(review.replace("\n", " ")))
.build())
.data()
.get(0)
.embedding();
writer.write(csvField(review) + "," + csvField(embedding.toString()) + "\n");
}
}
System.out.println(output);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26using System.Text.Json;
using OpenAI.Embeddings;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "text-embedding-3-small";
EmbeddingClient client = new(model, key);
string[] reviews = ["A rich cup of coffee.", "A bright herbal tea."];
Directory.CreateDirectory("output");
using StreamWriter writer = new("output/embedded_1k_reviews.csv");
await writer.WriteLineAsync("combined,ada_embedding");
foreach (string review in reviews)
{
float[] vector = await EmbedAsync(client, review.Replace("\n", " ", StringComparison.Ordinal));
string encoded = JsonSerializer.Serialize(vector);
await writer.WriteLineAsync($"{CsvField(review)},{CsvField(encoded)}");
}
Console.WriteLine("Saved output/embedded_1k_reviews.csv");
static async Task<float[]> EmbedAsync(EmbeddingClient client, string text)
{
OpenAIEmbedding result = await client.GenerateEmbeddingAsync(text);
return result.ToFloats().ToArray();
}
static string CsvField(string value) => "\"" + value.Replace("\"", "\"\"", StringComparison.Ordinal) + "\"";1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21require "csv"
require "fileutils"
require "json"
require "openai"
client = OpenAI::Client.new
table = CSV.read(ENV.fetch("REVIEWS_CSV", "data/fine_food_reviews_1k.csv"))
headers = table.shift or raise ArgumentError, "Review CSV is empty"
reviews = table.map { |fields| headers.zip(fields).to_h }
combined = reviews.map { |row| "#{row.fetch("Summary")} #{row.fetch("Text")}".tr("\n", " ") }
response = client.embeddings.create(model: "text-embedding-3-small", input: combined)
FileUtils.mkdir_p("output")
CSV.open("output/embedded_1k_reviews.csv", "w") do |csv|
csv << headers + ["combined", "ada_embedding"]
response.data.each do |embedding|
row = reviews.fetch(embedding.index)
csv << headers.map { |header| row.fetch(header) } + [combined.fetch(embedding.index), JSON.generate(embedding.embedding)]
end
end每个 Go 示例都是一个完整的程序,都会读取 output/embedded_1k_reviews.csv。请在您的 Go 模块中分别运行各个程序。文件中 ada_embedding 列的值必须是 JSON 数组,Score 列的值必须是整数。计算分组平均值还需要 UserId 和 ProductId 列。拆分数据集时使用固定的随机种子,但不同运行之间,随机森林的预测结果和 t-SNE 坐标可能有所不同。下文的指标仅对应链接中的 Python 笔记本,不适用于这些 Go 程序。
如果使用 Ruby,请使用 Ruby 3.3 并安装示例项目的依赖项。将加载和分析的代码片段追加到同一个 Ruby 文件中。运行任一森林模型示例之前,请先运行拆分数据集的代码。下文的指标来自链接中的 Python 笔记本,具体结果取决于数据集和实现方式。
要从已保存的文件中加载数据,可以运行以下代码:
1
2
3
4
5from ast import literal_eval
import pandas as pd
df = pd.read_csv("output/embedded_1k_reviews.csv")
df["ada_embedding"] = df.ada_embedding.apply(literal_eval).apply(np.array)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"log"
"os"
"strconv"
)
type review struct {
Vector []float64
Score int
UserID, ProductID string
}
func loadReviews(path string) ([]review, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
reader := csv.NewReader(file)
header, err := reader.Read()
if err != nil {
return nil, err
}
columns := map[string]int{}
for i, name := range header {
columns[name] = i
}
for _, name := range []string{"ada_embedding", "Score"} {
if _, ok := columns[name]; !ok {
return nil, fmt.Errorf("missing %s column", name)
}
}
rows := []review{}
dimensions := 0
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
var value review
if err := json.Unmarshal([]byte(record[columns["ada_embedding"]]), &value.Vector); err != nil {
return nil, err
}
if len(value.Vector) == 0 {
return nil, fmt.Errorf("empty embedding")
}
if dimensions == 0 {
dimensions = len(value.Vector)
}
if len(value.Vector) != dimensions {
return nil, fmt.Errorf("inconsistent embedding dimensions")
}
value.Score, err = strconv.Atoi(record[columns["Score"]])
if err != nil {
return nil, err
}
if column, ok := columns["UserId"]; ok {
value.UserID = record[column]
}
if column, ok := columns["ProductId"]; ok {
value.ProductID = record[column]
}
rows = append(rows, value)
}
if len(rows) == 0 {
return nil, fmt.Errorf("no reviews")
}
return rows, nil
}
func main() {
rows, err := loadReviews("output/embedded_1k_reviews.csv")
if err != nil {
log.Fatal(err)
}
result := map[string]any{"rows": len(rows), "first": rows[0].Vector, "last": rows[len(rows)-1].Vector}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
log.Fatal(err)
}
}1
2
3
4
5
6
7
8# standard:disable Lint/UselessAssignment -- Preserve named results for the next workflow step.
require "csv"
require "json"
rows = CSV.read("output/embedded_1k_reviews.csv", headers: true).map do |row|
row.to_h.merge("ada_embedding" => JSON.parse(row.fetch("ada_embedding")))
end
# standard:enable Lint/UselessAssignment与低维嵌入向量相比,使用高维嵌入向量(例如将其存入向量存储以供检索)通常成本更高,也需要更多算力、内存和存储空间。
我们的两款新嵌入模型在训练时都采用了一种技术,让开发者可以在性能和使用成本之间做出权衡。具体来说,开发者可以传入 dimensions API 参数来缩短嵌入向量(即去掉序列末尾的一部分数值),同时保留向量表示概念的能力。例如,在 MTEB 基准测试中,即使将 text-embedding-3-large 的嵌入向量缩短至 256 维,其表现仍优于完整长度为 1536 维的 text-embedding-ada-002 嵌入向量。有关调整维度如何影响性能,请参阅第三代嵌入模型发布公告。
通常建议在创建嵌入向量时使用 dimensions 参数。在某些情况下,您可能需要在生成向量后调整其维度。手动调整维度时,请务必按以下示例对调整后的向量进行归一化。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17import OpenAI from "openai";
const client = new OpenAI();
const response = await client.embeddings.create({
model: "text-embedding-3-small",
input: "Testing 123",
encoding_format: "float",
});
const shortened = response.data[0].embedding.slice(0, 256);
const magnitude = Math.hypot(...shortened);
const normalized = shortened.map((value) =>
magnitude === 0 ? 0 : value / magnitude
);
console.log(normalized);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26from openai import OpenAI
import numpy as np
client = OpenAI()
def normalize_l2(x):
x = np.array(x)
if x.ndim == 1:
norm = np.linalg.norm(x)
if norm == 0:
return x
return x / norm
else:
norm = np.linalg.norm(x, 2, axis=1, keepdims=True)
return np.where(norm == 0, x, x / norm)
response = client.embeddings.create(
model="text-embedding-3-small", input="Testing 123", encoding_format="float"
)
cut_dim = response.data[0].embedding[:256]
norm_dim = normalize_l2(cut_dim)
print(norm_dim)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22response, err := client.Embeddings.New(context.Background(), openai.EmbeddingNewParams{
Model: openai.EmbeddingModelTextEmbedding3Small,
Input: openai.EmbeddingNewParamsInputUnion{OfString: openai.String("Testing 123")},
EncodingFormat: openai.EmbeddingNewParamsEncodingFormatFloat,
})
if err != nil {
panic(err)
}
if len(response.Data) != 1 || len(response.Data[0].Embedding) < 256 {
panic("expected an embedding with at least 256 dimensions")
}
shortened := append([]float64(nil), response.Data[0].Embedding[:256]...)
magnitude := 0.0
for _, value := range shortened {
magnitude += value * value
}
magnitude = math.Sqrt(magnitude)
if magnitude != 0 {
for i := range shortened {
shortened[i] /= magnitude
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.embeddings.EmbeddingCreateParams;
import java.util.List;
private static List<Double> normalizeL2(List<Float> embedding) {
double norm =
Math.sqrt(embedding.stream().mapToDouble(value -> value.doubleValue() * value).sum());
return embedding.stream().map(value -> norm == 0 ? 0.0 : value / norm).toList();
}
var embedding =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.input("Testing 123")
.encodingFormat(EmbeddingCreateParams.EncodingFormat.FLOAT)
.build());
List<Float> shortened = embedding.data().get(0).embedding().subList(0, 256);
System.out.println(normalizeL2(shortened));1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16using OpenAI.Embeddings;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "text-embedding-3-small";
EmbeddingClient client = new(model, key);
OpenAIEmbedding embedding = await client.GenerateEmbeddingAsync("Testing 123");
float[] shortened = embedding.ToFloats().Span[..256].ToArray();
double magnitude = Math.Sqrt(shortened.Sum(value => (double)value * value));
double[] normalized =
magnitude == 0
? shortened.Select(value => (double)value).ToArray()
: shortened.Select(value => value / magnitude).ToArray();
Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(normalized));1
2
3
4
5
6
7
8
9
10
11
12
13
14
15require "openai"
client = OpenAI::Client.new
response = client.embeddings.create(
model: "text-embedding-3-small",
input: "Testing 123",
encoding_format: :float
)
shortened = response.data.fetch(0).embedding.first(256)
magnitude = Math.sqrt(shortened.sum { |value| value**2 })
normalized = shortened.map { |value| magnitude.zero? ? 0 : value / magnitude }
puts(normalized)动态调整维度可以让嵌入向量适用于更多场景。例如,即使向量存储最多只支持 1024 维的嵌入向量,开发者仍可使用我们性能最佳的嵌入模型 text-embedding-3-large。将 dimensions API 参数设为 1024,即可把向量从 3072 维缩短至 1024 维,以一定的精度损失换取更小的向量。
回答用户问题时,模型常常需要用到训练数据中没有的关键事实和信息。如下面的示例所示,一种解决办法是将补充信息放入模型的上下文窗口。这种方法适用于许多场景,但会增加 Token 费用。在这个笔记本中,我们探讨这种方法与基于嵌入向量的搜索各自的利弊。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25import OpenAI from "openai";
const client = new OpenAI();
const article =
"At the 2022 Winter Olympics, Great Britain won women's curling and Sweden won men's curling.";
const question = `Use the article below to answer the question. If the answer cannot be found, say "I don't know."
Article:
${article}
Question: Which athletes won the gold medal in curling at the 2022 Winter Olympics?`;
const response = await client.chat.completions.create({
model: "gpt-4.1-mini",
messages: [
{
role: "system",
content: "You answer questions about the 2022 Winter Olympics.",
},
{ role: "user", content: question },
],
temperature: 0,
});
console.log(response.choices[0].message.content);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22query = f"""Use the below article on the 2022 Winter Olympics to answer the subsequent question. If the answer cannot be found, write "I don't know."
Article:
\"\"\"
{wikipedia_article_on_curling}
\"\"\"
Question: Which athletes won the gold medal in curling at the 2022 Winter Olympics?"""
response = client.chat.completions.create(
messages=[
{
"role": "system",
"content": "You answer questions about the 2022 Winter Olympics.",
},
{"role": "user", "content": query},
],
model=GPT_MODEL,
temperature=0,
)
print(response.choices[0].message.content)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27import (
"context"
"fmt"
"log"
"github.com/openai/openai-go/v3"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
client := openai.NewClient()
completion, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
Model: "gpt-4.1-mini", Temperature: openai.Float(0), Messages: []openai.ChatCompletionMessageParamUnion{
openai.SystemMessage("You answer questions about the 2022 Winter Olympics."),
openai.UserMessage("Use the article to answer the question. If the answer cannot be found, write \"I don't know.\"\n\nArticle: At the 2022 Winter Olympics, Great Britain won women's curling and Sweden won men's curling.\n\nQuestion: Which athletes won the gold medal in curling at the 2022 Winter Olympics?")},
})
if err != nil {
return err
}
fmt.Println(completion.Choices[0].Message.Content)
return nil
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.chat.completions.ChatCompletionCreateParams;
String article =
"At the 2022 Winter Olympics, Great Britain won women's curling and Sweden won men's curling.";
String question =
"Use the below article on the 2022 Winter Olympics to answer the subsequent question. "
+ "If the answer cannot be found, write \"I don't know.\"\n\n"
+ "Article:\n"
+ article
+ "\n\nQuestion: Which athletes won the gold medal in curling at the 2022 Winter Olympics?";
ChatCompletionCreateParams params =
ChatCompletionCreateParams.builder()
.model("gpt-4.1-mini")
.addSystemMessage("You answer questions about the 2022 Winter Olympics.")
.addUserMessage(question)
.temperature(0)
.build();
client.chat().completions().create(params).choices().stream()
.flatMap(choice -> choice.message().content().stream())
.forEach(System.out::println);1
2
3
4
5
6
7
8
9
10
11
12using OpenAI.Chat;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "gpt-4.1-mini";
ChatClient client = new(model, key);
string article = "At the 2022 Winter Olympics, Great Britain won women's curling and Sweden won men's curling.";
string query = $"Use the article to answer the question. If the answer cannot be found, write I don't know.\nArticle: {article}\nQuestion: Which athletes won the gold medal in curling at the 2022 Winter Olympics?";
ChatCompletion result = await client.CompleteChatAsync(
[new SystemChatMessage("You answer questions about the 2022 Winter Olympics."), new UserChatMessage(query)],
new ChatCompletionOptions { Temperature = 0 });
Console.WriteLine(result.Content[0].Text);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29require "openai"
client = OpenAI::Client.new
article = "At the 2022 Winter Olympics, Great Britain won women's curling and Sweden won men's curling."
question = <<~QUESTION
Use the article below to answer the question. If the answer cannot be found, say "I don't know."
Article:
#{article}
Question: Which athletes won the gold medal in curling at the 2022 Winter Olympics?
QUESTION
response = client.chat.completions.create(
model: "gpt-4.1-mini",
messages: [
{
role: :system,
content: "You answer questions about the 2022 Winter Olympics."
},
{
role: :user,
content: question
}
],
temperature: 0
)
puts(response.choices.fetch(0).message.content)要检索最相关的文档,我们计算查询向量与每个文档的嵌入向量之间的余弦相似度,然后返回得分最高的文档。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32import OpenAI from "openai";
const client = new OpenAI();
const reviews = [
"A rich cup of coffee.",
"Smooth beans in tomato sauce.",
"Dark chocolate with orange.",
];
const { data } = await client.embeddings.create({
model: "text-embedding-3-small",
input: [...reviews, "delicious beans"],
});
const query = data[reviews.length].embedding;
const similarity = (embedding: number[]) => {
const dotProduct = embedding.reduce(
(total, value, index) => total + value * query[index],
0
);
return dotProduct / (Math.hypot(...embedding) * Math.hypot(...query));
};
const results = reviews
.map((review, index) => ({
review,
score: similarity(data[index].embedding),
}))
.sort((left, right) => right.score - left.score)
.slice(0, 3);
console.log(results);1
2
3
4
5
6
7
8
9
10def search_reviews(df, product_description, n=3, pprint=True):
embedding = get_embedding(product_description, model="text-embedding-3-small")
df["similarities"] = df.ada_embedding.apply(
lambda x: cosine_similarity(x, embedding)
)
res = df.sort_values("similarities", ascending=False).head(n)
return res
res = search_reviews(df, "delicious beans", n=3)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73import (
"context"
"fmt"
"log"
"math"
"sort"
"github.com/openai/openai-go/v3"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
client := openai.NewClient()
ctx := context.Background()
texts := []string{"A rich cup of coffee.", "Crunchy crackers with sea salt.", "Dark chocolate with orange.", "A bright herbal tea.", "Smooth beans in tomato sauce.", "A mild cheese with herbs.", "Spicy roasted nuts.", "A crisp sparkling water."}
vectors := make([][]float64, len(texts))
for i, text := range texts {
vector, err := embedding(ctx, &client, text)
if err != nil {
return err
}
vectors[i] = vector
}
query, err := embedding(ctx, &client, "delicious beans")
if err != nil {
return err
}
matches := nearest(query, vectors)
for _, match := range matches[:min(3, len(matches))] {
fmt.Printf("%0.3f: %s\n", match.Similarity, texts[match.Index])
}
return nil
}
func embedding(ctx context.Context, client *openai.Client, text string) ([]float64, error) {
response, err := client.Embeddings.New(ctx, openai.EmbeddingNewParams{
Model: openai.EmbeddingModelTextEmbedding3Small,
Input: openai.EmbeddingNewParamsInputUnion{OfString: openai.String(text)},
})
if err != nil {
return nil, err
}
return response.Data[0].Embedding, nil
}
func cosineSimilarity(a, b []float64) float64 {
var dot, left, right float64
for i := range a {
dot += a[i] * b[i]
left += a[i] * a[i]
right += b[i] * b[i]
}
return dot / math.Sqrt(left*right)
}
type match struct {
Index int
Similarity float64
}
func nearest(query []float64, vectors [][]float64) []match {
matches := make([]match, len(vectors))
for i, vector := range vectors {
matches[i] = match{Index: i, Similarity: cosineSimilarity(query, vector)}
}
sort.SliceStable(matches, func(i, j int) bool { return matches[i].Similarity > matches[j].Similarity })
return matches
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.embeddings.EmbeddingCreateParams;
import java.util.Comparator;
import java.util.List;
import java.util.stream.IntStream;
List<String> reviews =
List.of(
"A rich cup of coffee.",
"Smooth beans in tomato sauce.",
"Dark chocolate with orange.");
var reviewEmbeddings =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.inputOfArrayOfStrings(reviews)
.build())
.data();
List<Float> query =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.inputOfArrayOfStrings(List.of("delicious beans"))
.build())
.data()
.get(0)
.embedding();
IntStream.range(0, reviews.size())
.boxed()
.sorted(
Comparator.comparingDouble(
(Integer index) ->
cosineSimilarity(query, reviewEmbeddings.get(index).embedding()))
.reversed())
.limit(3)
.map(reviews::get)
.forEach(System.out::println);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33using OpenAI.Embeddings;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "text-embedding-3-small";
EmbeddingClient client = new(model, key);
string[] texts = ["A rich cup of coffee.", "Crunchy crackers with sea salt.", "Dark chocolate with orange.", "A bright herbal tea.", "Smooth beans in tomato sauce.", "A mild cheese with herbs.", "Spicy roasted nuts.", "A crisp sparkling water."];
OpenAIEmbeddingCollection batch = await client.GenerateEmbeddingsAsync(texts);
float[][] vectors = batch.Select(item => item.ToFloats().ToArray()).ToArray();
float[] query = await EmbedAsync(client, "delicious beans");
var ranked = vectors.Select((vector, index) => new { Index = index, Similarity = CosineSimilarity(query, vector) }).OrderByDescending(match => match.Similarity);
foreach (var match in ranked.Take(3))
{
Console.WriteLine($"{match.Similarity:F3}: {texts[match.Index]}");
}
static async Task<float[]> EmbedAsync(EmbeddingClient client, string text)
{
OpenAIEmbedding result = await client.GenerateEmbeddingAsync(text);
return result.ToFloats().ToArray();
}
static double CosineSimilarity(float[] left, float[] right)
{
double dot = 0, leftNorm = 0, rightNorm = 0;
for (int i = 0; i < left.Length; i++)
{
dot += (double)left[i] * right[i];
leftNorm += (double)left[i] * left[i];
rightNorm += (double)right[i] * right[i];
}
return dot / Math.Sqrt(leftNorm * rightNorm);
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30require "openai"
client = OpenAI::Client.new
reviews = [
"A rich cup of coffee.",
"Smooth beans in tomato sauce.",
"Dark chocolate with orange."
]
response = client.embeddings.create(
model: "text-embedding-3-small",
input: reviews + ["delicious beans"]
)
query = response.data.fetch(-1).embedding
similarity = lambda do |embedding|
dot_product = embedding.zip(query).sum { |value, query_value| value * query_value }
magnitude = Math.sqrt(embedding.sum { |value| value**2 })
query_magnitude = Math.sqrt(query.sum { |value| value**2 })
dot_product / (magnitude * query_magnitude)
end
results = reviews.map.with_index do |review, index|
{
review: review,
score: similarity.call(response.data.fetch(index).embedding)
}
end.sort_by { |result| -result.fetch(:score) }.first(3)
puts(results)代码搜索的原理与基于嵌入向量的文本搜索相似。示例演示了如何从指定代码仓库的所有 Python 文件中提取 Python 函数,然后使用 text-embedding-3-small 模型为每个函数生成向量并建立索引。
搜索代码时,我们先使用同一模型将自然语言查询转换为嵌入向量,再计算查询向量与每个函数向量之间的余弦相似度。余弦相似度最高的结果最相关。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30import OpenAI from "openai";
const client = new OpenAI();
const functions = [
"function add(a, b) { return a + b; }",
"function complete(prompt) { return prompt; }",
];
const { data } = await client.embeddings.create({
model: "text-embedding-3-small",
input: [...functions, "Completions API tests"],
});
const query = data[functions.length].embedding;
const similarity = (embedding: number[]) => {
const dotProduct = embedding.reduce(
(total, value, index) => total + value * query[index],
0
);
return dotProduct / (Math.hypot(...embedding) * Math.hypot(...query));
};
const results = functions
.map((source, index) => ({
source,
score: similarity(data[index].embedding),
}))
.sort((left, right) => right.score - left.score);
console.log(results);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16df["code_embedding"] = df["code"].apply(
lambda x: get_embedding(x, model="text-embedding-3-small")
)
def search_functions(df, code_query, n=3, pprint=True, n_lines=7):
embedding = get_embedding(code_query, model="text-embedding-3-small")
df["similarities"] = df.code_embedding.apply(
lambda x: cosine_similarity(x, embedding)
)
res = df.sort_values("similarities", ascending=False).head(n)
return res
res = search_functions(df, "Completions API tests", n=3)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73import (
"context"
"fmt"
"log"
"math"
"sort"
"github.com/openai/openai-go/v3"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
client := openai.NewClient()
ctx := context.Background()
texts := []string{"def add(a, b): return a + b", "def complete(prompt): return prompt"}
vectors := make([][]float64, len(texts))
for i, text := range texts {
vector, err := embedding(ctx, &client, text)
if err != nil {
return err
}
vectors[i] = vector
}
query, err := embedding(ctx, &client, "Completions API tests")
if err != nil {
return err
}
matches := nearest(query, vectors)
for _, match := range matches[:min(3, len(matches))] {
fmt.Printf("%0.3f: %s\n", match.Similarity, texts[match.Index])
}
return nil
}
func embedding(ctx context.Context, client *openai.Client, text string) ([]float64, error) {
response, err := client.Embeddings.New(ctx, openai.EmbeddingNewParams{
Model: openai.EmbeddingModelTextEmbedding3Small,
Input: openai.EmbeddingNewParamsInputUnion{OfString: openai.String(text)},
})
if err != nil {
return nil, err
}
return response.Data[0].Embedding, nil
}
func cosineSimilarity(a, b []float64) float64 {
var dot, left, right float64
for i := range a {
dot += a[i] * b[i]
left += a[i] * a[i]
right += b[i] * b[i]
}
return dot / math.Sqrt(left*right)
}
type match struct {
Index int
Similarity float64
}
func nearest(query []float64, vectors [][]float64) []match {
matches := make([]match, len(vectors))
for i, vector := range vectors {
matches[i] = match{Index: i, Similarity: cosineSimilarity(query, vector)}
}
sort.SliceStable(matches, func(i, j int) bool { return matches[i].Similarity > matches[j].Similarity })
return matches
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.embeddings.EmbeddingCreateParams;
import java.util.Comparator;
import java.util.List;
import java.util.stream.IntStream;
List<String> functions =
List.of("def add(a, b): return a + b", "def complete(prompt): return prompt");
var functionEmbeddings =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.inputOfArrayOfStrings(functions)
.build())
.data();
List<Float> query =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.input("Completions API tests")
.build())
.data()
.get(0)
.embedding();
IntStream.range(0, functions.size())
.boxed()
.sorted(
Comparator.comparingDouble(
(Integer index) ->
cosineSimilarity(query, functionEmbeddings.get(index).embedding()))
.reversed())
.map(functions::get)
.forEach(System.out::println);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36using OpenAI.Embeddings;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "text-embedding-3-small";
EmbeddingClient client = new(model, key);
string[] texts = ["def add(a, b): return a + b", "def complete(prompt): return prompt"];
List<float[]> vectors = [];
foreach (string text in texts)
{
vectors.Add(await EmbedAsync(client, text));
}
float[] query = await EmbedAsync(client, "Completions API tests");
var ranked = vectors.Select((vector, index) => new { Index = index, Similarity = CosineSimilarity(query, vector) }).OrderByDescending(match => match.Similarity);
foreach (var match in ranked.Take(3))
{
Console.WriteLine($"{match.Similarity:F3}: {texts[match.Index]}");
}
static async Task<float[]> EmbedAsync(EmbeddingClient client, string text)
{
OpenAIEmbedding result = await client.GenerateEmbeddingAsync(text);
return result.ToFloats().ToArray();
}
static double CosineSimilarity(float[] left, float[] right)
{
double dot = 0, leftNorm = 0, rightNorm = 0;
for (int i = 0; i < left.Length; i++)
{
dot += (double)left[i] * right[i];
leftNorm += (double)left[i] * left[i];
rightNorm += (double)right[i] * right[i];
}
return dot / Math.Sqrt(leftNorm * rightNorm);
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29require "openai"
client = OpenAI::Client.new
functions = [
"function add(a, b) { return a + b; }",
"function complete(prompt) { return prompt; }"
]
response = client.embeddings.create(
model: "text-embedding-3-small",
input: functions + ["Completions API tests"]
)
query = response.data.fetch(-1).embedding
similarity = lambda do |embedding|
dot_product = embedding.zip(query).sum { |value, query_value| value * query_value }
magnitude = Math.sqrt(embedding.sum { |value| value**2 })
query_magnitude = Math.sqrt(query.sum { |value| value**2 })
dot_product / (magnitude * query_magnitude)
end
results = functions.map.with_index do |source, index|
{
source: source,
score: similarity.call(response.data.fetch(index).embedding)
}
end.sort_by { |result| -result.fetch(:score) }
puts(results)嵌入向量之间的距离越短,相似度就越高,因此可以用嵌入向量进行推荐。
下面是一个简单的推荐函数。它接收一个字符串列表和一个“源”字符串,计算它们的嵌入向量,然后按与源字符串的相似度从高到低返回列表中的字符串。链接中的笔记本将这个函数的一个版本应用于 AG 新闻数据集(从中抽取了 2,000 条新闻简介),找出与任意给定新闻最相似的 5 篇文章。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28import OpenAI from "openai";
const client = new OpenAI();
const strings = [
"A cheetah is a fast land animal.",
"A peregrine falcon is a fast bird.",
"A tortoise moves slowly.",
];
const { data } = await client.embeddings.create({
model: "text-embedding-3-small",
input: strings,
});
const query = data[0].embedding;
const recommendations = data
.map(({ embedding }, index) => {
const dotProduct = embedding.reduce(
(total, value, dimension) => total + value * query[dimension],
0
);
const similarity =
dotProduct / (Math.hypot(...embedding) * Math.hypot(...query));
return { index, text: strings[index], similarity };
})
.sort((left, right) => right.similarity - left.similarity);
console.log(recommendations);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23def recommendations_from_strings(
strings: list[str],
index_of_source_string: int,
model="text-embedding-3-small",
) -> list[int]:
"""Return nearest neighbors of a given string."""
# get embeddings for all strings
embeddings = [embedding_from_string(string, model=model) for string in strings]
# get the embedding of the source string
query_embedding = embeddings[index_of_source_string]
# get distances between the source embedding and other embeddings (function from embeddings_utils.py)
distances = distances_from_embeddings(
query_embedding, embeddings, distance_metric="cosine"
)
# get indices of nearest neighbors (function from embeddings_utils.py)
indices_of_nearest_neighbors = indices_of_nearest_neighbors_from_distances(
distances
)
return indices_of_nearest_neighbors1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70import (
"context"
"fmt"
"log"
"math"
"sort"
"github.com/openai/openai-go/v3"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
client := openai.NewClient()
ctx := context.Background()
texts := []string{"A cheetah is a fast land animal.", "A peregrine falcon is a fast bird.", "A tortoise moves slowly."}
vectors := make([][]float64, len(texts))
for i, text := range texts {
vector, err := embedding(ctx, &client, text)
if err != nil {
return err
}
vectors[i] = vector
}
query := vectors[0]
matches := nearest(query, vectors)
for _, match := range matches {
fmt.Println(match.Index)
}
return nil
}
func embedding(ctx context.Context, client *openai.Client, text string) ([]float64, error) {
response, err := client.Embeddings.New(ctx, openai.EmbeddingNewParams{
Model: openai.EmbeddingModelTextEmbedding3Small,
Input: openai.EmbeddingNewParamsInputUnion{OfString: openai.String(text)},
})
if err != nil {
return nil, err
}
return response.Data[0].Embedding, nil
}
func cosineSimilarity(a, b []float64) float64 {
var dot, left, right float64
for i := range a {
dot += a[i] * b[i]
left += a[i] * a[i]
right += b[i] * b[i]
}
return dot / math.Sqrt(left*right)
}
type match struct {
Index int
Similarity float64
}
func nearest(query []float64, vectors [][]float64) []match {
matches := make([]match, len(vectors))
for i, vector := range vectors {
matches[i] = match{Index: i, Similarity: cosineSimilarity(query, vector)}
}
sort.SliceStable(matches, func(i, j int) bool { return matches[i].Similarity > matches[j].Similarity })
return matches
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.embeddings.EmbeddingCreateParams;
import java.util.Comparator;
import java.util.List;
import java.util.stream.IntStream;
List<String> strings =
List.of(
"A cheetah is a fast land animal.",
"A peregrine falcon is a fast bird.",
"A tortoise moves slowly.");
var embeddings =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.inputOfArrayOfStrings(strings)
.build())
.data();
List<Float> query = embeddings.get(0).embedding();
var nearestNeighbors =
IntStream.range(0, embeddings.size())
.boxed()
.sorted(
Comparator.comparingDouble(
(Integer index) -> {
List<Float> candidate = embeddings.get(index).embedding();
double dotProduct = 0;
double queryMagnitude = 0;
double candidateMagnitude = 0;
for (int dimension = 0; dimension < query.size(); dimension++) {
double queryValue = query.get(dimension);
double candidateValue = candidate.get(dimension);
dotProduct += queryValue * candidateValue;
queryMagnitude += queryValue * queryValue;
candidateMagnitude += candidateValue * candidateValue;
}
return 1 - dotProduct / Math.sqrt(queryMagnitude * candidateMagnitude);
}))
.toList();
System.out.println(nearestNeighbors);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36using OpenAI.Embeddings;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "text-embedding-3-small";
EmbeddingClient client = new(model, key);
string[] texts = ["A cheetah is a fast land animal.", "A peregrine falcon is a fast bird.", "A tortoise moves slowly."];
List<float[]> vectors = [];
foreach (string text in texts)
{
vectors.Add(await EmbedAsync(client, text));
}
float[] query = vectors[0];
var ranked = vectors.Select((vector, index) => new { Index = index, Similarity = CosineSimilarity(query, vector) }).OrderByDescending(match => match.Similarity);
foreach (var match in ranked)
{
Console.WriteLine(match.Index);
}
static async Task<float[]> EmbedAsync(EmbeddingClient client, string text)
{
OpenAIEmbedding result = await client.GenerateEmbeddingAsync(text);
return result.ToFloats().ToArray();
}
static double CosineSimilarity(float[] left, float[] right)
{
double dot = 0, leftNorm = 0, rightNorm = 0;
for (int i = 0; i < left.Length; i++)
{
dot += (double)left[i] * right[i];
leftNorm += (double)left[i] * left[i];
rightNorm += (double)right[i] * right[i];
}
return dot / Math.Sqrt(leftNorm * rightNorm);
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31require "openai"
client = OpenAI::Client.new
strings = [
"A cheetah is a fast land animal.",
"A peregrine falcon is a fast bird.",
"A tortoise moves slowly."
]
response = client.embeddings.create(
model: "text-embedding-3-small",
input: strings
)
query = response.data.fetch(0).embedding
similarity = lambda do |embedding|
dot_product = embedding.zip(query).sum { |value, query_value| value * query_value }
magnitude = Math.sqrt(embedding.sum { |value| value**2 })
query_magnitude = Math.sqrt(query.sum { |value| value**2 })
dot_product / (magnitude * query_magnitude)
end
recommendations = response.data.map.with_index do |embedding, index|
{
index: index,
text: strings.fetch(index),
similarity: similarity.call(embedding.embedding)
}
end.sort_by { |recommendation| -recommendation.fetch(:similarity) }
puts(recommendations)嵌入向量的维度随底层模型的复杂程度而变化。为了将这些高维数据可视化,我们使用 t-SNE 算法将数据转换为二维表示。
我们根据评论者给出的星级为每条评论着色:
- 1 星:红色
- 2 星:深橙色
- 3 星:金色
- 4 星:青绿色
- 5 星:深绿色

从图中大致可以看出 3 个簇,其中一个簇以负面评论为主。
Go 绘图示例使用 github.com/danaugrs/go-tsne/tsne(版本 v0.0.0-20220306155740-2250969e057f)、gonum.org/v1/gonum(版本 v0.16.0)和 gonum.org/v1/plot(版本 v0.16.0)。程序将图像保存为 local-plot.png,并打印降维后的坐标以及各评分对应的颜色索引。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24from ast import literal_eval
import numpy as np
import pandas as pd
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
import matplotlib
df = pd.read_csv("output/embedded_1k_reviews.csv")
matrix = np.array(df.ada_embedding.apply(literal_eval).to_list())
# Create a t-SNE model and transform the data
tsne = TSNE(
n_components=2, perplexity=15, random_state=42, init="random", learning_rate=200
)
vis_dims = tsne.fit_transform(matrix)
colors = ["red", "darkorange", "gold", "turquoise", "darkgreen"]
x = [x for x, y in vis_dims]
y = [y for x, y in vis_dims]
color_indices = df.Score.values - 1
colormap = matplotlib.colors.ListedColormap(colors)
plt.scatter(x, y, c=color_indices, cmap=colormap, alpha=0.3)
plt.title("Amazon ratings visualized in language using t-SNE")1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"github.com/danaugrs/go-tsne/tsne"
"gonum.org/v1/gonum/mat"
"gonum.org/v1/plot"
"gonum.org/v1/plot/plotter"
"gonum.org/v1/plot/vg"
"gonum.org/v1/plot/vg/draw"
"image/color"
"io"
"log"
"math"
"os"
"strconv"
)
type review struct {
Vector []float64
Score int
UserID, ProductID string
}
func loadReviews(path string) ([]review, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
reader := csv.NewReader(file)
header, err := reader.Read()
if err != nil {
return nil, err
}
columns := map[string]int{}
for i, name := range header {
columns[name] = i
}
for _, name := range []string{"ada_embedding", "Score"} {
if _, ok := columns[name]; !ok {
return nil, fmt.Errorf("missing %s column", name)
}
}
rows := []review{}
dimensions := 0
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
var value review
if err := json.Unmarshal([]byte(record[columns["ada_embedding"]]), &value.Vector); err != nil {
return nil, err
}
if len(value.Vector) == 0 {
return nil, fmt.Errorf("empty embedding")
}
if dimensions == 0 {
dimensions = len(value.Vector)
}
if len(value.Vector) != dimensions {
return nil, fmt.Errorf("inconsistent embedding dimensions")
}
value.Score, err = strconv.Atoi(record[columns["Score"]])
if err != nil {
return nil, err
}
if column, ok := columns["UserId"]; ok {
value.UserID = record[column]
}
if column, ok := columns["ProductId"]; ok {
value.ProductID = record[column]
}
rows = append(rows, value)
}
if len(rows) == 0 {
return nil, fmt.Errorf("no reviews")
}
return rows, nil
}
func main() {
rows, err := loadReviews("output/embedded_1k_reviews.csv")
if err != nil {
log.Fatal(err)
}
if len(rows) < 3 {
log.Fatal("need at least three reviews")
}
values := []float64{}
for _, row := range rows {
values = append(values, row.Vector...)
}
matrix := mat.NewDense(len(rows), len(rows[0].Vector), values)
perplexity := math.Min(15, float64(len(rows)-1))
embedded := tsne.NewTSNE(2, perplexity, 200, 1000, false).EmbedData(matrix, nil)
points := make(plotter.XYs, len(rows))
coordinates := make([][]float64, len(rows))
colors := make([]int, len(rows))
for i, row := range rows {
x, y := embedded.At(i, 0), embedded.At(i, 1)
if math.IsNaN(x) || math.IsNaN(y) || math.IsInf(x, 0) || math.IsInf(y, 0) {
log.Fatal("non-finite t-SNE output")
}
if row.Score < 1 || row.Score > 5 {
log.Fatal("review scores must be between 1 and 5")
}
points[i] = plotter.XY{X: x, Y: y}
coordinates[i] = []float64{x, y}
colors[i] = row.Score - 1
}
chart := plot.New()
chart.Title.Text = "Review embeddings"
chart.X.Label.Text = "t-SNE 1"
chart.Y.Label.Text = "t-SNE 2"
scatter, err := plotter.NewScatter(points)
if err != nil {
log.Fatal(err)
}
palette := []color.RGBA{{255, 0, 0, 255}, {255, 140, 0, 255}, {255, 215, 0, 255}, {64, 224, 208, 255}, {0, 100, 0, 255}}
scatter.GlyphStyleFunc = func(i int) draw.GlyphStyle {
return draw.GlyphStyle{Color: palette[colors[i]], Radius: vg.Points(3), Shape: draw.CircleGlyph{}}
}
chart.Add(scatter)
if err := chart.Save(6*vg.Inch, 4*vg.Inch, "local-plot.png"); err != nil {
log.Fatal(err)
}
plotted := make([][]float64, len(points))
for i := range points {
x, y := scatter.XY(i)
plotted[i] = []float64{x, y}
}
result := map[string]any{"points": coordinates, "plotted": plotted, "colors": colors}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
log.Fatal(err)
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25require "csv"
require "json"
require "rumale/manifold"
require "chunky_png"
rows = CSV.read("output/embedded_1k_reviews.csv", headers: true).map do |row|
row.to_h.merge("ada_embedding" => JSON.parse(row.fetch("ada_embedding")))
end
matrix = Numo::DFloat.asarray(rows.map { |row| row.fetch("ada_embedding") })
tsne = Rumale::Manifold::TSNE.new(n_components: 2, perplexity: 15, random_seed: 42, init: "random")
points = tsne.fit_transform(matrix).to_a
color_indices = rows.map { |row| Integer(row.fetch("Score")) - 1 }
palette = ["#ff0000", "#ff8c00", "#ffd700", "#40e0d0", "#006400"]
plot = ChunkyPNG::Image.new(600, 400, ChunkyPNG::Color::WHITE)
x_min, x_max = points.map(&:first).minmax
y_min, y_max = points.map(&:last).minmax
plotted = points.map(&:dup)
plotted.each_with_index do |(x, y), index|
pixel_x = 20 + ((x - x_min) / [x_max - x_min, Float::EPSILON].max * 560).round
pixel_y = 380 - ((y - y_min) / [y_max - y_min, Float::EPSILON].max * 360).round
color = ChunkyPNG::Color.from_hex(palette.fetch(color_indices.fetch(index)))
plot.circle(pixel_x, pixel_y, 3, color, color)
end
plot.save("local-plot.png")在机器学习模型中,嵌入向量可以用作自由文本的通用特征编码器。只要相关输入中包含自由文本,引入嵌入向量就能改善机器学习模型的表现。嵌入向量也可以用于编码模型的类别特征。当类别众多且名称本身具有含义时(例如职位名称),这种做法最有价值。对于这项任务,用于衡量相似度的嵌入向量通常比用于搜索的嵌入向量表现更好。
我们观察到,嵌入向量通常包含丰富的信息。例如,使用 SVD 或 PCA 对输入降维时,即使维度只减少 10%,模型在具体下游任务中的表现通常也会变差。
以下代码将数据拆分为训练集和测试集,供接下来的回归和分类两个示例使用。
1
2
3
4
5from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
list(df.ada_embedding.values), df.Score, test_size=0.2, random_state=42
)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"log"
"math/rand"
"os"
"strconv"
)
type review struct {
Vector []float64
Score int
UserID, ProductID string
}
func loadReviews(path string) ([]review, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
reader := csv.NewReader(file)
header, err := reader.Read()
if err != nil {
return nil, err
}
columns := map[string]int{}
for i, name := range header {
columns[name] = i
}
for _, name := range []string{"ada_embedding", "Score"} {
if _, ok := columns[name]; !ok {
return nil, fmt.Errorf("missing %s column", name)
}
}
rows := []review{}
dimensions := 0
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
var value review
if err := json.Unmarshal([]byte(record[columns["ada_embedding"]]), &value.Vector); err != nil {
return nil, err
}
if len(value.Vector) == 0 {
return nil, fmt.Errorf("empty embedding")
}
if dimensions == 0 {
dimensions = len(value.Vector)
}
if len(value.Vector) != dimensions {
return nil, fmt.Errorf("inconsistent embedding dimensions")
}
value.Score, err = strconv.Atoi(record[columns["Score"]])
if err != nil {
return nil, err
}
if column, ok := columns["UserId"]; ok {
value.UserID = record[column]
}
if column, ok := columns["ProductId"]; ok {
value.ProductID = record[column]
}
rows = append(rows, value)
}
if len(rows) == 0 {
return nil, fmt.Errorf("no reviews")
}
return rows, nil
}
func main() {
rows, err := loadReviews("output/embedded_1k_reviews.csv")
if err != nil {
log.Fatal(err)
}
if len(rows) < 2 {
log.Fatal("need at least two reviews")
}
indices := rand.New(rand.NewSource(42)).Perm(len(rows))
testSize := (len(rows) + 4) / 5
trainVectors, testVectors := [][]float64{}, [][]float64{}
trainScores, testScores := []int{}, []int{}
for i, index := range indices {
row := rows[index]
if i < testSize {
testVectors = append(testVectors, row.Vector)
testScores = append(testScores, row.Score)
} else {
trainVectors = append(trainVectors, row.Vector)
trainScores = append(trainScores, row.Score)
}
}
result := map[string]any{"train": trainScores, "test": testScores, "train_vectors": trainVectors, "test_vectors": testVectors}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
log.Fatal(err)
}
}1
2
3
4
5
6
7
8
9
10
11vectors = rows.map { |row| row.fetch("ada_embedding") }
scores = rows.map { |row| Integer(row.fetch("Score")) }
indices = (0...vectors.length).to_a.shuffle(random: Random.new(42))
test_size = (vectors.length * 0.2).ceil
test_indices = indices.take(test_size)
train_indices = indices.drop(test_size)
x_train = train_indices.map { |index| vectors.fetch(index) }
holdout_vectors = test_indices.map { |index| vectors.fetch(index) }
y_train = train_indices.map { |index| scores.fetch(index) }
holdout_scores = test_indices.map { |index| scores.fetch(index) }使用嵌入特征进行回归
嵌入向量提供了一种简洁的数值预测方法。在这个示例中,我们根据评论文本预测评论者给出的星级。由于嵌入向量包含丰富的语义信息,即使评论数量较少,也能取得不错的预测效果。
我们假设评分是 1 到 5 之间的连续变量,允许算法输出任意浮点值作为预测结果。机器学习算法尽量缩小预测值与真实评分之间的差距,最终的平均绝对误差为 0.39,即预测评分与真实评分平均相差不到半颗星。
Go 示例使用 github.com/wlattner/rf/forest 包,其所属模块的版本为 v0.0.0-20150304005020-e98c6ca28b25。程序使用训练集训练 100 棵决策树,然后预测测试集中每条评论的评分。请先安装指定版本的模块,再运行程序。
1
2
3
4
5from sklearn.ensemble import RandomForestRegressor
rfr = RandomForestRegressor(n_estimators=100)
rfr.fit(X_train, y_train)
preds = rfr.predict(X_test)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"github.com/wlattner/rf/forest"
"io"
"log"
"math/rand"
"os"
"strconv"
)
type review struct {
Vector []float64
Score int
UserID, ProductID string
}
func loadReviews(path string) ([]review, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
reader := csv.NewReader(file)
header, err := reader.Read()
if err != nil {
return nil, err
}
columns := map[string]int{}
for i, name := range header {
columns[name] = i
}
for _, name := range []string{"ada_embedding", "Score"} {
if _, ok := columns[name]; !ok {
return nil, fmt.Errorf("missing %s column", name)
}
}
rows := []review{}
dimensions := 0
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
var value review
if err := json.Unmarshal([]byte(record[columns["ada_embedding"]]), &value.Vector); err != nil {
return nil, err
}
if len(value.Vector) == 0 {
return nil, fmt.Errorf("empty embedding")
}
if dimensions == 0 {
dimensions = len(value.Vector)
}
if len(value.Vector) != dimensions {
return nil, fmt.Errorf("inconsistent embedding dimensions")
}
value.Score, err = strconv.Atoi(record[columns["Score"]])
if err != nil {
return nil, err
}
if column, ok := columns["UserId"]; ok {
value.UserID = record[column]
}
if column, ok := columns["ProductId"]; ok {
value.ProductID = record[column]
}
rows = append(rows, value)
}
if len(rows) == 0 {
return nil, fmt.Errorf("no reviews")
}
return rows, nil
}
func main() {
rows, err := loadReviews("output/embedded_1k_reviews.csv")
if err != nil {
log.Fatal(err)
}
if len(rows) < 2 {
log.Fatal("need at least two reviews")
}
indices := rand.New(rand.NewSource(42)).Perm(len(rows))
testSize := (len(rows) + 4) / 5
trainVectors, testVectors := [][]float64{}, [][]float64{}
trainScores, testScores := []int{}, []int{}
for i, index := range indices {
row := rows[index]
if i < testSize {
testVectors = append(testVectors, row.Vector)
testScores = append(testScores, row.Score)
} else {
trainVectors = append(trainVectors, row.Vector)
trainScores = append(trainScores, row.Score)
}
}
targets := make([]float64, len(trainScores))
for i, score := range trainScores {
targets[i] = float64(score)
}
model := forest.NewRegressor(forest.NumTrees(100), forest.NumWorkers(1))
model.Fit(trainVectors, targets)
predictions := model.Predict(testVectors)
result := map[string]any{"train": trainScores, "test": testScores, "train_vectors": trainVectors, "test_vectors": testVectors, "predictions": predictions, "trees": len(model.Trees), "features": len(trainVectors[0])}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
log.Fatal(err)
}
}1
2
3
4
5
6
7
8
9require "rumale/ensemble"
x_train = Numo::DFloat.asarray(x_train)
y_train = Numo::DFloat.asarray(y_train)
holdout_vectors = Numo::DFloat.asarray(holdout_vectors)
forest = Rumale::Ensemble::RandomForestRegressor.new(n_estimators: 100, random_seed: 42)
forest.fit(x_train, y_train)
predictions = forest.predict(holdout_vectors)这次,我们将预测目标从 1 到 5 之间的连续数值改为具体星级,尝试把每条评论归入 1 星到 5 星这五个类别中的一个。
训练后,模型对 1 星和 5 星评论的预测明显比对 2 至 4 星评论的预测更准确。这可能是因为两端星级的评论表达的情绪更强烈,而中间星级的评论态度更为微妙。
Go 示例使用 github.com/wlattner/rf/forest 包,其所属模块的版本为 v0.0.0-20150304005020-e98c6ca28b25。程序使用训练集训练 100 棵决策树,然后预测测试集中每条评论的星级。请先安装指定版本的模块,再运行程序。
1
2
3
4
5
6from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, accuracy_score
clf = RandomForestClassifier(n_estimators=100)
clf.fit(X_train, y_train)
preds = clf.predict(X_test)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"github.com/wlattner/rf/forest"
"io"
"log"
"math/rand"
"os"
"slices"
"strconv"
)
type review struct {
Vector []float64
Score int
UserID, ProductID string
}
func loadReviews(path string) ([]review, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
reader := csv.NewReader(file)
header, err := reader.Read()
if err != nil {
return nil, err
}
columns := map[string]int{}
for i, name := range header {
columns[name] = i
}
for _, name := range []string{"ada_embedding", "Score"} {
if _, ok := columns[name]; !ok {
return nil, fmt.Errorf("missing %s column", name)
}
}
rows := []review{}
dimensions := 0
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
var value review
if err := json.Unmarshal([]byte(record[columns["ada_embedding"]]), &value.Vector); err != nil {
return nil, err
}
if len(value.Vector) == 0 {
return nil, fmt.Errorf("empty embedding")
}
if dimensions == 0 {
dimensions = len(value.Vector)
}
if len(value.Vector) != dimensions {
return nil, fmt.Errorf("inconsistent embedding dimensions")
}
value.Score, err = strconv.Atoi(record[columns["Score"]])
if err != nil {
return nil, err
}
if column, ok := columns["UserId"]; ok {
value.UserID = record[column]
}
if column, ok := columns["ProductId"]; ok {
value.ProductID = record[column]
}
rows = append(rows, value)
}
if len(rows) == 0 {
return nil, fmt.Errorf("no reviews")
}
return rows, nil
}
func main() {
rows, err := loadReviews("output/embedded_1k_reviews.csv")
if err != nil {
log.Fatal(err)
}
if len(rows) < 2 {
log.Fatal("need at least two reviews")
}
indices := rand.New(rand.NewSource(42)).Perm(len(rows))
testSize := (len(rows) + 4) / 5
trainVectors, testVectors := [][]float64{}, [][]float64{}
trainScores, testScores := []int{}, []int{}
for i, index := range indices {
row := rows[index]
if i < testSize {
testVectors = append(testVectors, row.Vector)
testScores = append(testScores, row.Score)
} else {
trainVectors = append(trainVectors, row.Vector)
trainScores = append(trainScores, row.Score)
}
}
labels := make([]string, len(trainScores))
for i, score := range trainScores {
labels[i] = strconv.Itoa(score)
}
model := forest.NewClassifier(forest.NumTrees(100), forest.NumWorkers(1))
model.Fit(trainVectors, labels)
predictedClasses := model.Predict(testVectors)
predictions := make([]int, len(predictedClasses))
for i, class := range predictedClasses {
value, err := strconv.Atoi(model.Classes[class])
if err != nil {
log.Fatal(err)
}
predictions[i] = value
}
classes := make([]int, len(model.Classes))
for i, class := range model.Classes {
value, err := strconv.Atoi(class)
if err != nil {
log.Fatal(err)
}
classes[i] = value
}
slices.Sort(classes)
result := map[string]any{"train": trainScores, "test": testScores, "train_vectors": trainVectors, "test_vectors": testVectors, "predictions": predictions, "trees": len(model.Trees), "features": len(trainVectors[0]), "classes": classes}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
log.Fatal(err)
}
}1
2
3
4
5
6
7
8
9require "rumale/ensemble"
x_train = Numo::DFloat.asarray(x_train)
y_train = Numo::Int32.asarray(y_train)
holdout_vectors = Numo::DFloat.asarray(holdout_vectors)
forest = Rumale::Ensemble::RandomForestClassifier.new(n_estimators: 100, random_seed: 42)
forest.fit(x_train, y_train)
predictions = forest.predict(holdout_vectors)无需任何带标签的训练数据,即可使用嵌入向量进行零样本分类。我们为每个类别的名称或简短描述生成嵌入向量。对新文本进行零样本分类时,将其嵌入向量与所有类别的向量进行比较,选择相似度最高的类别作为预测结果。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21import OpenAI from "openai";
const client = new OpenAI();
const labels = ["negative", "positive"];
const { data } = await client.embeddings.create({
model: "text-embedding-3-small",
input: [...labels, "The coffee arrived quickly and tastes great."],
});
const review = data[labels.length].embedding;
const similarity = (embedding: number[]) => {
const dotProduct = embedding.reduce(
(total, value, index) => total + value * review[index],
0
);
return dotProduct / (Math.hypot(...embedding) * Math.hypot(...review));
};
const [negative, positive] = data.map(({ embedding }) => similarity(embedding));
console.log(positive > negative ? "positive" : "negative");1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18df = df[df.Score != 3]
df["sentiment"] = df.Score.replace(
{1: "negative", 2: "negative", 4: "positive", 5: "positive"}
)
labels = ["negative", "positive"]
label_embeddings = [get_embedding(label, model=model) for label in labels]
def label_score(review_embedding, label_embeddings):
return cosine_similarity(review_embedding, label_embeddings[1]) - cosine_similarity(
review_embedding, label_embeddings[0]
)
prediction = (
"positive" if label_score(get_embedding("Sample Review", model=model), label_embeddings) > 0 else "negative"
)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59import (
"context"
"fmt"
"log"
"math"
"github.com/openai/openai-go/v3"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
client := openai.NewClient()
ctx := context.Background()
negative, err := embedding(ctx, &client, "negative")
if err != nil {
return err
}
positive, err := embedding(ctx, &client, "positive")
if err != nil {
return err
}
review, err := embedding(ctx, &client, "Sample Review")
if err != nil {
return err
}
score := cosineSimilarity(review, positive) - cosineSimilarity(review, negative)
prediction := "negative"
if score > 0 {
prediction = "positive"
}
fmt.Println(prediction)
return nil
}
func embedding(ctx context.Context, client *openai.Client, text string) ([]float64, error) {
response, err := client.Embeddings.New(ctx, openai.EmbeddingNewParams{
Model: openai.EmbeddingModelTextEmbedding3Small,
Input: openai.EmbeddingNewParamsInputUnion{OfString: openai.String(text)},
})
if err != nil {
return nil, err
}
return response.Data[0].Embedding, nil
}
func cosineSimilarity(a, b []float64) float64 {
var dot, left, right float64
for i := range a {
dot += a[i] * b[i]
left += a[i] * a[i]
right += b[i] * b[i]
}
return dot / math.Sqrt(left*right)
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.embeddings.EmbeddingCreateParams;
import java.util.List;
var embeddings =
client
.embeddings()
.create(
EmbeddingCreateParams.builder()
.model("text-embedding-3-small")
.inputOfArrayOfStrings(List.of("negative", "positive", "Sample Review"))
.build())
.data();
List<Float> review = embeddings.get(2).embedding();
double negative = cosineSimilarity(review, embeddings.get(0).embedding());
double positive = cosineSimilarity(review, embeddings.get(1).embedding());
System.out.println(positive > negative ? "positive" : "negative");1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29using OpenAI.Embeddings;
string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
string model = "text-embedding-3-small";
EmbeddingClient client = new(model, key);
float[] negative = await EmbedAsync(client, "negative");
float[] positive = await EmbedAsync(client, "positive");
float[] review = await EmbedAsync(client, "Sample Review");
double score = CosineSimilarity(review, positive) - CosineSimilarity(review, negative);
Console.WriteLine(score > 0 ? "positive" : "negative");
static async Task<float[]> EmbedAsync(EmbeddingClient client, string text)
{
OpenAIEmbedding result = await client.GenerateEmbeddingAsync(text);
return result.ToFloats().ToArray();
}
static double CosineSimilarity(float[] left, float[] right)
{
double dot = 0, leftNorm = 0, rightNorm = 0;
for (int i = 0; i < left.Length; i++)
{
dot += (double)left[i] * right[i];
leftNorm += (double)left[i] * left[i];
rightNorm += (double)right[i] * right[i];
}
return dot / Math.Sqrt(leftNorm * rightNorm);
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22require "openai"
client = OpenAI::Client.new
labels = ["negative", "positive"]
response = client.embeddings.create(
model: "text-embedding-3-small",
input: labels + ["The coffee arrived quickly and tastes great."]
)
review = response.data.fetch(-1).embedding
similarity = lambda do |embedding|
dot_product = embedding.zip(review).sum { |value, review_value| value * review_value }
magnitude = Math.sqrt(embedding.sum { |value| value**2 })
review_magnitude = Math.sqrt(review.sum { |value| value**2 })
dot_product / (magnitude * review_magnitude)
end
negative, positive = response.data.first(2).map do |embedding|
similarity.call(embedding.embedding)
end
puts((positive > negative) ? "positive" : "negative")对某位用户发表的所有评论的嵌入向量取平均值,即可得到该用户的嵌入向量。同样,对某件商品收到的所有评论的嵌入向量取平均值,即可得到该商品的嵌入向量。为展示这种方法的效果,我们使用包含 5 万条评论的子集,以便为每位用户和每件商品纳入更多评论。
我们在单独的测试集上评估这些向量的效果,按评分绘制用户向量与商品向量之间的相似度分布。结果显示,即使用户尚未收到商品,这种方法也能预测用户是否会喜欢它,效果优于随机猜测。

user_embeddings = df.groupby("UserId").ada_embedding.apply(np.mean)
prod_embeddings = df.groupby("ProductId").ada_embedding.apply(np.mean)package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"log"
"os"
"strconv"
)
type review struct {
Vector []float64
Score int
UserID, ProductID string
}
func loadReviews(path string) ([]review, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
reader := csv.NewReader(file)
header, err := reader.Read()
if err != nil {
return nil, err
}
columns := map[string]int{}
for i, name := range header {
columns[name] = i
}
for _, name := range []string{"ada_embedding", "Score"} {
if _, ok := columns[name]; !ok {
return nil, fmt.Errorf("missing %s column", name)
}
}
rows := []review{}
dimensions := 0
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
var value review
if err := json.Unmarshal([]byte(record[columns["ada_embedding"]]), &value.Vector); err != nil {
return nil, err
}
if len(value.Vector) == 0 {
return nil, fmt.Errorf("empty embedding")
}
if dimensions == 0 {
dimensions = len(value.Vector)
}
if len(value.Vector) != dimensions {
return nil, fmt.Errorf("inconsistent embedding dimensions")
}
value.Score, err = strconv.Atoi(record[columns["Score"]])
if err != nil {
return nil, err
}
if column, ok := columns["UserId"]; ok {
value.UserID = record[column]
}
if column, ok := columns["ProductId"]; ok {
value.ProductID = record[column]
}
rows = append(rows, value)
}
if len(rows) == 0 {
return nil, fmt.Errorf("no reviews")
}
return rows, nil
}
func averageEmbeddings(rows []review, id func(review) string) (map[string][]float64, error) {
sums := map[string][]float64{}
counts := map[string]int{}
for _, row := range rows {
key := id(row)
if key == "" {
return nil, fmt.Errorf("missing user or product ID")
}
if sums[key] == nil {
sums[key] = make([]float64, len(row.Vector))
}
for i, value := range row.Vector {
sums[key][i] += value
}
counts[key]++
}
for key, vector := range sums {
for i := range vector {
vector[i] /= float64(counts[key])
}
}
return sums, nil
}
func main() {
rows, err := loadReviews("output/embedded_1k_reviews.csv")
if err != nil {
log.Fatal(err)
}
users, err := averageEmbeddings(rows, func(row review) string { return row.UserID })
if err != nil {
log.Fatal(err)
}
products, err := averageEmbeddings(rows, func(row review) string { return row.ProductID })
if err != nil {
log.Fatal(err)
}
result := map[string]any{"users": users, "products": products}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
log.Fatal(err)
}
}vectors = rows.map { |row| row.fetch("ada_embedding") }
user_ids = rows.map { |row| row.fetch("UserId") }
product_ids = rows.map { |row| row.fetch("ProductId") }
def average_embeddings(ids, vectors)
ids.zip(vectors).group_by(&:first).transform_values do |rows|
rows.map(&:last).transpose.map { |dimension| dimension.sum / dimension.length.to_f }
end
end
user_embeddings = average_embeddings(user_ids, vectors)
product_embeddings = average_embeddings(product_ids, vectors)聚类有助于理解大量文本数据。嵌入向量能表示每段文本的语义,因此很适合这项任务。通过无监督聚类,我们可以发现数据集中隐含的分组。
在这个示例中,我们发现了四个明显不同的簇:一个主要讨论狗粮,一个以负面评论为主,另外两个以正面评论为主。

1
2
3
4
5
6
7
8
9import numpy as np
from sklearn.cluster import KMeans
matrix = np.vstack(df.ada_embedding.values)
n_clusters = 4
kmeans = KMeans(n_clusters=n_clusters, init="k-means++", random_state=42)
kmeans.fit(matrix)
df["Cluster"] = kmeans.labels_1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"log"
"math"
"math/rand"
"os"
"strconv"
)
type review struct {
Vector []float64
Score int
UserID, ProductID string
}
func loadReviews(path string) ([]review, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
reader := csv.NewReader(file)
header, err := reader.Read()
if err != nil {
return nil, err
}
columns := map[string]int{}
for i, name := range header {
columns[name] = i
}
for _, name := range []string{"ada_embedding", "Score"} {
if _, ok := columns[name]; !ok {
return nil, fmt.Errorf("missing %s column", name)
}
}
rows := []review{}
dimensions := 0
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
var value review
if err := json.Unmarshal([]byte(record[columns["ada_embedding"]]), &value.Vector); err != nil {
return nil, err
}
if len(value.Vector) == 0 {
return nil, fmt.Errorf("empty embedding")
}
if dimensions == 0 {
dimensions = len(value.Vector)
}
if len(value.Vector) != dimensions {
return nil, fmt.Errorf("inconsistent embedding dimensions")
}
value.Score, err = strconv.Atoi(record[columns["Score"]])
if err != nil {
return nil, err
}
if column, ok := columns["UserId"]; ok {
value.UserID = record[column]
}
if column, ok := columns["ProductId"]; ok {
value.ProductID = record[column]
}
rows = append(rows, value)
}
if len(rows) == 0 {
return nil, fmt.Errorf("no reviews")
}
return rows, nil
}
func squaredDistance(a, b []float64) float64 {
sum := 0.0
for i, value := range a {
delta := value - b[i]
sum += delta * delta
}
return sum
}
func kmeans(vectors [][]float64, k int) ([]int, [][]float64, error) {
if len(vectors) < k || k < 1 {
return nil, nil, fmt.Errorf("need at least %d vectors", k)
}
random := rand.New(rand.NewSource(42))
centers := [][]float64{append([]float64(nil), vectors[random.Intn(len(vectors))]...)}
// k-means++ initialization samples proportional to squared distance.
for len(centers) < k {
distances := make([]float64, len(vectors))
total := 0.0
for i, vector := range vectors {
best := math.Inf(1)
for _, center := range centers {
best = math.Min(best, squaredDistance(vector, center))
}
distances[i] = best
total += best
}
if total == 0 {
return nil, nil, fmt.Errorf("need %d distinct vectors", k)
}
threshold := random.Float64() * total
chosen := len(vectors) - 1
for i, distance := range distances {
threshold -= distance
if threshold < 0 {
chosen = i
break
}
}
centers = append(centers, append([]float64(nil), vectors[chosen]...))
}
labels := make([]int, len(vectors))
for i := range labels {
labels[i] = -1
}
for iteration := 0; iteration < 300; iteration++ {
changed := false
counts := make([]int, k)
sums := make([][]float64, k)
for i := range sums {
sums[i] = make([]float64, len(vectors[0]))
}
for i, vector := range vectors {
closest := 0
for j := 1; j < k; j++ {
if squaredDistance(vector, centers[j]) < squaredDistance(vector, centers[closest]) {
closest = j
}
}
if labels[i] != closest {
changed = true
}
labels[i] = closest
counts[closest]++
for j, value := range vector {
sums[closest][j] += value
}
}
for i, count := range counts {
if count > 0 {
for j := range sums[i] {
sums[i][j] /= float64(count)
}
centers[i] = sums[i]
}
}
if !changed {
return labels, centers, nil
}
}
return nil, nil, fmt.Errorf("k-means did not converge")
}
func main() {
rows, err := loadReviews("output/embedded_1k_reviews.csv")
if err != nil {
log.Fatal(err)
}
vectors := make([][]float64, len(rows))
for i, row := range rows {
vectors[i] = row.Vector
}
labels, centers, err := kmeans(vectors, 4)
if err != nil {
log.Fatal(err)
}
result := map[string]any{"labels": labels, "centers": centers}
if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
log.Fatal(err)
}
}1
2
3
4
5
6
7require "rumale/clustering"
vectors = rows.map { |row| row.fetch("ada_embedding") }
matrix = Numo::DFloat.asarray(vectors)
kmeans = Rumale::Clustering::KMeans.new(n_clusters: 4, init: "k-means++", random_seed: 42)
labels = kmeans.fit_predict(matrix)常见问题
如何在生成嵌入向量之前确定字符串包含多少个 Token?
在 Python 中,您可以使用 OpenAI 的分词器 tiktoken 将字符串拆分为 Token。
示例代码:
1
2
3
4
5
6
7
8
9
10
11import tiktoken
def num_tokens_from_string(string: str, encoding_name: str) -> int:
"""Returns the number of tokens in a text string."""
encoding = tiktoken.get_encoding(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
num_tokens_from_string("tiktoken is great!", "cl100k_base")1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23package main
import (
"fmt"
"github.com/tiktoken-go/tokenizer"
)
func numTokensFromString(text, encodingName string) (int, error) {
encoding, err := tokenizer.Get(tokenizer.Encoding(encodingName))
if err != nil {
return 0, err
}
ids, _, err := encoding.Encode(text)
return len(ids), err
}
func main() {
count, err := numTokensFromString("tiktoken is great!", "cl100k_base")
if err != nil {
panic(err)
}
fmt.Println(count)
}1
2
3
4
5
6
7require "tiktoken_ruby"
def num_tokens_from_string(string, encoding_name)
Tiktoken.get_encoding(encoding_name).encode(string).length
end
puts(num_tokens_from_string("tiktoken is great!", "cl100k_base"))Go 示例使用第三方包 github.com/tiktoken-go/tokenizer,版本为 v0.10.0。运行示例前,请先在您的 Go 模块中安装该版本。
对于 text-embedding-3-small 等第三代嵌入模型,请使用 cl100k_base 编码。
更多说明和示例代码,请参阅 OpenAI Cookbook 指南如何使用 tiktoken 统计 Token 数量。
如何快速检索最近的 K 个嵌入向量?
要在大量向量中快速搜索,我们建议使用向量数据库。您可以在 GitHub 上的 OpenAI Cookbook 中找到结合使用向量数据库和 OpenAI API 的示例。
应该使用哪种距离函数?
我们建议使用余弦相似度。通常,距离函数的选择对结果影响不大。
OpenAI 的嵌入向量经过归一化,长度均为 1,因此:
- 只需计算点积即可得到余弦相似度,速度略快
- 使用余弦相似度和欧氏距离会得到完全相同的排序
可以在网上分享我的嵌入向量吗?
可以。客户拥有提交给我们模型的输入以及模型生成的输出,包括嵌入向量。您有责任确保提交给 API 的内容符合所有适用法律以及我们的使用条款。
第三代嵌入模型了解近期事件吗?
不了解。text-embedding-3-large 和 text-embedding-3-small 模型都不了解 2021 年 9 月之后发生的事件。相比文本生成模型,这一限制对嵌入模型的影响通常较小,但在某些特殊情况下仍可能导致性能下降。