Vald released the new Vald official ingress filter: vald-onnx-ingress-filter

Vald released the official ingress filter component named vald-onnx-ingress-filter. This post describes the introduction of it and the tutorial for using it.

What is the ingress filter?

The ingress filter is one of Vald’s components that can do some pre-processing on a request before processing it by connecting the ingress filter gateway component. Users can implement user-defined custom filtering logic as a ingress filter, such as vector modification, filtering function based on user ID, and so on, before inserting the vector into the Vald cluster. Here are the examples of specific logic:

• Modifying the sent vector, such as normalization
• Compressing the dimensions of the request vector
• Converting unstructured data such as text and images into a vector

Of course, a comprehensive variety of custom filtering can perform according to each use case.

What is the vald-onnx-ingress-filter?

vald-onnx-ingress-filter is one of the official ingress filter implementation provided by Vald. Its custom logic requires the input of the ONNX model as a request and outputs the result from the ONNX model as the request of the Vald Agent. Using this component, you can vectorize various unstructured data, such as text, audio, and image, using the ONNX model in the request inline of Vald API without building your external API. And you can directly perform CRUD processing using the unstructured data.

About the detail of ONNX, please refer to the official page

ONNX | Home

For more detail on Vald components, please refer to the below document.

Vald | Component

Get Started with vald-onnx-ingress-filter

Now, let’s start to try to vald-onnx-ingress-filter with the sample manifest! This tutorial requires:

• Docker
• Helm
• K3d (or another kubernetes cluster)
• Git

This tutorial follows the below steps:

1. Preparing Kubernetes cluster
2. Deploy vdaas/vald-onnx-ingress-filter
3. Deploy Vald cluster including vald-filter-gateway component
4. Insert and Search operations

Preparing Kubernetes Cluster

Let’s check your Kubernetes cluster is running. If you use K3d, the cluster starts with

// create cluster command
k3d cluster create vald-onnx

// check command
kubectl cluster-info

Deploy vald-onnx-ingress-filter

The manifests in public in the repository. Please clone to the local by following command.

git clone https://github.com/vdaas/vald-onnx-ingress-filter.git
cd vald-onnx-ingress-filter

This tutorial uses ResNet50-v2 from ONNX Model Zoo as the ONNX model. You can change the model by editing the example manifest files, k8s/deployment.yaml.

...
  - |
    curl -L "https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v2-7.onnx" -o /model/sample.onnx  #FIXME

And, when the ONNX model path changes, please do not forget to update k8s/deployment.yaml.

...
    env:
      - name: MODEL_PATH
        value: "/model/sample.onnx"  #FIXME

After editing manifest files, let’s deploy vald-onnx-ingress-filter to your Kubernetes cluster.

// deploy
kubectl apply -f k8s

// check
kubectl get pod
===
NAME                                        READY   STATUS    RESTARTS   AGE
vald-onnx-ingress-filter-55b8778f5c-chkqq   1/1     Running   0          68s
vald-onnx-ingress-filter-55b8778f5c-xxf54   1/1     Running   0          68s

Deploy Vald cluster with filter gateway

Next, let’s deploy the Vald cluster using Helm. We edit the example/helm/values.yaml in the Vald repository to make vald-filter-gateway available and use it for deployment.

git clone https://github.com/vdaas/vald.git
cd vald
vim example/helm/values.yaml
---
...
gateway:
...
    filter:
        enabled: true
...
agent:
    ngt:
        dimension: 1000
...

After editing, let’s try to deploy the Vald cluster by the helm install command.

# deploy
helm install vald vald/vald --values example/helm/values.yaml

# verify
kubectl get pods
===
NAME                                        READY   STATUS    RESTARTS   AGE
vald-agent-ngt-0                            1/1     Running   0          15m
vald-agent-ngt-1                            1/1     Running   0          15m
vald-agent-ngt-2                            1/1     Running   0          15m
vald-agent-ngt-3                            1/1     Running   0          15m
vald-agent-ngt-4                            1/1     Running   0          15m
vald-agent-ngt-5                            1/1     Running   0          15m
vald-discoverer-749cbf4cd4-kvs4v            1/1     Running   0          15m
vald-filter-gateway-7856bf7c6c-krwsb        1/1     Running   0          15m
vald-filter-gateway-7856bf7c6c-lhxfq        1/1     Running   0          15m
vald-filter-gateway-7856bf7c6c-tg6hg        1/1     Running   0          15m
vald-lb-gateway-6798bd4c8d-46b57            1/1     Running   0          15m
vald-lb-gateway-6798bd4c8d-b7bcn            1/1     Running   0          15m
vald-lb-gateway-6798bd4c8d-qwc2q            1/1     Running   0          15m
vald-manager-index-669d967797-j9smn         1/1     Running   0          15m
vald-onnx-ingress-filter-55b8778f5c-chkqq   1/1     Running   0          18m
vald-onnx-ingress-filter-55b8778f5c-xxf54   1/1     Running   0          18m

Insert and search operation

This section tries to insert vectors and search using the Vald cluster and vald-onnx-ingress-filter, which you deployed before. Before running operations, we use kubectl port-forward to connect vald-filter-gateway and the local environment. You do not have to port-forward if you use the Kubernetes Ingress.

kubectl port-forward deployment/vald-filter-gateway 8081:8081

Insert vectors

The insert dataflow in this situation is:

1. vald-filter-gateway gets the request, including byte objects as the insert object, and passes it to the vald-onnx-ingress-filter.
2. vald-onnx-ingress-filter converts vector from request byte object then uses it as the input of ONNX model in this component.
3. vald-onnx-ingress-filter gets the vector, the ONNX model's output, and tries to insert it into the Vald Agent.

Therefore, any object defined as a byte sequence, whether text, image, or voice, can be used as long as an ONNX model properly vectorizes the input vector from the object again. This time, we use a random vector as an example of the object.

sample code:

import grpc
import numpy as np
from vald.v1.payload import payload_pb2
from vald.v1.vald import (
    filter_pb2_grpc,
    search_pb2_grpc,
)

channel = grpc.insecure_channel("localhost:8081")
stub = filter_pb2_grpc.FilterStub(channel)

sample = np.random.random((1, 3, 224, 224)).astype(np.float32)
resize_vector = payload_pb2.Object.ReshapeVector(
    object=sample.tobytes(),
    shape=[1, 3, 224, 224],
)
resize_vector = resize_vector.SerializeToString()

req = payload_pb2.Insert.ObjectRequest(
    object=payload_pb2.Object.Blob(
        id="0",
        object=resize_vector
    ),
    config=payload_pb2.Insert.Config(skip_strict_exist_check=False),
    vectorizer=payload_pb2.Filter.Target(
        # Please edit host according to your environment.
        host="{Your k8s ingress host for vald-ingress-filter}",
        port=8081,
    )
)
stub.InsertObject(req)

sample output:

name: "vald-agent-ngt-1"
uuid: "0"
ips: "127.0.0.1"
ips: "127.0.0.1"
ips: "127.0.0.1"
ips: "127.0.0.1"
ips: "127.0.0.1"

Due to this tutorial using k3d, all “IPS” in output are 127.0.0.1.

Search neighbors

The search operation searches for the vector just inserted. Here, as an example, the search is performed with 0 vectors. Note that inserted vectors are random vectors regarding distance, so the search result may not be constant. Also, the below code users use the same channel in the previous section.

sample code:

sstub = search_pb2_grpc.SearchStub(channel)
scfg = payload_pb2.Search.Config(
    num=10, radius=-1.0, epsilon=0.01, timeout=3000000000
)
sstub.Search(
    payload_pb2.Search.Request(
        vector=np.zeros((1000)),
        config=scfg
    )
)

sample output:

results {
  id: "0"
  distance: 44.72878646850586
}

Congratulations! You completed the tutorial with vald-onnx-ingress-filter!

Conclusion

This post briefly introduced vald-onnx-ingress-filter and an easy tutorial. Thanks for reading this far and for your interest!

We introduced the official implementation of ingress filter component provided by the Vald team, but the ingress filter is not limited. You can implement the ingress filter to meet the vald-ingress-filter API interface and use it independently. Let’s try to create your custom the ingress filter :)

If you have any questions or any requests, please feel free to contact us!

Slack

We’re waiting for your contributions!

Vald | Contributing Guide

See you next post :)

Other Post

Vald. A highly scalable distributed fast approximate nearest neighbour dense vector search engine.

A New World created by similar search: Cases where Vald can be used.

A Super Easy Way to Try Similarity Search using Vald