Release Announcement: Vald v1.5.0

What is the CoW?

CoW mechanism

CoW step1: create new backup file when SaveIndex process starts
CoW step2: move current backup file to old index path directory
CoW step3: move new backup file to index path directory and create new tmp dir for next SaveIndex

The way to turn on CoW

defaults:
image:
tag: v1.5.0

gateway:
lb:
resources:
requests:
cpu: 100m
memory: 50Mi
gateway_config:
# a number of index replicas.
index_replica: 2

agent:
minReplicas: 6
maxReplicas: 6
podManagementPolicy: Parallel
resources:
requests:
cpu: 100m
memory: 50Mi
# We recommend setting this value long enough to ensure the backup speed of PV since the Index is backed up at the end of the pod.
terminationGracePeriodSeconds: 600
# This is the persistent volume setting.
# Please change it according to your environment.
persistentVolume:
enabled: true
accessMode: ReadWriteOnce
storageClass: local-path
size: 5Gi
ngt:
dimension: 784
index_path: "/var/ngt/index"
enable_in_memory_mode: false
# limit duration of automatic indexing.
auto_index_duration_limit: 730h
# check duration of automatic indexing.
auto_index_check_duration: 24h
# number of cache to trigger automatic indexing.
auto_index_length: 1000
# duration of automatic save index.
auto_save_index_duration: 365h
# batch process pool size of automatic create index operation.
auto_create_index_pool_size: 1000
# the flag of using CoW or not.
enable_copy_on_write: true

discoverer:
resources:
requests:
cpu: 100m
memory: 50Mi

manager:
index:
resources:
requests:
cpu: 100m
memory: 30Mi
indexer:
# concurrency for indexing operation.
concurrency: 1
# limit duration of automatic indexing.
auto_index_duration_limit: 10m
# check duration of automatic indexing.
auto_index_check_duration: 1m
# limit duration of automatic index saving.
auto_save_index_duration_limit: 1h
# duration of automatic index saving wait duration for next saving.
auto_save_index_wait_duration: 10m
# number of cache to trigger automatic indexing.
auto_index_length: 100
# number of pool size of create index processing.
creation_pool_size: 10000
  • Set agent.persitentVolumne config because CoW requires PV
  • Set agent.ngt.in_memory_mode as false to use PV for backup
  • Set agent.ngt.enable_copy_on_write as true

--

--

--

A highly scalable distributed fast approximate nearest neighbor dense vector search engine.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

My take on Brian Douglas’ take on the JAMStack

Build Your First Spring Boot Project!

Making Readable Code With Dependency Injection and Jakarta CDI

Horizontally Scale your Services in WSO2 Cloud

How to Create Emails to Cross-Sell Other Products

Building your API: Design Mode

An Introduction to The MMP Discord Server

Boolean Operators and Its Types

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
vald.vdaas.org

vald.vdaas.org

A highly scalable distributed fast approximate nearest neighbor dense vector search engine.

More from Medium

Multi Master Multi-Cluster LDAP(OpenDJ) replication in Kubernetes? A controversial view

Revisiting Container Image builds inside a Container

Isolated Build and Dev environments using Kata Containers

The Need For Open Source Security

My real-life Istio Performance Tuning Part 2 — Starvating threads and SO_REUSEPORT socket