The Unit Test Strategy In Vald

The quality of the software is mainly divided into two categories:

• Software functional quality
• Software structural quality

Software functional quality reflects how the software meets the requirement or specification. And software structural quality refers to how well the software meets the non-functional requirements to support the delivery of the functional requirements.

In this article, we will focus on talking about unit test (a part of software functional quality).

What is the unit test?

According to Wikipedia, unit testing is a software testing method to test individual units (most likely functions/methods) of your source code to ensure it meets the requirements and behaves as intended. Unit tests are typically automated, which means they can be executed automatically before deploying your application.

You can think that a unit test is a piece of source code to validate your implementation.

For example, we implemented a function in Go to add two numbers:

func add(i, j int) int {
    return i + j
}

To implement the test code, we need to define the input and the expected output of the function. In this case, we define the input, 1 and 2, and the expected output is 3.

func TestAdd(t *testing.T) {
    got := add(1,2)
    if got != 3 {
        t.Errorf("add(1,2) = %d; want 3", got)
    }

}

In this implementation, we only implemented one test case. We may need to implement multiple test cases to cover more functionality of the implementation.

In reality, the implementation is much more complex than this example, hence the test implementation is more complex than this example.

In the next chapter, we will introduce how the unit test applies to our product and how we implemented it.

What is Vald?

Vald is an open-source project working on a highly scalable distributed ANN dense vector search engine written in Go.

Vald is one of the fastest ANN dense vector search engines in the world, with many functionalities to make Vald one of the best ANN dense vector search engines in the world.

We have posted a blog post to explain what is Vald, please visit the below link if you are interested in Vald :)

Vald. A highly scalable distributed fast approximate nearest neighbour dense vector search engine.

Also, please visit our official website or feel free to contact us.

Why do we need unit tests in Vald?

In Vald, we are working on many functions and the implementation is complex, it may have bugs hidden inside the implementation. To provide a better user experience, we try our best to keep Vald bug-free, and we’re working on bug fixes in high priority.

We also try to deliver the latest functionality to our users, users can enjoy the benefits from the new technologies.

In Vald, we use a lot of third-party libraries, and there are chances that the library contains bugs, so we frequently update the third-party library to keep the best quality of the software.

Frequent update of the implementation is not a bad thing. However, it makes it hard to keep the quality of your software. The unit test acts as a first guard to protect your application. It ensures that all code meets quality standards defined in the unit test before it’s deployed, to provide more reliable software to the user.

Unit test strategy

In this chapter, we will talk about the strategy of how do we implement unit tests in Vald. With these strategies, we believe that we will improve the unit test quality to help us to find more potential bugs and defects in our implementation to provide better quality software to our users.

Know what you are testing

Before thinking and designing the test case, we need to know and understand what we are going to test. We need to know the purpose of the target function or the class before we work on it, it helps us to think about what is required for the testing.

Test behaviors and results, not implementation

We should focus on testing the behaviors and results when we start thinking about the unit test case.

Test one thing at a time

Each test should have a clear, concise, and single objective. For each test case, it should only test for a single objective or use case, we should not mix multiple things into one unit test to avoid complexity. Also, it helps us to specify what is failed so we can easily fix the bug when it occurred.

For example, to test the function to get the user data from the database, we can divide the test into multiple test cases with different objectives.

func TestGetUserData(t *testing.T) {
    t.Run("success to get user data", func(t *testing.T) {
        // test implementation
    })
    t.Run("failed to connect to database (unauthorized)", func(t *testing.T) {
        // test implementation
    })
    t.Run("failed to connect to database (host not found)", func(t *testing.T) {
        // test implementation
    })
    t.Run("failed to get non-exists user data", func(t *testing.T) {
        // test implementation
    })
    // other test cases
}

In this example, each test case has clearly defined with a single objective. When one of the test case failed, we can easily fix it.

Make tests readable and understandable

Even unit test code is not production code, we should still maintain the test code as better as we should.

In this strategy, it can be divided into 2 main concepts:

• Test readability
• Test understandability

Test readability refers to how well the test is implemented, while test understandability refers to how a user can understand the test implementation.

For test readability, Vald holds test code to a similar standard as production code and implement test as simple as we can.

For test understandability, we should make the test implementation understandable. For instance, name a test case with an understandable name, or use production-like data as test data.

func TestInsertUserData(t *testing.T) {
    t.Run("test insert user data", func(t *testing.T) { // define a understandable test case name
        // define production-like data as test data
        u := User {
            Name: "Peter Parker",
            Age:  17,
        }
        result := InsertUserData(u)
        // check result
    })
}

Vald has defined some standards to name a test case and define the testing data format. Also, we applied table-driven test to simplify and standardize test implementation, hence easy-to-read and easy-to-maintain test implementation. If you are interested, please visit our coding guideline for more details.

Make tests deterministic

A test should pass all the time or fail all the time until fixed. In some cases, it may be hard to maintain the test result stable in the same test. For example, if the test depends on runtime variables (e.g. time, random number, etc).

func genRand() int {
    return rand.Intn(100) 
}

We can not predict the return value of the function above. For the function above or the functions using this function, it is not possible to write a stable test without applying other testing techniques (e.g. mocking).

But generally, we should try our best to keep the test result as deterministic as we should.

Make tests independent and self-sufficient

Each test should be independent of another test. For each test case, the setup, execution, and verification steps should not depend on running other tests before it. For example, to test a function with an environment variable, we should pay attention to handle the environment variable correctly (e.g. clear the environment variable after the test case is finished), to avoid it affecting other test cases.

func GetEnvs(en string) []string {
    e := os.Getenv(en)
    if e == "" {
        return nil
    }
    return strings.Split(e, ",")
}

When we writing the test case for the function above, we need to set up the environment first.

func TestGetEnvs(t *testing.T) {
    if err := os.Setenv("env_name", "value"); err != nil {
        t.Error(err)
    }
    // implement the test case
}

It is fine if we are testing with one test, but it may be a conflict with other test cases which are using the same environment name. To avoid this, we need to unset the environment variable first or use t.Setenv() to set the environment variable.

Repeat yourself when necessary

Generally, we intended to keep the test as simple as we can. Sometimes duplicated test code might help us to understand the test case easier, if it makes the implementation simpler and easier to read, it is okay to violate the ‘do not repeat yourself’ principle.

Measure code coverage but focus on test coverage

The main difference of code coverage and test coverage is that code coverage only measure how well the production code is covered by the test, while test coverage measure how well the user requirement is covered.

In Vald, we measure the test quality by measuring the test coverage, we also focus on test coverage to ensure all the real-world cases are covered by the test.

In the next section, we will describe test coverage and code coverage more deeply.

Code coverage & Test coverage

Code coverage

The code coverage is an indicator to determine how many implementation codes are covered by the test code. It is examined by determining how much code is exercised when we run those test cases. It helps us to determine how reliable is the test implementation is. Higher coverage means the software is tested well by the unit test.

For example, we implemented the following function.

func calcSum(val ...int32) (sum int32) {
    if len(val) == 0 {
        return
    } else {
        for _, v := range val {
            sum += v
        }
        return sum
    }
}

In the implementation, it checks if the function argument length, and if it is more than zero it will calculate the sum and return the result.

For example, if the test implementation of the above function is as follows.

func TestCalcSum(t *testing.T) {
    sum := calcSum(1, 2)
    if sum != 3 {
        t.Errorf("calcSum return incorrect result")
    }
}

In the test implementation, it tests the calcSum function using 2 elements and checks the result of it.

In Go, we can execute the following command to execute the test and get the test coverage result from it.

$ go test . -cover
ok      github.com/vdaas/vald   0.201s  coverage: 80.0% of statements

It shows that the coverage is 80.0%, but what does it mean?

It means that only 80% of executable code is covered by the test, and 20% is not covered by the test.

The explanation is as below.

func calcSum(val ...int32) (sum int32) {
    if len(val) == 0 { // this statement is executed to check the argument length
        return // this is not covered by the test
    } else { // not executable code
        for _, v := range val { // this part is covered until the return statement below
            sum += v
        } // not executable code
        return sum
    }
}

Test coverage

Test coverage is a test type to ensure the functional quality of the software. The goal of test coverage is that we want to cover as many use cases as we can, even it is not required by users. Unlike code coverage, test coverage focuses on thinking about the testing data and covers the part which is not defined in the requirement. It helps to find more hidden bugs or defects in the software.

For each type of input data of the function, we can design different test data to improve test coverage.

func doSomething(i int) {
    // do something
}

Given the type int of the function input like the example above, we may implement mathematics calculation and comparison logic base on the input value. By doing these operations, there is a chance that value overflow happens. To avoid this issue, we should consider not only using normal int values ( e.g. 100, 1000, etc) but also including the following values to improve test coverage.

• math.MaxInt32
• math.MinInt32

If it is slice type, it may be more difficult to include empty value and empty slice in the test.

func doSomethingWithSlice(ss []string) {
    // do something
}

Given the type []string, with inappropriate handling, nil pointer error or unexpected behavior may happen. To avoid this problem, we should consider including the following values in your test to improve test coverage.

• nil
• []string{“”}
• []string{}
• []string{“string”}
• []string{“string1”, “string2”}

Other than thinking about the function input type, we also think about the boundary test cases. To think about the boundary test cases, we need to understand each input and the corresponding meaning.

For example, we need to define a function to check if the given age is adult. The age of adulthood is different in each country. In this example, we will use 20 years old, the adulthood age in Japan. It returns true if the input age is above and equal to 20, otherwise, it will return false.

func isAdult(a int) bool {
    if a >= 20 {
        return true
    }
    return false
}

Given this requirement, we should include the below test data in your test cases to cover all the possible cases in your test.

• 19
• 20
• 21

By covering these cases, we can ensure that the requirement is correctly implemented, to avoid possible bugs in your software. There should be more tests in this requirement, but you can get an idea of how we should check the boundary of the input value.

We believe that improving test coverage will improve the overall testing and software quality, to find defects inside your software more effectively and efficiently.

How do we implement unit tests in Vald?

Vald is implemented by Go. In Go, we use the standard testing framework to implement the unit test. To reduce the complexity of the implementation of the unit test, we found that table-driven test is suitable in Vald. We use a tool called gotests to generate the test template in table-driven format, with some Vald-specific modification on the template, to make it suitable for our use case.

For more about unit test implementation in Vald, please find our coding style of the unit test and please feel free to visit our github.

We will write another blog post to introduce how Vald team implements unit tests, please look forward to it :)

Conclusion

Thanks for taking the time to read this blog post. In this post, we introduced Vald and few unit test strategies applied in Vald.

It is important to deliver products as high quality as it can, and the unit test is one of the key elements to keep the product sustainable. We hope that this post is useful for you :)

If you are interested in this post, or if you are interested in Vald, please feel free to visit our official website or feel free to contact us. :)

We will keep updating the blog post in the future, see you again :)