Weaviate Average Querying Time for Scenarios S1 through S9.

Conclusions

We set out on this survey to compare popular open-source Vector Similarity Engine (VSE) solutions to facilitate embedding search through approximated nearest high-dimensional vectors. These new frameworks enable a more efficient approach to storing vectorial data. However, they are still relatively recent yet subject to fundamental feature development, such as horizontal scaling, pagination, sharding and GPU support.Here at FARFETCH, we are hoping to accommodate a product catalogue from 300k to 5 million products and hidden relations between products, queries and products and other customer impression data. We need the VSE to support a large dataset with several vectorial representations and efficiency for multiple accesses to fulfil our vision. Therefore, it is vital to ensure that we meet the following criteria in this research:

• The VSE provides high quality (accurate) results;
• Rendered possible by low-level implementations of performing index types.
• The VSE indexes the embeddings in a satisfactory timely fashion;
• Rendered possible by low-level implementations of performing index types.
• The VSE executes queries at high speed;
• Rendered possible by low-level implementations of performing index types.
• The VSE allows horizontal scaling to accommodate load balancing and data replicas to protect against hardware failure while increasing service capacity;
• Sharding.
• The VSE provides a top K results collection, but the system that integrates it can iterate over the top K results by a window of size N.
• Pagination.

This experiment focused chiefly on two engines: Milvus and Weaviate. Quality analysis of the retrieved results was not included in the context of this study, as it would require an additional configuration exploring:

• the used embedding model(s) to encode information;
• the index type in use.

Given that the experimental setup depended on a similar configuration throughout the studied engines, we fixed the indexing algorithm on HNSW. For the time being, please refer to this blog article to understand the impact of the indexing algorithm on search speed, quality and memory.

From the analysis on the indexation and querying times, Milvus consistently outperformed Weaviate, emphasising the indexing time for scenario S9, closely resembling the FARFETCH product catalogue's dimensions.

These technologies did impose technical limitations, such as a lack of support for multiple encodings (effectively warranting a change to adapt scenarios S4 to S9). However, in

References

[1] Milvus: A Purpose-Built Vector Data Management System

[2] https://github.com/erikbern/ann-benchmarks

Acknowledgements

This work was partially funded by the iFetch project, Ref. 45920, co-financed by ERDF, COMPETE 2020, NORTE 2020 and FCT under CMU Portugal.