Amazon S3 Vectors adds metadata pre-filtering for improved recall in filtered searches

New Metadata Pre-filtering Feature

Amazon S3 Vectors has introduced metadata pre-filtering, a new capability that processes metadata filters prior to executing a similarity search. This functionality is designed to enhance the accuracy of filtered queries by ensuring that only vectors matching the specified metadata criteria are considered in the subsequent similarity search.

Functionality and Scope

The pre-filtering feature allows users to filter on various attributes such as tenant, category, status, or time. It also includes prefix matching with '$startsWith' for paths, URLs, and hierarchical keys. Each vector can store up to 2 KB of filterable metadata, and a single query can incorporate up to 100 filter constraints. This update is available without additional cost, re-ingestion, or changes to existing queries.

Impact on Search Recall

By evaluating metadata filters first, the system returns a higher proportion of relevant matches within the filtered subset of an index. This is particularly beneficial for applications that typically search only a portion of a larger index, such as those scoped to a specific user or category. The result is improved recall for filtered searches, meaning more pertinent results are retrieved.

Use Cases

This feature is applicable in scenarios where search results require both relevance and correct scoping. Examples include legal and professional services filtering documents by client or matter number, financial services searching analyst notes by issuer or date, and media and entertainment services filtering content by rating or licensing. Agentic applications also benefit by ensuring searches cover material relevant to a user's session, improving task reliability.