- [Rupesh] Are slow queries and lengthy analysis processes
hindering your productivity?
Discover OpenSearch Msearch feature
for bulk vector search
optimizing query efficiency
and accelerating response times.
Say goodbye to sluggish analysis
and embrace the data-driven efficiency.
Hi there, I am Rupesh Tiwari,
a senior solutions architect at Amazon Web Services
specializing in data analytics and OpenSearch.
Together we will unlock your data's potential.
So join me on this transformative journey
to revolutionize your analysis
and empower your organization
with confident data-driven decisions.
Let's unleash the power of OpenSearch bulk vector search
and dive in.
OpenSearch powered by Apache Lucene
is a community-driven, open-source,
full-text search engine.
With OpenSearch Service on AWS, harness its capabilities
to build scalable cloud applications.
This robust engine enables text analytics
with full-text searches, aggregations and boolean searches
leveraging the accurate BM25 algorithm.
It goes beyond text analytics
by integrating machine learning tools
for generating and storing vector embeddings,
enabling semantic and predictive searches.
Unlock advanced data analysis
and search functionalities with OpenSearch.
Explore the potential
of this community-driven, open-source solution
combined with AWS scalability and managed services.
Customers often encounter challenges
when handling large vector data sets
and performing multiple searches
report generation and batch operation using API calls.
Let's consider a customer who faced similar difficulties
making 32,000 API calls for vector search in OpenSearch.
However, this approach brought significant issues.
Response time increased
leading to performance degradation, resource exhaustion,
strain CPU, memory and network bandwidth,
network congestions, cost, package loss, and latency.
Scalability challenges arose impacting performance
when exceeding systems limit.
They experienced performance reductions of 30 to 50%.
Despite their attempt to implement rate limiting
and concurrency management,
they did not achieve the desired improvements
in performance and stability.
Luckily, OpenSearch has a surprise for you.
The multi-search API,
it's a game changer in a data retrieval.
Perform multiple searches in single API call
saving time and boosting performance.
Say goodbye to latency and unnecessary network traffic.
Bundle your queries with the Msearch endpoint
and unlock the full potential of your applications.
OpenSearch multi-search APIs here
to surprise you with seamless data analysis
and unmatched efficiency.
Get ready to be amazed.
This example highlights the power of Msearch feature
for bulk k-nearest neighbors, also known as KNN search.
KNN enables users to find the k-nearest neighbors
to a query point in a vector index.
By utilizing Msearch endpoint,
we bundle multiple KNN search request
into a single API call.
Customization options include the vector field,
query vector and number of nearest neighbors.
Optimize your search process
with Msearch achieving faster results and reduced overhead.
The API response includes results from each KNN queries.
Join me for a live demo
where I will showcase the implementation
of Bulk KNN search using Msearch in OpenSearch.
Let's dive in and unlock the potential of this feature.
I have already set up an OpenSearch domain in my AWS account
and access the Dev Tools in the OpenSearch dashboard.
If you have not created an OpenSearch domain yet,
I recommend following the AWS developer guide documentation
to do so.
This will allow you to follow along with me
and make the most of our session together.
In this demo, I will guide you
through a simplified example of OpenSearch
bulk vector search capabilities.
Please note that this demo provides a basic understanding
but can be expanded for large-scale API call handling.
First, I will create an index
with properties named as vector5 and vector6
forming the foundation for vector search.
After that, I will try to go ahead
and put some data into OpenSearch.
So after ingesting of sample data,
using prices as vector,
we can perform a search to find the price range
associated with a specific vector value:
two, three, five, six.
And you can notice it is resulting in a range
of the prices 5.5 and 8.9.
Next, let's explore the potential of multiple queries
in a single search operation.
OpenSearch Msearch feature
and enables simultaneously execution of multiple queries.
By specifying the index name and search parameters
for each query, we can handle numerous queries efficiently.
So I'm going to execute the Msearch operation.
So after executing the Msearch operation,
it retrieves the prices associated with each query.
This demo demonstrates the efficiency of OpenSearch
in handling both searches
and providing relevant results,
such as the price like 5.5 and 8.9 for the first query,
and the other numbers for the other queries.
With Msearch, we eliminate the need
for individual API calls in a loop.
Instead, we optimize the search process
by sending a single document with multiple queries.
Improving efficiency akin to bulk indexing.
Although this demo showcases a simplified example,
it's important to note that the bulk vector search
can handle a significant number of API calls
in a single operation.
OpenSearch architecture and scalability
allow for high-performance searches
across various real-world scenarios.
I hope this demo has provided valuable insights
into OpenSearch potential for bulk vector search.
Thank you for joining me, and I look forward
to exploring advanced use cases in the future.
Unlock the potential of your data with ease
using Amazon OpenSearch Service.
Feel free to reach out to us
if you have any further questions, comments,
or thoughts, or suggestions.
We are here to support you on your journey
with Amazon OpenSearch Service
and ensure you get the most out of your data.
Our team is ready to assist and provide guidance
as you explore the capabilities of OpenSearch.
So do not hesitate to contact us.
We value your feedback and are committed to your success.
Thank you.