Hello everybody today what we want to discuss really interesting topic of web development so for today is going to be elastic search
We're going to discuss some basics
And specific sorts work and how to integrate properly with knowledge and some other things. So let's start
And actually we'll the first question what I believe we need to resolve before we start in section-ally
Why do we need is technology at all? There are lots of technologies. Why do we need exactly this one?
So what main reason here is?
users of her assertion
starting from with the simplest
landing page and ending with
complicated CRM system
They want to search because nobody wants to waste their time just searching for content manually. Actually, that's why we have Google
So, what does this mean for us common web developers
It means well actually storage functionalities, one of the most important things in the entire application
So if your search is not responsive enough or a user will tell you for sure I don't want to use your system
I'll just go to some other one whereas search will be responsive and fast
So we need to think carefully about search we need to optimize it and we need to be sure that it's fast enough
for comfort usage
Actually here you can tell me okay
I can just do select from my favorite sql database and that's it
Of course, of course you can do it
you can do something like this using like a crater do some conditions between all the parameters that you want to have your
result data set
But there is one big problem
first of all many just I believe almost every
SQL databases have small disclaimer inside very documentation about such operations
Which will tell you
attention this can be
Heavy up creation try to avoid usage of this one even know sequel databases like MongoDB have such disclaimers
Everything what you should take into account join of data
so when you want to do some joint operations, it can make your query even heavier and
If you just plan schools two things you'll see immediately if I still have a problem with performance
Especially when your data size is large or you need to query many parameters like we have here in test client entry
Actually, both two problems are not new ones and
Developers of sequel databases try to resolve it by themself how they could do it
So some of databases like PostgreSQL that are going to use for today
It provides like footage search native
You can try to use this one but you'll understand immediately, but but it's like just
operators so you can do substitute search for example, and
Actually, that's not already
Serious enough. I mean nobody wants to search starting from exactly start of string
You just want to type something see results and that's it. And
Actually, it pushes us and not only ask because many other developers try to resolve those three problems
Well first first and second
I believe in the first place and with third one is kind of a problem for developers because if you want to use again
native in society SQL database you'll see problems immediately and
Actually trying to resolve those problems. We created something that is called search engine
Actually elastic search is search engine
Main purpose of the screen is formatting storing and retrieving of data
So yeah, but simple but with one small exception
When search engine creates something that is called search index but set of formatted data
It uses some smart algorithms to optimize
querying for this data
just
Speeding up your queries for some data a changes order. It can store it in different format
And that saturates that right and want to go to events with such algorithms. You can just google one if you're interested in
but the main thing here
Actually shaped perfectly for assertion in your data
And it can do much faster assertion when you will ever do with native sequel
simple and instrument tray
But actually the solution it seems to be perfect but to use it properly in the first place
You need to know your data and of a second one. You need to know your queries
So what I'm trying to say here it is not required
Really even forbidden just work everything that you'd have inside sushi index
So for example even here you can see the difference
I have this set
of data inside
but I query only for some subset from one entry like four six progresses instead of instead of eight and
in reality of this subset of a query word
It can be even smaller like two three properties from my entire entry to search for
And you need to know where well needs of your new user
You need to plan it before you start to create search index of rice will fail because again, no magic
of course, you have smart algorithms based big things up, but we can speed things up just were like
Entry were twenty properties and be efficient. Now you need to plan everything accordingly as for any our technical solutions
You need to know your data need to know your queries plan everything accordingly before you start
So again before
We're before we start with some live example. I want to underline our step for today
So it's going to be no jazz was SQL elasticsearch a search engine, and I'm we're on front and side
So and
actually our
Components what we're going to using how they interact with each public
So everything starts with front ends request to our API?
Like the user wants to search for customers actually
customers will have the same data schema as we used for explanation for
And actually let's see what happens. So real data. So here I have data grid over both centuries actually
designed in minimalistic manner, but anyways, I have around 200 pages so I have
10,000 entries here 50 for each page. So as you can see we have pretty large data set notice or
For example, I want to search for some specific country like 1 2 3 4
Actually, I should have it somewhere inside account names because I trade it exactly this one
So and as you can see data comes immediately just immediately
so request is super fast and
If you open developer console and check request, you'll find exactly the same
Same schema that we used in our chart. So what happened here? I send a request with our search criteria
That's how is going to work on your client side. So only thing that you need to provide for the server or relevant
Press the cayenne point and some search query version and that's it
So let's go back to our chart what happens on API side Beverly?
so on the first place API request elasticsearch server, I would your query and
The last user server returns you various IDs like unique identifier were all the way interested it found
And after you can use it just for query the data from progress. We also know search actions. No substring
Nothing is required from your actual database so
responsibility for certian fully goes to the elastic search server
Your database is just responsible for data storage and getting them back to you. And that's it
If you know what I want to underline here as well as you can see word here
There's a bit exotic like 9 to 0 0 actually, what's a standard word for elastic search?
So if you just install it your server will be up immediately and you can start to use it for indexing
So, um, let's tell tell about our workflow actually
It's related to data processing
So what happens with data processing normally client wants to do a few things of the data music obviously. No so it can request for
Create update or delete a country actually read operation already discuss tips or crud is not full here, but nevertheless
and what happens here as you can see we want to
Operations with data two times
I want to underline it two times not one because first of all you want to store something inside your
database like paas or SQL
kind of straightforward
but in other hand you want to store your search index to search for data and
Actually one more time. I want to underline it you have data duplication. And actually that's the only way how
Search Indexing kind of work you provided some data it stores it separately
It optimizes version four B's data chunks and after you retrieve the data from the database
That's how it works. And actually it can sound a bit
frustrating and complicated because
approaches with relational databases
Hopefully tell us directly duplication of data is bad
well, not really nowadays because well disk space is not so expensive as it used to be and
It's not bad when you store some duplicates to increase performance of your application
It's kind of widely used practice. So
For us that's the only way here used to duplicate of this data here
but main question now how to make this approach be easy enough and extendable enough and
Not lose a like connection between both two type of data
Well at least because if you have data inconsistency, you can have some problems like use port index and elasticsearch
but there is no related ID inside progress for this data and in
Opposite way it works the same so it can break logic of application
and actually
We can just provide some middleware so what I'm trying to say here
In the common way API just tells your database server Postgres in our case
Create update or delete entry with sequel query and with it
but in our case
We're really good approach would be usage of forum. So
object model for our tables not usage sequel immediately from your endpoint
Of course, you can of course, if you go to the code and find our and point inside rules
So you can see just here. I have only get requests because I don't create anything in application
But you can have just post requests next to it and in the first place send sequel query to database for
entry creation after you can just send an hour query to
Search engine and create index there. Of course you can but we're talking about extendable code which can be
Actually extended with extra properties or extra date easily. We've also write in many things
that's why I
would like to recommend you usage of OM or PostgreSQL ver is a great Orem school supplies so we can try with this one and
basically
What does this mean for us and the match of the code?
Let's look into the model actually model small tutorial for guys who are not familiar with Orem
Model is just an object representation of a table. So so so easy basically it represents
entry inside of a table and it provides you many mijos to interact with
Your data like creation of entries deletion of entries update
And some other functionality, but we know what we want to do here. We want to expand functionality but Orem provides to us
So in the first place, we want to store some config for our model. For example, we have model customers and
I'm going to index only few of our purposes in the data like account name city and country
We don't want to store everything. Actually. That's a golden rule of
Let's innovate. Hmm actually told you before
So you don't want to store everything you want to store only the things what you are going to search for
So in the first place we want to extend this
Moderate presentation for our data with some configuration for elastic search I believe that's quite understandable
The second thing
Telev in the trick
before
We tell our main code part like to API that operation is done
Before we do it actually we need to do one extra action kind extension
So how it's going to work user requests an update from API or create or delete?
you tell - ORM because it's now an interface for our
sequel database
It's in the first place
sends a comment to sequel database with appropriate query to create something or update something or delete or october and
Somewhere between and this operation and before we tell API what everything is done even modify user but everything is ok
We need to inject one more action. What will go inside elastic search and
Create appropriate index for us
And actually that's the truth for any operational to want to do like whenever we want to create update or delete
entry we will just
Do this extra operation?
So let's look into the code example how it can be done
Actually some of the databases we already have such functionality
I'm gonna worms for databases like Mongoose or M. It has special plugin
It's Mongoose elastic, and actually it provides setup decorators for models, and they do exactly based thing
So what's going to happen here?
here we just take
Options for our indexing I said you before why do we need them?
After ways we are going to say original operation. So original movement
already tails and the soul of a magical for sequel database and
After we're going to overwrite our original create method
with extra action
so I
What's going to happen here in the first place with a common action original create create entry after your entry is created
we're going to take all the required data from the century and
Simply index it inside elasticsearch and send exactly the same status that our program
To API and that's it
And actually you can override any of factions in this manner
And with it, so all of your models will Auto index every day that goes for them
so what's why I call that middleware because it's it's just between
actual requests and database and just
mirrors illuminator and
you can do the same with for Saoirse quest so you can extend model search functionality as well likes and search and we'll just
sensors to your
Master engine and absolute will filter results from the database for you. And that's it
So
one more time how approach this goes user request something from iti you request from om or is
extended with
Mironova data into the last sip server server
first of all, you save data into common data storage up through save it here and
It's true for any operation by changing the data
So one last thing for today, but I want to tell for you guys is
Well, that's basically about silver bullets
So there are no silver bullets in development is you obviously know and if you see that massive Sergius doesn't work
As you want it to work or you search queries just all that can be kind of primitive
More like you search only for one parameter and only in one model and it means it's no reason to integrate 1 over technology
If search in your case can be done like with some other way
Is like on client only if you have just a really small set of thrush results
Well, just get rid of last six your server into the client site. For example, it can be true for
Landing page will have just a really small piece of content. You can just return it as
An object and filter it there if you see that it won't hire all the traffic
but if you have a large set of entries that you need to search through all
Volcom elasticsearch on word and try it
So, thanks for attention
All blinks for code sample and some other materials can be found
Under this video. If you have any question for me, please ask me and we can discuss it and delivery solve it
So, thanks for attention one more time guys. See you. Bye. Bye