Elasticsearch and the Elastic stack

ELASTICSEARCH

THE PROBLEM OF SEARCH!

WHY IT's SUPER DIFFICULT, aND WHY IT's SUPER IMPORTANT

Search is often one of the core Features for most platforms

SEARCH at quizizz

But let's take a moment to talk about the problem statement of search

​Problem statement

​FROM

​Problem statement

​TO

But it's not very easy to do

Imagine searching through millions, if not billions of documents when...

TYPOS

WHEN You may want custom logic like showing out of stock items in the end, or sorting by ratings

RELEVANCE

ELASTICSEARCH AND THE ELASTIC STACK

AND HOW IT ALL FITS TOGETHER

INTRODUCING Elasticsearch!

​INTRODUCING Elasticsearch!

ElasticSearch is a document oriented database

​

​

​INTRODUCING Elasticsearch!

At the first glance it may seem very similar to Mongo

​

​

​But it's actually a really powerful full text search database

​INTRODUCING Elasticsearch!

We'd come to why it is an extremely powerful search engine soon,

​

But for now, understand that it scans every single document and maintains a list of every unique word that appears in any document and identifies which documents contain it

​INTRODUCING Elasticsearch!

Apart from all this,

​

​

​INTRODUCING Elasticsearch!

Apart from all this,

​

​1. It is extremely scalable, promising almost linear scaling*

​

​

​

​

*This has some conditions, more on it later

​INTRODUCING Elasticsearch!

Apart from all this,

​

​1. It is extremely scalable, promising almost linear scaling*

​

​2. Supports a lot of integrations with various tools

​

​

*This has some conditions, more on it later

​INTRODUCING Elasticsearch!

Apart from all this,

​

​1. It is extremely scalable, promising almost linear scaling*

​

​2. Supports a lot of integrations with various tools

​

​3. Can do search(among other things) really well

*This has some conditions, more on it later

​Where does Kibana fit in?

Kibana is a tool built by the same company elastic, and it works mostly as a dashboarding tool

DEMO!​

​https://demo.elastic.co/app/kibana

​What is logstash?

Logstash is another tool by elastic that helps in ingesting data into ES(and other databases)

​

It allows us to parse data, transform it, and load it into ES​

​How it all fits together

OPERATIONS ON ELASTICSEARCH

​http://localhost:5601/app/dev_tools#/console

Architecture​

INVERTED INDICES AND FULL TEXT SEARCH

​How does Elasticsearch index data

ElasticSearch essentially builds an inverted index, which would map every unique word in all the documents to every document that contains it

​Score

Score mainly considers two parameters,

the total number of times the search term comes up in the document​, and

how frequently the term comes up in other documents​

Architecture .....AGAIN​

​Nodes, Shards, Clusters

Let's Walk through a Scenario:

Assume we are a small startup

​Implement a simple search

Architecture | Nodes, Shards, Clusters

​IT WORKS!

Architecture | Nodes, Shards, Clusters

Architecture | Nodes, Shards, Clusters

​We are too successful....

Architecture | Nodes, Shards, Clusters

Let's look at the problems

​Problems with our current Architecture

​All the data is stored in a single node, which leads to a single point of failure

​

​

​

​

Let's look at the problems

​Problems with our current Architecture

​All the data is stored in a single node, which leads to a single point of failure

​

ES is complex, for every document, it scans the entire document and it maintains an inverted index of every unique word in the doc in the memory, which means there is too much load on the single node since its resources are limited.

​

​

Let's look at the problems

​Problems with our current Architecture

​All the data is stored in a single node, which leads to a single point of failure

​

ES is complex, for every document, it scans the entire document and it maintains an inverted index of every unique word in the doc in the memory, which means there is too much load on the single node since its resources are limited.

​

Since all the requests are being handled by the single node with limited CPU/memory, users get slow responses​

​A BETTER ARCHITECTURE

Architecture | Nodes, Shards, Clusters

​We add another node to the architecture, which can handle half the requests and store half the data!​

​A BETTER ARCHITECTURE

Architecture | Nodes, Shards, Clusters

Just to build some terminology, we'd combine multiple documents into a shard, and say that a single node contains multiple shards

​Everything seems great...

Architecture | Nodes, Shards, Clusters

​BUT EVENTUALLY SOMETHING FAILS

Architecture | Nodes, Shards, Clusters

​AND we lose one node, along with all its DATA

Architecture | Nodes, Shards, Clusters

​BUILDING A MORE ROBUST ARCHITECTURE

Architecture | Nodes, Shards, Clusters

​We add Replica Shards, that contain the same documents as the primary shards.

​

We also ensure that ​a replica shard and its associated primary never live on the same node

​

​COST OF USING ELASTICSEARCH?

​What we know so Far....

​

​1. It is extremely scalable, promising almost linear scaling*

​

​2. Supports a lot of integrations with various tools

​

​3. Can do search(among other things) really well

*This has some conditions, more on it later

​But what does it cost

​

​

​1. It is heavy. A lot of the inverted index is maintained in heap, which means it requires a lot of memory to serve requests. This also limits its ability to perform as a primary DB.

​

​2. It is difficult to manage. Along with these layers of abstractions of shards, indexes, clusters comes the responsibility of managing them. It is time consuming or even impossible to change a lot of configuration without reindexing everything

​

​

*This has some conditions, more on it later

​But what does it cost

​

​

​3. Adding new documents can be time consuming, since it needs to read every word, it takes a little time before documents are searchable.

​

​4. Querying is not straightforward. Any query that involves multiple shards may require a lot of network hops, compute of other nodes, and may be slow

​

​

*This has some conditions, more on it later

​A common architecture

*This has some conditions, more on it later

A FEW POINTS I MISSED...

​What I missed

​

​

Inverted index is just one type of index, but ES would use other indexes for datatypes apart from text.​

​

​

​What I missed

​

​

There are other tools in the Elastic Stack that I didn't include in this discussion

​

​

​What I missed

​

​

Different type of nodes, like master nodes and replica nodes.

​

​

​SEARCH IN QUIZIZZ!

​AMAZON Opensearch

​

Our ES clusters are hosted on Amazon Opensearch

​

​

​AMAZON Opensearch

​

Our ES clusters sit behind search service

​

​

​AMAZON Opensearch

​

​This also provides a hosted Kibana environment

​

​AMAZON Opensearch

​

Demo!

​https://us-east-1.console.aws.amazon.com/esv3/home?region=us-east-1#opensearch/domains/dev-testing

​

​