Elasticsearch: An Introduction to Search, Log Management, and the ELK Stack
Are you looking for a powerful and flexible search solution for your project, or a toolkit for logging and monitoring a system composed of multiple services? Elasticsearch is an open-source search and analytics engine designed to process, search, and visualize large volumes of data at high speed.
1. Overview of Elasticsearch
1.1. What Is Elasticsearch?
Elasticsearch is a search engine built on top of Apache Lucene. It provides a distributed search engine with a comprehensive set of capabilities and an HTTP-based interface that supports JSON data. Elasticsearch is developed in Java and is available as open-source software under the Apache License 2.0.
Elasticsearch is a distributed, RESTful search and analytics engine.Official website: https://www.elastic.co/elasticsearch/
Elasticsearch can also be considered a document-oriented data store. Its primary purpose is to store and retrieve documents. In Elasticsearch, documents are represented in JSON format.
It is built on Apache Lucene, a search library with more than 15 years of experience in full-text indexing and retrieval.
When discussing Elasticsearch, there are several fundamental characteristics worth remembering:
- JSON-based data: Elasticsearch stores and queries structured and semi-structured documents in JSON format. Its indexing architecture is optimized for search performance, enabling near-real-time search across large datasets.
- RESTful APIs: Elasticsearch provides RESTful APIs that allow applications and external services to interact with it easily. This makes integrating Elasticsearch into existing applications flexible and convenient.
- Multiple data sources: Elasticsearch can ingest data from various sources, including application logs, system metrics, and other data generated by different applications.
1.2. Why Use Elasticsearch?
Elasticsearch is widely used for two important types of workloads: search and log management.
Search functionality is essential in modern websites and software systems, particularly in e-commerce applications. Finding products quickly and accurately has become increasingly important for delivering a good user experience.
In the past, developers could rely on traditional approaches by querying the database directly. However, these approaches can become slow and inefficient as datasets grow or search requirements become more sophisticated.
Elasticsearch stands out as a useful solution, particularly for full-text search, keyword matching, and language-specific search requirements, such as Vietnamese text.
Consider a SQL query using
LIKE '%one%'. The result will include any string containing the substring one, such as phone, zone, money, and alone. Depending on the application's requirements, this may produce many irrelevant results.With Elasticsearch, you can configure text analysis and query behavior to return results that better match the intended search terms. For example, searching for
one can be configured to prioritize documents containing that term rather than simply matching arbitrary substrings.Another consideration is accented text. A SQL query using
LIKE does not automatically normalize accented characters. For example, searching for co may not match cĂł, depending on the database's collation and comparison rules. Elasticsearch can be configured with appropriate analyzers and character filters to handle such language-specific requirements.Performance is another important advantage. A conventional SQL query using a leading-wildcard pattern such as
LIKE '%keyword%' may not be able to use a standard B-tree index efficiently. Elasticsearch builds an inverted index for searchable fields, allowing it to retrieve matching documents efficiently without scanning every document for many common full-text search operations.The ELK Stack addresses an important challenge in log analytics.
As more IT infrastructure moves to public cloud environments, organizations need effective solutions to manage and analyze logs from their infrastructure and applications. These sources may include server logs, application logs, and clickstream data.
The ELK Stack provides a practical and cost-effective log analytics solution for developers and DevOps engineers. It helps them extract valuable insights for troubleshooting, application performance analysis, infrastructure monitoring, and security investigations.
1.3. Key Characteristics of Elasticsearch
Elasticsearch stores complex data structures as serialized JSON documents. Once a document is indexed, it becomes searchable in near real time, typically within approximately one second under normal conditions.
Elasticsearch uses a data structure called an inverted index to support fast full-text search.
An inverted index maps terms to the documents in which they appear. Instead of scanning every document to find a particular word, Elasticsearch can use this index to identify matching documents efficiently.
Search
Beyond searching for individual terms, Elasticsearch supports phrase queries, fuzzy matching, prefix searches, and autocomplete suggestions.
When working with numerical or geospatial data, Elasticsearch can index non-text fields using optimized data structures that support efficient numerical and geographic queries.
Analyze
Elasticsearch's aggregation capabilities allow you to build complex summaries of your data and gain insights into important metrics, patterns, and trends.
Instead of merely searching for a needle in a haystack, aggregations allow you to answer questions such as:
- How many needles are in the haystack?
- What is the average length of the needles?
- What is the average needle length grouped by manufacturer?
- How many needles have been added to the haystack over the past six months?
You can also use aggregations to answer more nuanced questions, such as:
- Which needle manufacturer is the most popular?
- Are there any unusual clusters or outliers among the needles?
These capabilities enable you to analyze and visualize data in near real time. Reports and dashboards can reflect changes in your data, helping you make decisions based on the latest available information.
An Elasticsearch index consists of one or more physical shards, each of which is itself a Lucene index.
By distributing documents from an index across multiple shards and distributing those shards across multiple nodes, Elasticsearch can scale horizontally.
Elasticsearch also supports replication, helping protect against hardware failures and potentially increasing search throughput as additional nodes are added to a cluster.
As the cluster grows or shrinks, Elasticsearch can redistribute shards to maintain an appropriate balance across the available nodes.
1.4. Core Concepts in Elasticsearch
- An index in Elasticsearch is a collection of documents organized for searching and storage.
- An index can be thought of as a logical container for a particular dataset, although it is not exactly equivalent to a relational database.
- Indices are used to organize, search, filter, and store data. An Elasticsearch deployment can contain multiple indices.
- Before Elasticsearch 7.x, an index could contain multiple document types. Each type represented a category of documents within that index.
- Different types could define different document structures. For example, an index might contain
userandblogtypes, each representing a different kind of document.
- Mapping types were deprecated in Elasticsearch 7.x and removed in Elasticsearch 8.0. Modern applications generally use separate indices or other data-modeling strategies instead.
- A document is the fundamental unit of data that Elasticsearch stores and searches.
- Documents are typically represented in JSON (JavaScript Object Notation) format.
- For example, in an index containing user information, each user can be represented as an individual document.
- Fields are the individual attributes that make up a document.
- A field can contain values such as strings, numbers, dates, or other supported data types.
- For example, a user document might contain fields such as
name,age, andemail.
2. The ELK Stack Ecosystem
ELK Stack is an acronym for three widely known technologies: Elasticsearch, Logstash, and Kibana.
The ELK Stack provides a solution for collecting logs from applications and infrastructure, processing those logs, analyzing the resulting data, and visualizing it for monitoring, troubleshooting, security analysis, and other operational use cases.
2.1. Components of the ELK Stack
Elasticsearch is a distributed search and analytics engine built on Apache Lucene.
Its support for multilingual search, high-performance indexing, and flexible JSON documents makes it suitable for a wide range of search and log analytics use cases.
Beats are lightweight, open-source data shippers that can be installed as agents on servers to send operational data to Elasticsearch or Logstash.
Different Beats collect different types of data, including:
- Audit data: Auditbeat
- Log files and journals: Filebeat
- Cloud data: Functionbeat, historically used for serverless environments
- Availability monitoring: Heartbeat
- Metrics: Metricbeat
- Network traffic: Packetbeat
- Windows event logs: Winlogbeat
Developers can also build custom Beats or other data shippers using community libraries and programming languages such as Golang.
Logstash is an open-source data ingestion and processing pipeline.
It allows you to collect data from different sources, transform and enrich that data, and forward it to a destination of your choice.
With built-in filters and a large ecosystem of plugins, Logstash can process many different data formats and ingestion sources.
Kibana is a data exploration and visualization platform commonly used for log and event analysis.
It provides interactive charts, dashboards, filters, and aggregations. It also supports geospatial visualization, making it a useful interface for exploring data stored in Elasticsearch.
2.2. How the ELK Stack Works
A typical ELK Stack logging pipeline works as follows:
- Log collection: Logs are collected and forwarded to Logstash through various mechanisms. For example, a server may send log events through a network input, or a Beat may read log files and forward their contents to Logstash.
- Log processing and indexing: Logstash parses the incoming events, extracts relevant fields, and enriches them with metadata such as timestamps, IP addresses, source servers, severity levels, and log messages. The processed events are then sent to Elasticsearch for indexing and storage.
- Log exploration and visualization: When users want to inspect logs, they access Kibana. Kibana queries Elasticsearch and presents the results through an interface where users can search, filter, analyze, and visualize log data.
3. Elasticsearch Architecture
3.1. Architecture Overview
An Elasticsearch cluster consists of one or more Elasticsearch nodes connected to form a distributed system.
Let's first look at how the architecture is organized in the following diagram:
Although each node can have its own roles and responsibilities, nodes can also participate in coordinating client requests and forwarding them to the appropriate nodes.
The following are the main node roles in an Elasticsearch cluster.
- Master-eligible node
Master-eligible nodes participate in the master election process. The elected master node is responsible for lightweight, cluster-wide coordination tasks, including creating and deleting indices, tracking cluster nodes, and managing shard allocation.
By default, the master role is enabled. A master-eligible node can be elected as the cluster's master through the master election process.
You can disable this role for a node by setting
node.roles appropriately in the elasticsearch.yml configuration file.- Data node
Data nodes store indexed documents and handle data-related operations, including indexing, searching, and aggregations.
These nodes are responsible for performing operations against the shards assigned to them. Their roles can be configured through
node.roles in elasticsearch.yml.- Ingest node
Ingest nodes preprocess documents through ingest pipelines before the documents are indexed.
These pipelines can perform operations such as parsing, transforming, and enriching incoming documents. Ingest capabilities can also be configured through
node.roles.- Coordinating-only node
A coordinating-only node does not perform the master, data, or ingest roles. Instead, it coordinates client requests, routes requests to the relevant nodes, gathers and reduces search results, and distributes bulk indexing requests.
Every node can perform coordinating tasks, but dedicated coordinating-only nodes can be useful in larger deployments where request coordination needs to be separated from other workloads.
3.2. How Elasticsearch Indexes and Stores Data
In Elasticsearch, an index is a logical structure used to organize and store documents for searching. Although it is sometimes compared to a database in a relational system, the two concepts are not exactly equivalent.
You can create, delete, and manage indices through the Elasticsearch API.
For example:
- Create an index: Use the Create Index API.
- Delete an index: Use the Delete Index API.
These APIs allow applications and administrators to manage indices programmatically.
Elasticsearch uses mappings to define how document fields are interpreted and indexed.
There are two common approaches to defining mappings:
- Dynamic mapping: Elasticsearch automatically detects field types and creates mappings when new fields appear in documents. This is convenient when ingesting new data quickly, but automatic type detection may not always match the application's intended data model.
- Explicit mapping: Developers define field types and indexing options before indexing documents. This provides greater control over data representation, query behavior, and indexing performance.
For production systems, carefully designing mappings can help avoid unexpected field types and unnecessary index growth.
An analyzer processes text during indexing and, depending on the query type, during search as well. It typically consists of character filters, a tokenizer, and token filters.
A tokenizer is a specific component that divides text into individual terms according to a defined set of rules.
Common examples include:
- Standard Analyzer (default): Tokenizes text according to language-independent word-boundary rules and converts tokens to lowercase. For example,
Quick Brown Fox!can produce the termsquick,brown, andfox.
- Whitespace Analyzer: Splits text wherever whitespace occurs.
- Custom Analyzers: Developers can combine character filters, tokenizers, and token filters to meet specific application requirements.
Choosing the right analyzer is particularly important for multilingual applications, where differences in language, punctuation, and character normalization can affect search results.
Elasticsearch provides a powerful Domain-Specific Language (DSL) for querying and searching indices. This JSON-based query language supports a wide range of search operations.
Common examples include:
- Match query: Searches analyzed text for terms that match the query.
- Bool query: Combines multiple queries and conditions using clauses such as
must,should,filter, andmust_not.
- Term query: Matches exact indexed terms, commonly used for keyword fields and other non-analyzed values.
- Range query: Finds documents with values within a specified range.
These are just a few examples. Elasticsearch DSL is highly flexible and supports complex queries, aggregations, and combinations of filtering and scoring conditions.
Remember to adapt your queries to your index mappings and application requirements.
4. Conclusion
In summary, Elasticsearch is more than just a database. It is a powerful distributed search and analytics engine designed to retrieve, process, and analyze large volumes of data.
Its fast search capabilities, flexible querying, and rich analytical features make it an important tool for extracting insights from complex datasets.
Meanwhile, the ELK Stack—Elasticsearch, Logstash, and Kibana—provides an integrated solution for collecting, processing, searching, and visualizing data.
By combining these components, development and operations teams can monitor applications, investigate incidents, analyze system behavior, and manage operational data more efficiently.