ElasticSearch 入門基礎操作實例

Windows下安裝ES

進入到ES官網(wǎng)的下載頁面：
https://www.elastic.co/cn/downloads/elasticsearch
然后點擊下載，選擇版本，我這里選擇了最新版本：

image.png

然后在Windows下解壓，直接到解壓后的bin目錄中，執(zhí)行下面命令啟動：

D:\bigdata\elasticsearch-7.15.0\bin>elasticsearch

然后出現(xiàn)下面信息表示啟動成功：

[2021-09-24T09:43:44,940][INFO ][o.e.h.AbstractHttpServerTransport] [JTYSL-27LYMT2] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2021-09-24T09:43:44,941][INFO ][o.e.n.Node ] [JTYSL-27LYMT2] started

接下來在瀏覽器中訪問 http://localhost:9200/,得到看到下面的結果，說明整個ES已經(jīng)啟動可用

image.png

Windows下安裝Kibana

進入到ES官網(wǎng)的下載頁面：
https://www.elastic.co/cn/downloads/kibana
然后點擊下載，選擇版本，我這里選擇了最新版本：

image.png

然后在Windows下解壓，直接到解壓后的bin目錄中，執(zhí)行下面命令啟動：

D:\bigdata\kibana-7.15.0-windows-x86_64\bin\kibana.bat

接下來在瀏覽器中訪問 http://localhost:5601/,得到看到下面的結果，說明整個ES已經(jīng)啟動可用

image.png

ES基礎操作

一個簡單的查詢

ES使用REST API 的方式對接提供查詢接口，一個最簡單的查詢實例：

C:\Users\shikenian>curl -X GET http://localhost:9200/
{
  "name" : "JTYSL-27LYMT2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "EmJN6HdSTQuFvAzoMnSDRw",
  "version" : {
    "number" : "7.15.0",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "79d65f6e357953a5b3cbcc5e2c7c21073d89aa29",
    "build_date" : "2021-09-16T03:05:29.143308416Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

添加數(shù)據(jù)

通過POST添加數(shù)據(jù)到ES的簡單樣例（Windows下CURL不是很好用，我用的是POSTMAN）

POST URL
localhost:9200/logs-my_app-default/_doc?pretty
JSON參數(shù)
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "event": {
    "original": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] GET /images/bg.jpg HTTP/1.0 200 24736"
  }
}

返回值：
{
  "_index": ".ds-logs-my_app-default-2021.09.24-000001",
  "_type": "_doc",
  "_id": "sHr7FXwBEPmjP8ocrxhf",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

批量插入數(shù)據(jù)

HTTP類型：PUT
URL: localhost:9200/logs-my_app-default/_bulk
參數(shù)：
{ "create": { } }
{ "@timestamp": "2099-05-07T16:24:32.000Z", "event": { "original": "192.0.2.242 - - [07/May/2020:16:24:32 -0500] \"GET /images/hm_nbg.jpg HTTP/1.0\" 304 0" } }
{ "create": { } }
{ "@timestamp": "2099-05-08T16:25:42.000Z", "event": { "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" } }

結果：
{
    "took": 17,
    "errors": false,
    "items": [
        {
            "create": {
                "_index": ".ds-logs-my_app-default-2021.09.24-000001",
                "_type": "_doc",
                "_id": "t3oMFnwBEPmjP8ocCRiq",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 1,
                "_primary_term": 1,
                "status": 201
            }
        },
        {
            "create": {
                "_index": ".ds-logs-my_app-default-2021.09.24-000001",
                "_type": "_doc",
                "_id": "uHoMFnwBEPmjP8ocCRiq",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 2,
                "_primary_term": 1,
                "status": 201
            }
        }
    ]
}

查詢數(shù)據(jù)

接下來是應用Kibina的DEV TOOLS 來操作。

  • 查詢所有的數(shù)據(jù)
GET logs-my_app-default/_search
{
  "query": {
    "match_all": { }
  },
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

結果：
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "uHoMFnwBEPmjP8ocCRiq",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2099-05-08T16:25:42.000Z",
          "event" : {
            "original" : """192.0.2.255 - - [08/May/2099:16:25:42 +0000] "GET /favicon.ico HTTP/1.0" 200 3638"""
          }
        },
        "sort" : [
          4081940742000
        ]
      },
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "t3oMFnwBEPmjP8ocCRiq",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2099-05-07T16:24:32.000Z",
          "event" : {
            "original" : """192.0.2.242 - - [07/May/2020:16:24:32 -0500] "GET /images/hm_nbg.jpg HTTP/1.0" 304 0"""
          }
        },
        "sort" : [
          4081854272000
        ]
      },
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "sHr7FXwBEPmjP8ocrxhf",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2099-05-06T16:21:15.000Z",
          "event" : {
            "original" : "192.0.2.42 - - [06/May/2099:16:21:15 +0000] GET /images/bg.jpg HTTP/1.0 200 24736"
          }
        },
        "sort" : [
          4081767675000
        ]
      }
    ]
  }
}
  • 指定查詢某個列，不查詢所有列
查詢：
指定查詢 @timestamp 字段
不展示原始JSON文檔，也就是排除 _source
GET logs-my_app-default/_search
{
  "query": {
    "match_all": { }
  },
  "_source": false,
  "fields": [
    "@timestamp"
  ], 
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

結果如下：
{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "uHoMFnwBEPmjP8ocCRiq",
        "_score" : null,
        "fields" : {
          "@timestamp" : [
            "2099-05-08T16:25:42.000Z"
          ]
        },
        "sort" : [
          4081940742000
        ]
      },
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "t3oMFnwBEPmjP8ocCRiq",
        "_score" : null,
        "fields" : {
          "@timestamp" : [
            "2099-05-07T16:24:32.000Z"
          ]
        },
        "sort" : [
          4081854272000
        ]
      },
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "sHr7FXwBEPmjP8ocrxhf",
        "_score" : null,
        "fields" : {
          "@timestamp" : [
            "2099-05-06T16:21:15.000Z"
          ]
        },
        "sort" : [
          4081767675000
        ]
      }
    ]
  }
}
  • 時間范圍查詢
指定timestamp的時間范圍大小 >= <=
指定具體的查詢出來的列為timestamp
不展示原始JSON Object
GET logs-my_app-default/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2099-05-05",
        "lt": "2099-05-08"
      }
    }
  },
  "_source": false,
  "fields": [
    "@timestamp"
  ], 
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

結果:
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "t3oMFnwBEPmjP8ocCRiq",
        "_score" : null,
        "fields" : {
          "@timestamp" : [
            "2099-05-07T16:24:32.000Z"
          ]
        },
        "sort" : [
          4081854272000
        ]
      },
      {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "sHr7FXwBEPmjP8ocrxhf",
        "_score" : null,
        "fields" : {
          "@timestamp" : [
            "2099-05-06T16:21:15.000Z"
          ]
        },
        "sort" : [
          4081767675000
        ]
      }
    ]
  }
}

也有一些表達式，可以生成和當前日期相關的日期，例如：

"query": {
    "range": {
      "@timestamp": {
        "gte": "now-1d/d",
        "lt": "now/d"
      }
    }
  },
  • 從非結構化值中mapping出字段
1.請求中加入下面的mapping
2.在查詢的Field中加入指定的類型
GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "source.ip": {
      "type": "ip",
      "script": """
        String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
        if (sourceip != null) emit(sourceip);
      """
    }
  }, 
  
  "query": {
    "range": {
      "@timestamp": {
       "gte": "2099-05-05",
        "lt": "2099-05-08"
      }
    }
  },
  "_source": false,
  "fields": [
    "@timestamp",
    "source.ip"
  ], 
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

展示部分的查詢結果：
截取部分查詢結果，里面已經(jīng)多了source.ip，且該字段是從原始文檔中的event.original里面抽取出來
 {
        "_index" : ".ds-logs-my_app-default-2021.09.24-000001",
        "_type" : "_doc",
        "_id" : "sHr7FXwBEPmjP8ocrxhf",
        "_score" : null,
        "fields" : {
          "@timestamp" : [
            "2099-05-06T16:21:15.000Z"
          ],
          "source.ip" : [
            "192.0.2.42"
          ]
        },
        "sort" : [
          4081767675000
        ]
      }
  • 復雜條件查詢組合
    在定義mapping抽取source.ip和時間范圍查詢的基礎上。通過多source.ip 和 時間范圍一起做為過濾條件：
修改QUERY命令：
"query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
             "gte": "2099-05-05",
              "lt": "2099-05-08"
            }
          }
        },
        {
          "range": {
            "source.ip": {
              "gte": "192.0.2.0",
              "lte": "192.0.2.240"
            }
          }
        }
      ]
    }
  },

Bool查詢的作用：
相當于 and， 對bool下面的多個條件要同時符合的數(shù)據(jù)才能夠被篩選出來。

  • 聚合操作
    在aggs選中聚合的列，然后最終的結果會把聚合的結果放在JSON的尾部
GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "source.ip": {
      "type": "ip",
      "script": """
        String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
        if (sourceip != null) emit(sourceip);
      """
    },
     "http.response.body.bytes": {
      "type": "long",
      "script": """
        String bytes=grok('%{COMMONAPACHELOG}').extract(doc[ "event.original" ].value)?.bytes;
        if (bytes != null) emit(Integer.parseInt(bytes));
      """
    }
  }, 
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
             "gte": "2099-05-05",
              "lt": "2099-05-08"
            }
          }
        }
      ]
    }
  
  },
    "aggs": {
    "http.response.body.bytes": {
      "avg": {
        "field": "http.response.body.bytes"
      }
    }
  }, 
  
  "_source": false,
  "fields": [
    "@timestamp",
    "source.ip",
    "http.response.body.bytes"
  ], 
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

結果：
{
  ...
  "aggregations" : {
    "average_response_size" : {
      "value" : 12368.0
    }
  }
}

刪除數(shù)據(jù)

DELETE _data_stream/logs-my_app-default
?著作權歸作者所有,轉載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務。

相關閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容