1.Mac電腦安裝
brew install elasticsearch
#最后幾行可以看到相關(guān)的配置目錄
Data: /usr/local/var/lib/elasticsearch/
Logs: /usr/local/var/log/elasticsearch/elasticsearch_guodong.log
Plugins: /usr/local/var/elasticsearch/plugins/
Config: /usr/local/etc/elasticsearch/
可以安裝
https://github.com/NLPchina/elasticsearch-sql
可以用SQL語句查詢ES的內(nèi)容。
2.安裝中文分詞插件
https://github.com/medcl/elasticsearch-analysis-ik
根據(jù)es版本選擇分詞插件的版本，剛剛安裝的6.7。
mv elasticsearch-analysis-ik-6.7.0.zip /usr/local/var/elasticsearch/plugins/
3. 初步使用
3.1測試分詞器
使用默認(rèn)的分詞器
curl -X POST "localhost:9200/_analyze" -H 'Content-Type: application/json' -d'
{
"analyzer": "standard",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog\u0027s bone."
}'
返回結(jié)果如下
{"tokens":[{"token":"the","start_offset":0,"end_offset":3,"type":"<ALPHANUM>","position":0},{"token":"2","start_offset":4,"end_offset":5,"type":"<NUM>","position":1},{"token":"quick","start_offset":6,"end_offset":11,"type":"<ALPHANUM>","position":2},{"token":"brown","start_offset":12,"end_offset":17,"type":"<ALPHANUM>","position":3},{"token":"foxes","start_offset":18,"end_offset":23,"type":"<ALPHANUM>","position":4},{"token":"jumped","start_offset":24,"end_offset":30,"type":"<ALPHANUM>","position":5},{"token":"over","start_offset":31,"end_offset":35,"type":"<ALPHANUM>","position":6},{"token":"the","start_offset":36,"end_offset":39,"type":"<ALPHANUM>","position":7},{"token":"lazy","start_offset":40,"end_offset":44,"type":"<ALPHANUM>","position":8},{"token":"dog's","start_offset":45,"end_offset":50,"type":"<ALPHANUM>","position":9},{"token":"bone","start_offset":51,"end_offset":55,"type":"<ALPHANUM>","position":10}]}%
可以看出默認(rèn)的分詞器standard對英文分詞有著不錯的效果。
3.1 創(chuàng)建一個索引
一個索引相當(dāng)于一個數(shù)據(jù)庫,每個索引都可以有自己的單獨的單獨配置。
中文測試
curl -X POST "localhost:9200/_analyze" -H 'Content-Type: application/json' -d'
{
"analyzer": "standard",
"text": "創(chuàng)業(yè)邦，幫助創(chuàng)業(yè)者走向成功的平臺"
}
'
{"tokens":[{"token":"創(chuàng)","start_offset":0,"end_offset":1,"type":"<IDEOGRAPHIC>","position":0},{"token":"業(yè)","start_offset":1,"end_offset":2,"type":"<IDEOGRAPHIC>","position":1},{"token":"邦","start_offset":2,"end_offset":3,"type":"<IDEOGRAPHIC>","position":2},{"token":"幫","start_offset":4,"end_offset":5,"type":"<IDEOGRAPHIC>","position":3},{"token":"助","start_offset":5,"end_offset":6,"type":"<IDEOGRAPHIC>","position":4},{"token":"創(chuàng)","start_offset":6,"end_offset":7,"type":"<IDEOGRAPHIC>","position":5},{"token":"業(yè)","start_offset":7,"end_offset":8,"type":"<IDEOGRAPHIC>","position":6},{"token":"者","start_offset":8,"end_offset":9,"type":"<IDEOGRAPHIC>","position":7},{"token":"走","start_offset":9,"end_offset":10,"type":"<IDEOGRAPHIC>","position":8},{"token":"向","start_offset":10,"end_offset":11,"type":"<IDEOGRAPHIC>","position":9},{"token":"成","start_offset":11,"end_offset":12,"type":"<IDEOGRAPHIC>","position":10},{"token":"功","start_offset":12,"end_offset":13,"type":"<IDEOGRAPHIC>","position":11},{"token":"的","start_offset":13,"end_offset":14,"type":"<IDEOGRAPHIC>","position":12},{"token":"平","start_offset":14,"end_offset":15,"type":"<IDEOGRAPHIC>","position":13},{"token":"臺","start_offset":15,"end_offset":16,"type":"<IDEOGRAPHIC>","position":14}]}%
結(jié)果會把每個漢字都分詞，效果不好。下面采用ik分詞器進行分詞測試。
curl -X POST "localhost:9200/_analyze" -H 'Content-Type: application/json' -d'
{
"analyzer": "ik_max_word",
"text": "創(chuàng)業(yè)邦，幫助創(chuàng)業(yè)者走向成功的平臺"
}
'
分詞良好
{"tokens":[{"token":"創(chuàng)業(yè)","start_offset":0,"end_offset":2,"type":"CN_WORD","position":0},{"token":"邦","start_offset":2,"end_offset":3,"type":"CN_CHAR","position":1},{"token":"幫助","start_offset":4,"end_offset":6,"type":"CN_WORD","position":2},{"token":"創(chuàng)業(yè)者","start_offset":6,"end_offset":9,"type":"CN_WORD","position":3},{"token":"創(chuàng)業(yè)","start_offset":6,"end_offset":8,"type":"CN_WORD","position":4},{"token":"業(yè)者","start_offset":7,"end_offset":9,"type":"CN_WORD","position":5},{"token":"走向","start_offset":9,"end_offset":11,"type":"CN_WORD","position":6},{"token":"成功","start_offset":11,"end_offset":13,"type":"CN_WORD","position":7},{"token":"的","start_offset":13,"end_offset":14,"type":"CN_CHAR","position":8},{"token":"平臺","start_offset":14,"end_offset":16,"type":"CN_WORD","position":9}]}%
3.2 創(chuàng)建索引并使用分詞器
curl -X PUT "localhost:9200/cyzone" -H 'Content-Type: application/json' -d'
{
"settings":{
"index":{
"number_of_shards":3,
"number_of_replicas":2
}
},
"mappings":{
"goods":{
"properties":{
"name":{
"type":"text",
"analyzer": "ik_max_word",
"search_analyzer": "ik_smart"
},
"content":{
"type":"text",
"analyzer": "ik_max_word",
"search_analyzer": "ik_smart"
},
"id":{
"type":"long"
}
}
}
}
}'
3.3給索引的type為goods的添加一些數(shù)據(jù)
? ~ curl -X POST "localhost:9200/cyzone/goods" -H 'Content-Type: application/json' -d'
{
"id" : "36",
"name" : "創(chuàng)業(yè)邦會員 — 找人、找錢、找項目！",
"content" : "這其實是內(nèi)容，但是中間有雙引號和單引號，我暫時不拼接了"
}'
{"_index":"cyzone","_type":"goods","_id":"1qtDo2oBACchbOnLTGLF","_version":1,"result":"created","_shards":{"total":3,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}% ? ~ curl -X POST "localhost:9200/cyzone/goods" -H 'Content-Type: application/json' -d'
{
"id" : "143",
"name" : "創(chuàng)業(yè)邦會員 — 優(yōu)惠價格 happy",
"content" : "just for test vip，創(chuàng)業(yè)"
}'
{"_index":"cyzone","_type":"goods","_id":"16tEo2oBACchbOnLZmK4","_version":1,"result":"created","_shards":{"total":3,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}%
可以指定id，也可以不指定id，指定id的時候就是cyzone/goods/具體id號，但是要把POST改為PUT，不指定ID，ES內(nèi)部有自增的id。
3.4測試搜索功能是否好用
簡單搜索
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/search-uri-request.html
? ~ curl -X GET "localhost:9200/cyzone/_search?q=id:143"
{"took":1,"timed_out":false,"_shards":{"total":3,"successful":3,"skipped":0,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"cyzone","_type":"goods","_id":"16tEo2oBACchbOnLZmK4","_score":1.0,"_source":
{
"id" : "143",
"name" : "創(chuàng)業(yè)邦會員 — 優(yōu)惠價格 happy",
"content" : "just for test vip，創(chuàng)業(yè)"
}}]}}%
復(fù)雜的搜索
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/search-request-body.html
? ~ curl -XPOST "http://localhost:9200/cyzone/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query" : { "match" : { "content" : "創(chuàng)業(yè)" }}
}
'
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.2876821,
"hits" : [
{
"_index" : "cyzone",
"_type" : "goods",
"_id" : "16tEo2oBACchbOnLZmK4",
"_score" : 0.2876821,
"_source" : {
"id" : "143",
"name" : "創(chuàng)業(yè)邦會員 — 優(yōu)惠價格 happy",
"content" : "just for test vip，創(chuàng)業(yè)"
}
}
]
}
}
再次嘗試搜索name
? ~ curl -XPOST "http://localhost:9200/cyzone/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query" : { "match" : { "name" : "創(chuàng)業(yè)" }}
}
'
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 0.2876821,
"hits" : [
{
"_index" : "cyzone",
"_type" : "goods",
"_id" : "16tEo2oBACchbOnLZmK4",
"_score" : 0.2876821,
"_source" : {
"id" : "143",
"name" : "創(chuàng)業(yè)邦會員 — 優(yōu)惠價格 happy",
"content" : "just for test vip，創(chuàng)業(yè)"
}
},
{
"_index" : "cyzone",
"_type" : "goods",
"_id" : "1qtDo2oBACchbOnLTGLF",
"_score" : 0.2876821,
"_source" : {
"id" : "36",
"name" : "創(chuàng)業(yè)邦會員 — 找人、找錢、找項目！",
"content" : "這其實是內(nèi)容，但是中間有雙引號和單引號，我暫時不拼接了"
}
}
]
}
}
4.導(dǎo)入數(shù)據(jù)庫已有的數(shù)據(jù)
使用logstash工具同步數(shù)據(jù)庫的數(shù)據(jù)到ES，不僅僅是logstash，但是這個推薦的比較多，官方網(wǎng)站上有這個。
https://www.elastic.co/cn/products/logstash
? Desktop brew install logstash
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
imapsync
==> Updated Formulae
aliyun-cli conan i2pd joplin lxc postgresql@9.6 socat wtf
calicoctl doctl imagemagick juju netpbm pygobject ttyd yarn
cfn-lint glooctl istioctl kotlin oniguruma serverless ucloud youtube-dl
checkbashisms gtk-doc jhipster kubeprod opencoarrays skaffold utf8proc
==> Downloading https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.0.1.tar.gz
######################################################################## 100.0%
==> Caveats
Configuration files are located in /usr/local/etc/logstash/
To have launchd start logstash now and restart at login:
brew services start logstash
Or, if you don't want/need a background service you can just run:
logstash
==> Summary
/usr/local/Cellar/logstash/7.0.1: 12,515 files, 283.9MB, built in 22 minutes 49 seconds
進入logstash 相關(guān)目錄下安裝插件logstash-input-jdbc
cd /usr/local/Cellar/logstash/7.0.1
./logstash-plugin install logstash-input-jdbc
插件使用文檔:https://www.elastic.co/guide/en/logstash/7.0/plugins-inputs-jdbc.html
input {
jdbc {
type=> 'goods'
jdbc_driver_library => "/usr/local/Cellar/logstash/7.0.1/bin/mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://127.0.0.1:3307/topcms_cyzone_cn_test"
jdbc_user => "localhost"
jdbc_password => "123456"
statement => "SELECT id,name,content from shop order by id desc"
tracking_column => "id"
jdbc_paging_enabled=>"true"
jdbc_page_size => "5000"
#tracking_column => "timestamp"
#tracking_column_type=>numeric
schedule => "* * * * *"
}
}
filter {
}
output {
if[type]=="goods"{
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "cyzone"
document_id => "%{id}"
document_type=>"goods"
}
}
stdout {
codec => json_lines
}
}
配置參數(shù)具體以官方文檔為準(zhǔn)，不同的版本有不同的設(shè)置效果，比如document_type這個字段文檔說以后要廢棄，但是我用的是7.0，這個地方還必須用。
document_type參數(shù)在下一個大版本廢棄.png
重復(fù)之前的測試，發(fā)現(xiàn)數(shù)據(jù)確實導(dǎo)入了，這里只導(dǎo)入了goods表的數(shù)據(jù)。如果需要導(dǎo)入其他的數(shù)據(jù)，可以在上面腳本接著擴展，后面還要對參數(shù)具體研究，來實現(xiàn)數(shù)據(jù)庫變化的時候自動同步數(shù)據(jù)到ES。
5.安裝Elasticsearch的PHP庫
https://github.com/elastic/elasticsearch-php
自己根據(jù)實際情況封裝ES管理類。
1.基礎(chǔ)抽象類，提供公共的增刪改成方法。
<?php
namespace app\common\lib\es;
use Elasticsearch\ClientBuilder;
use app\common\es\lib\ESBaseDoc;
/**
* 基本的ES工具類
* Class CyEsTool
* @package app\common\es\goods
*/
abstract class AbstractES
{
/**
* @var ClientBuilder
*/
private $client;
private static $instance;
/**
* 索引名稱相當(dāng)于數(shù)據(jù)庫
* @var string
*/
protected $index = "cyzone";
/**
* 索引類型，相當(dāng)于表
* @var string
*/
protected $type = "";
/**
* 創(chuàng)建索引的時候的mapping信息
* @var array
*/
protected $mappings = [
];
/**
* 默認(rèn)的mappings信息
* @var array
*/
private $defaultMappings = [
'_default_' => [ //默認(rèn)配置，每個類型缺省的配置使用默認(rèn)配置
'_all' => [ // 關(guān)閉所有字段的檢索
'enabled' => 'false'
],
'_source' => [ // 存儲原始文檔
'enabled' => 'true'
],
]
];
/**
* 創(chuàng)建索引的時候的配置信息
* @var array
*/
private $setting = [
"index" => [
"number_of_shards" => 3,
"number_of_replicas" => 2
]
];
private function __construct()
{
$this->client = ClientBuilder::create()
->setHosts(['127.0.0.1:9200'])
->build();
}
public static function getInstance()
{
if (is_null(self::$instance)) {
self::$instance = new static();
}
return self::$instance;
}
/**
* 獲取默認(rèn)的搜索字段，就是mapping里面的配置
* @param array $field
* @param bool $exceptId
* @return array
*/
protected function getSearchFiled($field = [], $exceptId = true)
{
if ($field) {
return $field;
}
$properties = $this->mappings[$this->type]['properties']??[];
if (empty($properties)) {
return [];
}
$fields = array_keys($properties);
foreach ($fields as $key => $value) {
if ($exceptId && strpos($value, "id") !== false) {
unset($fields[$key]);
}
}
return $fields;
}
/**
* 查看Mapping
*/
public function getMappings()
{
$params = [
'index' => $this->index
];
$res = $this->client->indices()->getMapping($params);
return $res;
}
/**
* 修改Mapping
* @return array
*/
public function putMappings()
{
$mappings = array_merge($this->defaultMappings, $this->mappings);
$params = [
'index' => $this->index,
'type' => $this->type,
'body' => [
$mappings
]
];
return $this->client->indices()->putMapping($params);
}
/**
* 插入單條的文檔
* @param ESBaseDoc $baseDoc
* @return array
*/
public function insertOneDoc(ESBaseDoc $baseDoc)
{
//可以對param適當(dāng)做些檢查
$params = [
'index' => $this->index,
'type' => $this->type,
'body' => [
$baseDoc->toArray()
]
];
return $this->client->index($params);
}
/**
* @param ESBaseDoc[] $docArray
*/
public function postBulkDoc(array $docArray)
{
if (count($docArray) == 0) {
return;
}
$params = [];
for ($i = 0; $i < count($docArray); $i++) {
$params['body'][] = [
'index' => [
'_index' => $this->index,
'_type' => $this->type,
]
];
$params['body'][] = [
$docArray[$i]->toArray()
];
}
$this->client->bulk($params);
}
/**
* 根據(jù)id獲得doc
* @param $id
* @return array|bool
*/
public function getDocById($id)
{
$params = [
'index' => $this->index,
'type' => $this->type,
'id' => $id
];
try {
return $this->client->get($params);
} catch (\Exception $exception) {
return false;
}
}
/**
* 根據(jù)id更新文檔的內(nèi)容
* @param $id
* @param ESBaseDoc $baseDoc
* @return array|bool
*/
public function updateDocById($id, ESBaseDoc $baseDoc)
{
$params = [
'index' => $this->index,
'type' => $this->type,
'id' => $id,
'body' => [
'doc' => [
$baseDoc->toArray()
]
]
];
try {
return $this->client->update($params);
} catch (\Exception $exception) {
return false;
}
}
/**
* 根據(jù)id刪除文檔的內(nèi)容
* @param $id
* @return array |bool
*/
public function deleteDocById($id)
{
$params = [
'index' => $this->index,
'type' => $this->type,
'id' => $id
];
try {
return $this->client->delete($params);
} catch (\Exception $exception) {
return false;
}
}
//Query的參數(shù) https://www.elastic.co/guide/en/elasticsearch/reference/6.7/query-filter-context.html
//https://es.xiaoleilu.com/054_Query_DSL/70_Important_clauses.html
/**
* 多個字段查詢搜索,默認(rèn)搜索可以用這個
* @param $keyWords
* @param array $field
* @return array
*/
public function search($keyWords, $field = [])
{
$params = [
'index' => $this->index,
'type' => $this->type,
'body' => [
'query' => [
'multi_match' => [
'query' => $keyWords,
"fields" => $this->getSearchFiled($field)
]
]
]
];
return $this->client->search($params);
}
}
- 具體業(yè)務(wù)類
商品相關(guān)的ES類
<?php
/**
* Created by PhpStorm.
* User: guodong
* Date: 2019/5/12
* Time: 下午5:04
*/
namespace app\common\es\lib;
class ESBaseDoc
{
private $initParams;
public function __construct(array $param)
{
foreach ($param as $key => $value){
$reflect = new \ReflectionProperty(static::class,$key);
if ($reflect->isPublic()){
if (property_exists($this,$key)){
$this->$key = $value;
}
}
}
$this->initParams = $param;
}
public function toArray()
{
return $this->initParams;
}
}
- 具體的數(shù)據(jù)對應(yīng)document對象。
<?php
/**
* Created by PhpStorm.
* User: guodong
* Date: 2019/5/12
* Time: 下午5:04
*/
namespace app\common\es\lib;
class ESBaseDoc
{
private $initParams;
public function __construct(array $param)
{
foreach ($param as $key => $value){
$reflect = new \ReflectionProperty(static::class,$key);
if ($reflect->isPublic()){
if (property_exists($this,$key)){
$this->$key = $value;
}
}
}
$this->initParams = $param;
}
public function toArray()
{
return $this->initParams;
}
}
具體的業(yè)務(wù)類要繼承這個類
namespace app\common\lib\es\goods;
use app\common\es\lib\ESBaseDoc;
class ESGoodsDoc extends ESBaseDoc
{
public $id;
public $name;
public $content;
}
參考文檔
https://www.cnblogs.com/ajianbeyourself/p/5529575.html
https://help.aliyun.com/document_detail/58107.html?spm=a2c4g.11186623.6.543.778473bf6G1rrB
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/index.html
https://es.xiaoleilu.com/052_Mapping_Analysis/45_Mapping.html
