ElasticSearch入门学习

ElasticSearch

分布式全文搜索引擎

6.X和7.X区别特别大

1、入门

大数据需要解决的两个问题：存储、计算

Google和Hadoop技术对比

Google	Hadoop
GFS	HDFS
MapReduce	MapReduce
BidTable	Hbase

回归主题

Lucene是一套信息检索工具包，是jar包

不包含搜索引擎系统！

包含以下功能：

索引结构读写索引的工具排序搜索规则

Lucene和ES的关系

ES是基于Lucene的，在Lucene上做了一些封装和增强

1.1、ES概述

Elasticsearch 是一个分布式、高扩展、高实时的搜索与数据分析引擎。它能很方便的使大量数据具有搜索、分析和探索的能力。充分利用Elasticsearch的水平伸缩性，能使数据在生产环境变得更有价值。Elasticsearch 的实现原理主要分为以下几个步骤，首先用户将数据提交到Elasticsearch 数据库中，再通过分词控制器去将对应的语句分词，将其权重和分词结果一并存入数据，当用户搜索数据时候，再根据权重将结果排名，打分，再将返回结果呈现给用户。

Elasticsearch是与名为Logstash的数据收集和日志解析引擎以及名为Kibana的分析和可视化平台一起开发的。这三个产品被设计成一个集成解决方案，称为“Elastic Stack”（以前称为“ELK stack”）。

Elasticsearch、Logstash、Kibana

1.2、ES和Solr对比及选型

功能：全文搜索、结构化搜索、分析

ES和Solr对比

单纯对已有的数据，Solr的速度快简历索引时，Solr会产生I/O阻塞数据量增加，Solr效率变低Solr使用Zookeeper进行分布式管理，ES自带分布式协调管理工具Solr支持JSON、XML、CSV，ES只支持JSONSolr比较成熟，ES相对开发维护着较少，更新快，学习使用成本高 1.3、ES安装和head插件安装

官网下载：https://www.elastic.co/start

目录结构

bin				# 启动文件
config			# 配置文件
	-log4j2.properties
	-jvm.options
	-elasticsearch.yml
	默认9200端口
jdk				# 环境
lib				# 相关jar包
logs			# 日志
modules			# 功能模块
plugins			# 插件

修改jvm.options文件的内存参数

-Xms256m
-Xmx256m

启动elasticsearch.bat文件，默认访问9200端口，通信端口9300

访问127.0.0.1:9200得到json字符串

{
  "name" : "DESKTOP-HQU412E",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ct067Y-dRqejNoOwIvqDog",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

ES8.x

问题

配置文件配置内存参数换地方了

需要在jvm.options.d中新建一个xxx.options来配置内存参数需要使用https进行访问，且第一次启动时，自动生成elastic用户的密码 elastic-head插件

环境需求：需要npm、node.js和python2

初始化并启动elastic-head

cd elasticsearch-head-master
npm install
npm run start

启动后访问9100端口，要连接elasticsearch，必须解决跨域问题(跨端口、跨IP、跨网站)

配置跨域

配置elasticsearch.yml

http.cors.enabled: true
http.cors.allow-origin: "*"

重启elasticsearch.bat，9100上连接

将elastic-head当作可视化工具，不要用它来查询，后续使用Kibana来做

1.4、Kibana安装

了解ELK

Elasticsearch、Logstash、Kibana

收集清洗数据 => 分析 => 数据展示

一般提到ELK，就是日志分析架构技术栈总称

下载

官网下载压缩包后解压

注意：Elasticsearch和Kibana必须一致！

启动

点击binkibana.bat启动服务

默认端口5601

选择测试工具

使用Kibana测试工具

汉化Kibana

configkibana.yml下配置国际化，然后重启服务器

#i18n.locale: "en"
i18n.locale: "zh-CN"

1.5、ES核心概念

Elasticsearch面向文档，关系型数据库和ES可以进行客观地对比

RDB	Elasticsearch
数据库(database)	索引(indices)
表(tables)	types
行(rows)	documents
字段(columns)	fields

Elasticsearch中一切都是JSON

索引 > 类型 > 文档

Elasticsearch集群分布

Elasticsearch-head中新建索引默认分片是5

分片即每个碎片分布在不同的集群中

倒排索引

Lucene底层采用的就是倒排索引，这种结构适用于快速的全文搜索

trem	doc_1	doc_2
to	√	×
forever	√	√
total	2	1

例如博客文章

博客文章(原始数据)		索引列表(倒排索引)
博客文章ID	标签	标签	博客文章ID
1	python	python	1，2，3
2	python	linux	3，4
3	linux，python
4	linux

一个Elasticsearch索引是多个Lucene索引组成的

1.6、IK分词器

什么是IK分词器？

即把一段中文划分成一个个的关键字，IK分词器是一个插件

分词算法

IK提供了两个分词算法

ik_smart：最少切分ik_max_word：最细粒度切分

GitHub下载地址：https://github.com/medcl/elasticsearch-analysis-ik

使用步骤

什么版本的ES就下载什么版本的ik

下载的压缩包有两种类型，一种未打包的源代码，一种打包好的

以下情况为未打包的源代码

下载后解压，并执行maven命令打包 mvn clean package打包好后进入目录targetreleases下，解压里面的压缩包到ES的plugins文件夹下

重启ES

可以使用命令行确认是否载入插件

E:environmentELKelasticsearch-7.6.2bin>elasticsearch-plugin list
future versions of Elasticsearch will require Java 11; your Java version from [E:environmentjavaJDKjre] does not meet this requirement
ik

测试使用

配置自定义扩展字典

ik/config/目录下新建自己的字典文件

hu.dic

狂神说

IKAnalyzer.cfg.xml




	IK Analyzer 扩展配置
	
	hu.dic

重启ES测试

1.7、Rest风格操作索引

关于索引的基础操作

使用Kibana创建索引

PUT /索引名/类型名/文档id
{
	请求体
}

例如

PUT /test1/type1/1
{
  "name": "hu",
  "age": 18
}

1.7.1、创建索引

1.7.2、查看索引

数据类型

字符串

text：可以被分词keyword：不可分词数值

byteshortintegerlongfloathalf floatscaled float 日期

date te布尔值

boolean 二级制

binary …

指定字段的类型

创建索引并设置规则

PUT /test2
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "age": {
        "type": "long"
      },
      "birthday":{
        "type": "date"
      }
    }
  }
}

执行

获得规则

GET test2

其他命令

GET _cat/health			# 获取ES健康状态
GET _cat/indices?v		# 查看索引信息

1.7.3、修改索引

# 以前的方法
PUT /test1/type1/1
{
  "name": "hu123",
  "age": 18123
}

# 现在的方法
POST /test1/type1/1/_update
{
  "doc": {
  	"name": "hu123",
  }
}

修改索引后，版本version会增加，result变为update

1.7.4、删除索引

DELETE test2/_doc/1

{
  "acknowledged" : true
}

注意

若不写文档类型，则必须使用POSTrestful风格不允许url为驼峰 1.8、回顾上节

添加数据

PUT /user_list/user/1
{
  "name": "hu",
  "age": 18,
  "desc": "一顿操作猛如虎",
  "tags": ["技术宅","暖"]
}
PUT /user_list/user/2
{
  "name": "张三",
  "age": 23,
  "desc": "法外狂徒",
  "tags": ["打人","狠"]
}
PUT /user_list/user/3
{
  "name": "李四",
  "age": 19,
  "desc": "无",
  "tags": ["唱","跳","rap"]
}

查询数据

GET user_list/user/3						# 简单查询
GET user_list/user/_search?q=name:hu		# 条件查询

修改数据

POST /user_list/user/3/_update
{
  "doc": {
	  "name": "李四233",
  }
}

1.9、花式查询

文档复杂查询——构建查询方式

1、模糊查询文档匹配所有的数据

GET user_list/user/_search
{
  "query": {
    "match": {		# match匹配条件
      "name": "李"
    }
  }
}

注意：中文可以分词，模糊检索，拼音不会分词

hits：对应Java中的对象Hits
score：权重
source：数据

2、模糊查找documents的部分fields

GET user_list/user/_search
{
  "query": {
    "match": {
      "name": "李"
    }
  },
  "_source": [
    "name",
    "desc"
  ]
}

3、排序

"sort": [
    {
        "age": {
            "order": "desc"
        }
    }
]

4、分页

从第0条数据开始，一页显示2条数据

GET user_list/user/_search
{
  "query": {
    "match": {
      "name": "李"
    }
  },
  "_source": [
    "name",
    "desc"
  ],
  "sort": [
    {
      "age": {
        "order": "desc"
      }
    }
  ],
  "from": 0,
  "size": 2
}

5、布尔查询

多条件精确查询

GET user_list/user/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "name": "hu"
          }
        },
        {
          "match": {
            "age": "18"
          }
        }
      ]
    }
  }
}

must：所有的条件都要符合must_not：所有条件都不符合should：或，满足一个即可

6、过滤器

GET user_list/user/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "name": "hu"
          }
        }
      ],
      "filter": {		# 过滤
        "range": {
          "age": {		# field
            "gte": 3,	# 大于等于
            "lte": 18	# 小于等于
          }
        }
      }
    }
  }
}

7、多条件匹配

模糊查询

满足其中一个条件即可被查询出

GET user_list/user/_search
{
  "query": {
    "match": {
      "tags": "唱 rap 跳"
    }
  }
}

精确查询

term使用倒排索引精确查询

关于分词

term直接精确查询match会使用分词器解析

GET user_list/user/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "age": {
              "value": "18"
            }
          }
        },
        {
          "term": {
            "age": {
              "value": "23"
            }
          }
        }
      ]
    }
  }
}

8、高亮查询

GET user_list/user/_search
{
  "query": {
    "match": {
      "name": "李"
    }
  },
  "highlight": {
    "fields": {
      "name": {}
    }
  }
}

自动增加html标签高亮显示

自定义标签样式

GET user_list/user/_search
{
  "query": {
    "match": {
      "name": "李"
    }
  },
  "highlight": {
    "pre_tags": "",
    "post_tags": "", 
    "fields": {
      "name": {}
    }
  }
}

2、进阶 2.1、SpringBoot集成ES

查看官方文档：https://www.elastic.co/guide/index.html找到客户端Clients链接

推荐使用Java REST Client选择高级客户端(新版本的全都只有高级客户端)

原生maven依赖


    org.elasticsearch.client
    elasticsearch-rest-high-level-client
    7.6.2

SpringBoot依赖


   org.springframework.boot
   spring-boot-starter-data-elasticsearch

初始化

RestHighLevelClient client = new RestHighLevelClient(
        RestClient.builder(
                new HttpHost("localhost", 9200, "http"),
                new HttpHost("localhost", 9201, "http")));
client.close();

配置基本的项目

新建SpringBoot项目，并添加ES依赖

一定要保证SpringBoot下的依赖和ES版本一致，这边使用的是7.6.2

修改默认版本


   1.8
   7.6.2

创建ES配置类，注入bean

package com.kuangshen.elaticsearch.config;

import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ElasticSearchClientConfig {

    @Bean
    public RestHighLevelClient restHighLevelClient(){
        RestHighLevelClient client = new RestHighLevelClient(
                RestClient.builder(
                        new HttpHost("localhost", 9200, "http")
                        ));
        return client;
    }
}

2.2、索引API操作

创建空索引

@Test
void createIndexTest() throws IOException {
   // 创建索引请求
   CreateIndexRequest request = new CreateIndexRequest("text_index");
   // 获得请求响应体
   CreateIndexResponse response = restHighLevelClient.indices().create(request, RequestOptions.DEFAULT);
   System.out.println(response);
}

判断索引是否存在

@Test
void existsIndexTest() throws IOException {
   GetIndexRequest request = new GetIndexRequest("text_index");
   boolean exists = restHighLevelClient.indices().exists(request, RequestOptions.DEFAULT);
   System.out.println(exists);
}

删除索引

@Test
void deleteIndexTest() throws IOException {
   DeleteIndexRequest request = new DeleteIndexRequest("text_index");
   AcknowledgedResponse response = restHighLevelClient.indices().delete(request, RequestOptions.DEFAULT);
   System.out.println(response.isAcknowledged());
}

2.3、文档API操作

创建实体类

package com.kuangshen.elaticsearch.dto;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.springframework.stereotype.Component;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Component
public class User {
    private String name;
    private int age;
}

创建文档

@Test
void createdocumentTest() throws IOException {
   User user = new User("xiaoming", 3);
   // 请求索引
   IndexRequest request = new IndexRequest("text_index");
   // 文档id
   request.id("1");
   request.timeout(Timevalue.timevalueSeconds(1));
   // 文档内容
   request.source(JSON.toJSONString(user), XContentType.JSON);
   // 客户端发送请求
   IndexResponse indexResponse = restHighLevelClient.index(request, RequestOptions.DEFAULT);
   System.out.println(indexResponse.status());
   System.out.println(indexResponse.toString());
}

判断文档是否存在

@Test
void existsdocumentTest() throws IOException {
   GetRequest request = new GetRequest("text_index","1");
   // 不获取_source上下文，判断效率更高
   request.fetchSourceContext(new FetchSourceContext(false));
   // 设置字段
   request.storedFields("_none_");
   boolean exists = restHighLevelClient.exists(request, RequestOptions.DEFAULT);
   System.out.println(exists);
}

获取文档的信息

@Test
void getdocumentTest() throws IOException {
    GetRequest request = new GetRequest("text_index", "1");
    restHighLevelClient.exists(request, RequestOptions.DEFAULT);
    GetResponse response = restHighLevelClient.get(request, RequestOptions.DEFAULT);
    System.out.println(response.getSourceAsString());
    System.out.println(response);
}

结果

{"age":3,"name":"xiaoming"}
{"_index":"text_index","_type":"_doc","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"age":3,"name":"xiaoming"}}

修改文档信息

   @Test
   void updatedocumentTest() throws IOException{
   UpdateRequest request = new UpdateRequest("text_index", "1");
   request.timeout("1s");
   request.doc(JSON.toJSONString(new User("xiaohong",3)),XContentType.JSON);
   UpdateResponse response = restHighLevelClient.update(request, RequestOptions.DEFAULT);
   System.out.println(response.toString());
}

删除文档

@Test
void deletedocumentTest() throws IOException {
    DeleteRequest request = new DeleteRequest("test_index", "1");
    request.timeout("1s");
    DeleteResponse response = restHighLevelClient.delete(request, RequestOptions.DEFAULT);
    System.out.println(response.status());
}

插入多条数据

@Test
void batchInsertdocumentTest() throws IOException {
    BulkRequest request = new BulkRequest();
    request.timeout(ElasticSearchConstants.TIME_OUT);

    ArrayList userList = new ArrayList<>();
    userList.add(new User("小红", 3));
    userList.add(new User("小明", 35));
    userList.add(new User("小刚", 23));
    userList.add(new User("小芳", 18));
    userList.add(new User("小键", 50));

    userList.forEach((user) -> {
        request.add(new IndexRequest(ElasticSearchConstants.ES_INDEX).source(JSON.toJSONString(user), XContentType.JSON));
    });

    BulkResponse responses = restHighLevelClient.bulk(request, RequestOptions.DEFAULT);
    System.out.println(!responses.hasFailures() ? responses.toString() : null);
}

复杂查询

@Test
void searchdocumentTest() throws IOException {
    SearchRequest request = new SearchRequest(ElasticSearchConstants.ES_INDEX);
    SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();

    sourceBuilder.timeout(new Timevalue(1, TimeUnit.SECONDS));
    sourceBuilder.query(QueryBuilders.termQuery("name", "小红"));
    

    request.source(sourceBuilder);
    SearchResponse response = restHighLevelClient.search(request, RequestOptions.DEFAULT);

    JSON.toJSONString(response.getHits());
    System.out.println("===============循环遍历===============");
    for (SearchHit hit : response.getHits().getHits()) {
        System.out.println(hit.getSourceAsMap());
    }
}

3、实战

京东搜索

项目搭建

新建一个SpringBoot项目

导入依赖



    4.0.0
    
        org.springframework.boot
        spring-boot-starter-parent
        2.6.4
         
    
    com.kuangshen
    elasticsearch
    0.0.1-SNAPSHOT
    elasticsearch
    仿京东搜索
    
        1.8
        7.6.2
    
    
        
            org.springframework.boot
            spring-boot-starter-data-elasticsearch
        
        
            org.springframework.boot
            spring-boot-starter-thymeleaf
        
        
            org.springframework.boot
            spring-boot-starter-web
        

        
            org.projectlombok
            lombok
            true
        
        
            org.springframework.boot
            spring-boot-starter-test
            test
        
        
            com.alibaba
            fastjson
            1.2.62
        
    

    
        
            
                org.springframework.boot
                spring-boot-maven-plugin
                
                    
                        
                            org.projectlombok
                            lombok

配置文件

server.port=9090
spring.thymeleaf.cache=false

测试项目启动

爬取数据

数据哪里来？

数据库获取消息队列中获取爬虫

爬取数据：获取请求返回的页面信息，筛选出我们想要的数据

导入jsoup，可以解析网页，不能解析视频，tiki包可以


    org.jsoup
    jsoup
    1.10.2

创建工具类测试

public static void main(String[] args) throws IOException {
    String url = "https://search.jd.com/Search?keyword=java";
    // 返回js页面对象，可以调用js的所有方法
    document document = Jsoup.parse(new URL(url), 30000);
    document.getElementById("J_goodsList")
            .getElementsByTag("li")
            .forEach((element) -> {
                // 关于图片多的网站，所有的图片都是延迟加载的
                String image = element.getElementsByTag("img").eq(0).attr("data-lazy-img");
                String price = element.getElementsByClass("p-price").eq(0).text();
                String title = element.getElementsByClass("p-name").eq(0).text();
                System.out.println(image + "t" + price + "t" + title);
                System.out.println("========================================");
            });
}

爬取成功，进行项目准备工作

配置类

package com.kuangshen.elasticsearch.config;

import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ElasticSearchConfig {
    @Bean
    public RestHighLevelClient restHighLevelClient(){
        RestHighLevelClient client = new RestHighLevelClient(
                RestClient.builder(
                        new HttpHost("localhost", 9200, "http")
                ));
        return client;
    }
}

创建实体类

package com.kuangshen.elasticsearch.dto;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.springframework.stereotype.Component;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Component
public class Content {
    private String title;
    private String price;
    private String img;
}

创建工具类方法

package com.kuangshen.elasticsearch.utils;

import com.kuangshen.elasticsearch.dto.Content;
import org.jsoup.Jsoup;
import org.jsoup.nodes.document;
import org.springframework.stereotype.Component;

import java.io.IOException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;

@Component
public class HtmlParseUtil {
    public List parseJD(String keyword) throws IOException {
        String url = "https://search.jd.com/Search?keyword=" + keyword;
        document document = Jsoup.parse(new URL(url), 30000);
        ArrayList goodList = new ArrayList<>();
        document.getElementById("J_goodsList")
                .getElementsByTag("li")
                .forEach((element) -> {
                    // 关于图片多的网站，所有的图片都是延迟加载的
                    String image = element.getElementsByTag("img").eq(0).attr("data-lazy-img");
                    String price = element.getElementsByClass("p-price").eq(0).text();
                    String title = element.getElementsByClass("p-name").eq(0).text();
                    goodList.add(new Content(title,price,image));
                });
        return goodList;
    }
}

业务编写

Service层

package com.kuangshen.elasticsearch.service;

import com.alibaba.fastjson.JSON;
import com.kuangshen.elasticsearch.dto.Content;
import com.kuangshen.elasticsearch.utils.HtmlParseUtil;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.Timevalue;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Service
public class ContentServiceImpl {
    @Autowired
    private HtmlParseUtil htmlParseUtil;
    @Autowired
    private RestHighLevelClient restHighLevelClient;

    public boolean parseContent(String keyword) throws IOException {
        List contents = htmlParseUtil.parsejd(keyword);
        BulkRequest request = new BulkRequest();
        request.timeout("2m");
        contents.forEach((content) -> {
            request.add(new IndexRequest("jd_good").source(JSON.toJSONString(content), XContentType.JSON));
        });
        BulkResponse response = restHighLevelClient.bulk(request, RequestOptions.DEFAULT);
        return !response.isFragment();
    }

    public List> searchPage(String keyword, int page, int size) throws IOException {
        if (page < 1){
            page = 1;
        }

        SearchRequest request = new SearchRequest("jd_good");

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        sourceBuilder.timeout(new Timevalue(60, TimeUnit.SECONDS));
        sourceBuilder.query(QueryBuilders.termQuery("title",keyword));
        sourceBuilder.from(page);
        sourceBuilder.size(size);

        request.source(sourceBuilder);
        SearchResponse response = restHighLevelClient.search(request, RequestOptions.DEFAULT);

        ArrayList> list = new ArrayList<>();
        for (SearchHit hit : response.getHits().getHits()) {
            list.add(hit.getSourceAsMap());
        }
        return list;
    }
}

Controller层

package com.kuangshen.elasticsearch.controller;

import com.kuangshen.elasticsearch.service.ContentServiceImpl;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

import java.io.IOException;
import java.util.List;
import java.util.Map;

@RestController
public class ContentController {
    @Autowired
    ContentServiceImpl contentService;

    @GetMapping("/parse/{keyword}")
    public boolean parse(@PathVariable String keyword) throws IOException {
        return contentService.parseContent(keyword);
    }

    @GetMapping("/search/{keyword}/{page}/{size}")
    public List> searchPage(@PathVariable String keyword,
                                                @PathVariable int page,
                                                @PathVariable int size) throws IOException {
        return contentService.searchPage(keyword, page, size);
    }
}

前后端交互

本地下载vue.js和axios.js并导入项目

npm init
npm install vue
npm install axios

导入html文件、css、images等静态文件





    
    狂神说Java-ES仿京东实战
    




    

        
        
            
                
                    
                    
                        
                    

                    

                        
                        
                            
                                
                                    天猫搜索
                                    
                                        
                                            
                                                
                                            
                                        
                                        
                                    
                                
                            
                            
                                狂神说Java
                                狂神说前端
                                狂神说Linux
                                狂神说大数据
                                狂神聊理财
                            
                        
                    
                
            
        

        
        
            
                
                
                    
                        
                            
                                
                                    品牌
                                
                                
                                    
                                         狂神说 
                                         Java 
                                    
                                
                            
                        
                    
                

                
                
                    综合
                    人气
                    新品
                    销量
                    价格
                

                
                

                    
                        
                            
                            
                                
                                    
                                
                            
                            
                            
                                {{result.price}}
                            
                            
                            
                                
                                {result.title}}-->
                            
                            
                            
                                店铺： 狂神说Java 
                            
                            
                            
                                月成交999笔
                                评价 3

关键字高亮

public List> searchPageHighlight(String keyword, int page, int size) throws IOException {
    if (page < 1) {
        page = 1;
    }

    SearchRequest request = new SearchRequest("jd_good");

    SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
    sourceBuilder.timeout(new Timevalue(60, TimeUnit.SECONDS));
    sourceBuilder.query(QueryBuilders.termQuery("title", keyword));
    sourceBuilder.from(page);
    sourceBuilder.size(size);

    // 高亮
    HighlightBuilder highlightBuilder = new HighlightBuilder();
    highlightBuilder.field("title");
    // highlightBuilder.requireFieldMatch(false);
    highlightBuilder.preTags("");
    highlightBuilder.postTags("");
    sourceBuilder.highlighter(highlightBuilder);

    request.source(sourceBuilder);
    SearchResponse response = restHighLevelClient.search(request, RequestOptions.DEFAULT);

    ArrayList> list = new ArrayList<>();
    for (SearchHit hit : response.getHits().getHits()) {
        // 解析高亮字段
        Map highlightFields = hit.getHighlightFields();
        HighlightField title = highlightFields.get("title");
        Map map = hit.getSourceAsMap();
        if (title != null) {
            StringBuilder highlightTitle = new StringBuilder();
            for (Text fragment : title.getFragments()) {
                highlightTitle.append(fragment);
            }
            map.put("title", highlightTitle.toString());
        }
        list.add(map);
    }
    return list;
}

总结

Elasticsearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java语言开发的，并作为Apache许可条款下的开放源码发布，是一种流行的企业级搜索引擎。

Elasticsearch只支持JSON格式，可以搭分布式集群，大数据下高性能，基于Lucene的倒排索引，查询效率很高

Service层优化

package com.kuangshen.elasticsearch.service;

import com.alibaba.fastjson.JSON;
import com.kuangshen.elasticsearch.dto.Content;
import com.kuangshen.elasticsearch.utils.HtmlParseUtil;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.text.Text;
import org.elasticsearch.common.unit.Timevalue;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Service
public class ContentServiceImpl {
    @Autowired
    private HtmlParseUtil htmlParseUtil;
    @Autowired
    private RestHighLevelClient restHighLevelClient;

    public boolean parseContent(String keyword) throws IOException {
        List contents = htmlParseUtil.parsejd(keyword);
        BulkRequest request = new BulkRequest();
        request.timeout("2m");
        contents.forEach((content) -> {
            request.add(new IndexRequest("jd_good").source(JSON.toJSONString(content), XContentType.JSON));
        });
        BulkResponse response = restHighLevelClient.bulk(request, RequestOptions.DEFAULT);
        return !response.isFragment();
    }

    public List> searchPage(String keyword, int page, int size) throws IOException {
        SearchRequest request = new SearchRequest("jd_good");
        SearchSourceBuilder sourceBuilder = searchRequest(keyword, page, size);
        request.source(sourceBuilder);
        SearchResponse response = restHighLevelClient.search(request, RequestOptions.DEFAULT);
        ArrayList> list = new ArrayList<>();
        for (SearchHit hit : response.getHits().getHits()) {
            list.add(hit.getSourceAsMap());
        }
        return list;
    }

    public List> searchPageHighlight(String keyword, int page, int size) throws IOException {
        SearchRequest request = new SearchRequest("jd_good");

        // 高亮
        HighlightBuilder highlightBuilder = new HighlightBuilder();
        highlightBuilder.field("title");
        highlightBuilder.requireFieldMatch(false);
        highlightBuilder.preTags("");
        highlightBuilder.postTags("");

        SearchSourceBuilder sourceBuilder = searchRequest(keyword, page, size);
        sourceBuilder.highlighter(highlightBuilder);
        request.source(sourceBuilder);

        SearchResponse response = restHighLevelClient.search(request, RequestOptions.DEFAULT);

        ArrayList> list = new ArrayList<>();
        for (SearchHit hit : response.getHits().getHits()) {
            // 解析高亮字段
            Map highlightFields = hit.getHighlightFields();
            HighlightField title = highlightFields.get("title");
            // 不高亮的结果
            Map map = hit.getSourceAsMap();
            if (title != null) {
                // 这边使用StringBuilder不会出现使用+=的字符串串联
                StringBuilder highlightTitle = new StringBuilder();
                for (Text fragment : title.getFragments()) {
                    highlightTitle.append(fragment);
                }
                map.put("title", highlightTitle.toString());
            }
            list.add(map);
        }
        return list;
    }

    private SearchSourceBuilder searchRequest(String keyword, int page, int size){
        if (page < 1) {
            page = 1;
        }

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        // 超时时间
        sourceBuilder.timeout(new Timevalue(60, TimeUnit.SECONDS));
        // 精确查询
        sourceBuilder.query(QueryBuilders.termQuery("title", keyword));
        // 分页
        sourceBuilder.from(page);
        sourceBuilder.size(size);

        return sourceBuilder;
    }
}

本文是观看狂神说Java总结的，有兴趣的可以去B站看看他的视频，全部免费而且非常棒
B站链接：https://www.bilibili.com/video/BV17a4y1x7zq?p=20&spm_id_from=pageDriver

ElasticSearch入门学习

大数据系统相关栏目本月热门文章