ELK6.2.4集群安装使用详解大数据

# ======================== Elasticsearch Configuration ========================= 
# NOTE: Elasticsearch comes with reasonable defaults for most settings. 
#       Before you set out to tweak and tune the configuration, make sure you 
#       understand what are you trying to accomplish and the consequences. 
# 
# The primary way of configuring a node is via this file. This template lists 
# the most important settings you may want to configure for a production cluster. 
# 
# Please consult the documentation for further information on configuration options: 
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html 
# 
# ---------------------------------- Cluster ----------------------------------- 
# 
# Use a descriptive name for your cluster: 
#cluster.name: my-application 
# 
# ------------------------------------ Node ------------------------------------ 
# 
# Use a descriptive name for the node: 
# 
#node.name: node-1 
# 
# Add custom attributes to the node: 
# 
#node.attr.rack: r1 
# 
# ----------------------------------- Paths ------------------------------------ 
# 
# Path to directory where to store the data (separate multiple locations by comma): 
# 
#path.data: /path/to/data 
# 
# Path to log files: 
# 
#path.logs: /path/to/logs 
# 
# ----------------------------------- Memory ----------------------------------- 
# 
# Lock the memory on startup: 
# 
#bootstrap.memory_lock: true 
# 
# Make sure that the heap size is set to about half the memory available 
# on the system and that the owner of the process is allowed to use this 
# limit. 
# 
# Elasticsearch performs poorly when the system is swapping the memory. 
# 
# ---------------------------------- Network ----------------------------------- 
# 
# Set the bind address to a specific IP (IPv4 or IPv6): 
# 
#network.host: 192.168.0.1 
# 
# Set a custom port for HTTP: 
# 
#http.port: 9200 
# 
# For more information, consult the network module documentation. 
# 
# --------------------------------- Discovery ---------------------------------- 
# 
# Pass an initial list of hosts to perform discovery when new node is started: 
# The default list of hosts is ["127.0.0.1", "[::1]"] 
# 
#discovery.zen.ping.unicast.hosts: ["host1", "host2"] 
# 
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): 
# 
#discovery.zen.minimum_master_nodes: 
# 
# For more information, consult the zen discovery module documentation. 
# 
# ---------------------------------- Gateway ----------------------------------- 
# 
# Block initial recovery after a full cluster restart until N nodes are started: 
# 
#gateway.recover_after_nodes: 3 
# 
# For more information, consult the gateway module documentation. 
# 
# ---------------------------------- Various ----------------------------------- 
# 
# Require explicit names when deleting indices: 
# 
#action.destructive_requires_name: true

[[email protected] software]$ tar zxvf elasticsearch-6.2.4.tar.gz -C /opt/module/elk

cluster.name: myescluster         #自定义修改一个集群名称 
node.name: node21                 #节点名称 
network.host: 192.168.100.21      #节点IP（或者解析的主机名） 
bootstrap.memory_lock: true          #设置elasticsearch的进程锁住内存 
discovery.zen.ping.unicast.hosts: ["192.168.100.21", "192.168.100.22","192.168.100.23"]  #集群个节点IP地址 
discovery.zen.minimum_master_nodes: 2      #为了避免脑裂，集群节点数最少为 半数+1

vi elasticsearch/config/jvm.options 
-Xms1g                                                  # JVM最大、最小使用内存-Xmx1g

[[email protected] elk]$ scp -r elasticsearch-6.2.4 [email protected]:/opt/module/elk 
[[email protected] elk]$ scp -r elasticsearch-6.2.4 [email protected]:/opt/module/elk 
修改node22，node23节点配置文件elasticsearch.yml中的主机名和ip

[[email protected] bin]$ ./elasticsearch

nohup./bin/elasticsearch& 
或者 /bin/elasticsearch -d

http://node21:9200/

[[email protected] software]$ xz -d node-v8.11.2-linux-x64.tar.xz  
[[email protected] software]$ tar  xvf node-v8.11.2-linux-x64.tar -C /opt/module/elk

export NODE_HOME=/opt/module/elk/node-v8.11.2-linux-x64 
export PATH=$NODE_HOME/bin:$PATH

[[email protected] elk]$ wget https://github.com/mobz/elasticsearch-head/archive/master.zip

[[email protected] elk]$ unzip master.zip

 [[email protected] elasticsearch-head-master]$ npm config set registry https://registry.npm.taobao.org

[[email protected] elasticsearch-head-master]$ npm install -g grunt 
[[email protected] elasticsearch-head-master]$ npm install

[[email protected] software]$ node -v 
v8.11.2 
[[email protected] software]$ npm -v 
6.1.0 
[[email protected] software]$ grunt -version 
grunt-cli v1.2.0
grunt v1.0.1

[[email protected] elasticsearch-head-master]$ vi Gruntfile.js  
connect: { 
server: { 
options: { 
   hostname: '0.0.0.0', 
port: 9100, 
base: '.', 
keepalive: true 
} 
} 
}

http.cors.enabled: true 
http.cors.allow-origin: "*"

[[email protected] _site]$ vi app.js  
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://localhost:9200"; 
把localhost修改成你es的服务器地址，如: 
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.100.21:9200";

[[email protected] bin]$ ./elasticsearch

[[email protected] elasticsearch-head-master]$ grunt server &

[[email protected] elk]$ wget https://github.com/hlstudio/bigdesk/archive/master.zip
[[email protected] elk]$ unzip master.zip 

[[email protected] elk]$ sudo mv bigdesk-master/ /var/www/html/

[[email protected] elasticsearch-6.2.4]$ ./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.2.4/elasticsearch-analysis-ik-6.2.4.zip

[[email protected] bin]$ curl -XPUT http://192.168.100.21:9200/index

curl -XPOST http://192.168.100.21:9200/index/fulltext/_mapping -H 'Content-Type:application/json' -d' 
{ 
"properties": { 
"content": { 
"type": "text", 
"analyzer": "ik_max_word", 
"search_analyzer": "ik_max_word" 
} 
} 
}'

curl -XPOST http://192.168.100.21:9200/index/fulltext/1 -H 'Content-Type:application/json' -d' 
{"content":"美国留给伊拉克的是个烂摊子吗"}'
 
curl -XPOST http://192.168.100.21:9200/index/fulltext/2 -H 'Content-Type:application/json' -d' 
{"content":"公安部：各地校车将享最高路权"}' 
 
curl -XPOST http://192.168.100.21:9200/index/fulltext/3 -H 'Content-Type:application/json' -d' 
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}' 
 
curl -XPOST http://192.168.100.21:9200/index/fulltext/4 -H 'Content-Type:application/json' -d' 
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}'

curl -XPOST http://192.168.100.21:9200/index/fulltext/_search  -H 'Content-Type:application/json' -d' 
{ 
"query" : { "match" : { "content" : "中国" }}, 
"highlight" : { 
"pre_tags" : ["<tag1>", "<tag2>"], 
"post_tags" : ["</tag1>", "</tag2>"], 
"fields" : { 
"content" : {} 
} 
} 
}'

[[email protected] elk]$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-x86_64.rpm 
[[email protected] elk]$ sudo rpm -vi filebeat-6.2.4-x86_64.rpm

filebeat.prospectors: 
- type: log 
enabled: true 
paths: 
- /var/log/*.log

sudo ./filebeat -e -c filebeat.yml

[[email protected] elk]$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz 
[[email protected] elk]$ tar zxvf logstash-6.2.4.tar.gz 

[[email protected] datas]$ pwd 
/opt/datas 
[[email protected] datas]$ vi logstash-simple.conf

input { stdin { } } 
output { 
elasticsearch { hosts => ["192.168.100.21:9200"] } 
stdout { codec => rubydebug } 
}

[[email protected] bin]$ pwd 
/opt/module/elk/logstash-6.2.4/bin 
[[email protected] bin]$ ./logstash -f /opt/datas/logstash-simple.conf

[[email protected] bin]$ ./logstash-plugin prepare-offline-pack logstash-input-beats 
Offline package created at: /opt/module/elk/logstash-6.2.4/logstash-offline-plugins-6.2.4.zip 
You can install it with this command `bin/logstash-plugin install file:///opt/module/elk/logstash-6.2.4/logstash-offline-plugins-6.2.4.zip` 
[[email protected] bin]$ ./logstash-plugin install file:///opt/module/elk/logstash-6.2.4/logstash-offline-plugins-6.2.4.zip

input { 
beats { 
port => 5044 
} 
} 
output { 
elasticsearch { 
hosts => "192.168.100.21:9200" 
manage_template => false 
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
document_type => "%{[@metadata][type]}" 
} 
}

./bin/logstash -f config/logstash.conf &

[[email protected] elk]$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz
[[email protected] elk]$ tar -xzf kibana-6.2.4-linux-x86_64.tar.gz

[[email protected] elk]$ vi kibana-6.2.4-linux-x86_64/config/kibana.yml  
#server.host: "localhost" 
server.host: "192.168.100.21"   #设置自己机器的IP 
#elasticsearch.url: "http://localhost:9200" 
elasticsearch.url: "http://192.168.100.21:9200"

[[email protected] bin]$ ./kibana &

ERROR: bootstrap checks failed 
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] 
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
max number of threads [3812] for user [admin] is too low, increase to at least [4096]

*  hard  nofile  65536 
*  soft  nofile  65536

vm.max_map_count=262144

*      soft   nproc   4096 
root   soft   nproc   unlimited

*    soft   nproc    4096 
*    hard   nproc    4096

npm WARN [email protected] license should be a valid SPDX license expression 
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.4 (node_modules/fsevents): 
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.4: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"}) 
npm ERR! code ELIFECYCLE 
npm ERR! errno 1 
npm ERR! phantomjs-prebuilt@2.1.16 install: `node install.js` 
npm ERR! Exit status 1 
npm ERR!  
npm ERR! Failed at the phantomjs-prebuilt@2.1.16 install script. 
npm ERR! This is probably not a problem with npm. There is likely additional logging output above. 
npm ERR! A complete log of this run can be found in: 
npm ERR!     /home/admin/.npm/_logs/2018-06-06T11_41_22_784Z-debug.log

{ 
"name": "elasticsearch-head", 
"version": "0.0.0", 
"description": "Front end for an elasticsearch cluster", 
"main": "_site/index.html", 
"directories": { 
"test": "test" 
}, 
"scripts": { 
"start": "grunt server", 
"test": "grunt jasmine", 
"proxy": "node proxy/index.js" 
}, 
"repository": { 
"type": "git", 
"url": "https://github.com/mobz/elasticsearch-head.git" 
}, 
"author": "", 
"license": "Apache-2.0", 
"gitHead": "0c2ac0b5723b493e4454baa7398f386ecb829412", 
"readmeFilename": "README.textile", 
"devDependencies": { 
"grunt": "1.0.1", 
"grunt-contrib-concat": "1.0.1", 
"grunt-contrib-watch": "1.0.0", 
"grunt-contrib-connect": "1.0.2", 
"grunt-contrib-copy": "1.0.0", 
"grunt-contrib-clean": "1.0.0", 
"grunt-contrib-jasmine": "1.0.3", 
"karma": "1.3.0", 
"grunt-karma": "2.0.0", 
"http-proxy": "1.16.x" 
}

[2018-06-06T08:24:28,869][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node21] uncaught exception in thread [main] 
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: property [elasticsearch.version] is missing for plugin [head] 
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4] 
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4] 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4] 
Caused by: java.lang.IllegalArgumentException: property [elasticsearch.version] is missing for plugin [head] 
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:226) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:184) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.bootstrap.Spawner.spawnNativePluginControllers(Spawner.java:75) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4] 
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4] 
... 6 more

http.cors.enabled: true 
http.cors.allow-origin: "*"

[[email protected] bin]$ ./elasticsearch 
[2018-06-07T23:50:12,370][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory 
[2018-06-07T23:50:12,374][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out. 
[2018-06-07T23:50:12,374][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536 
[2018-06-07T23:50:12,374][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:  
# allow user 'admin' mlockall 
admin soft memlock unlimited 
admin hard memlock unlimited 
[2018-06-07T23:50:12,375][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect. 
[2018-06-07T23:50:12,875][INFO ][o.e.n.Node               ] [node21] initializing ... 
[2018-06-07T23:50:13,033][INFO ][o.e.e.NodeEnvironment    ] [node21] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [90.7gb], net total_space [97.6gb],  
types [rootfs][2018-06-07T23:50:13,034][INFO ][o.e.e.NodeEnvironment    ] [node21] heap size [1015.6mb], compressed ordinary object pointers [true] 
[2018-06-07T23:50:13,036][INFO ][o.e.n.Node               ] [node21] node name [node21], node ID [sM6bNfHmT7CknreP2b9RHw] 
[2018-06-07T23:50:13,037][INFO ][o.e.n.Node               ] [node21] version[6.2.4], pid[2164], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/3.10.0-862.el7.x86 
_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_171/25.171-b11][2018-06-07T23:50:13,037][INFO ][o.e.n.Node               ] [node21] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX 
:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.OUkgncgb, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/opt/module/elk/elasticsearch-6.2.4, -Des.path.conf=/opt/module/elk/elasticsearch-6.2.4/config][2018-06-07T23:50:15,004][INFO ][o.e.p.PluginsService     ] [node21] loaded module [aggs-matrix-stats] 
[2018-06-07T23:50:15,004][INFO ][o.e.p.PluginsService     ] [node21] loaded module [analysis-common] 
[2018-06-07T23:50:15,004][INFO ][o.e.p.PluginsService     ] [node21] loaded module [ingest-common] 
[2018-06-07T23:50:15,005][INFO ][o.e.p.PluginsService     ] [node21] loaded module [lang-expression] 
[2018-06-07T23:50:15,005][INFO ][o.e.p.PluginsService     ] [node21] loaded module [lang-mustache] 
[2018-06-07T23:50:15,005][INFO ][o.e.p.PluginsService     ] [node21] loaded module [lang-painless] 
[2018-06-07T23:50:15,005][INFO ][o.e.p.PluginsService     ] [node21] loaded module [mapper-extras] 
[2018-06-07T23:50:15,005][INFO ][o.e.p.PluginsService     ] [node21] loaded module [parent-join] 
[2018-06-07T23:50:15,005][INFO ][o.e.p.PluginsService     ] [node21] loaded module [percolator] 
[2018-06-07T23:50:15,005][INFO ][o.e.p.PluginsService     ] [node21] loaded module [rank-eval] 
[2018-06-07T23:50:15,006][INFO ][o.e.p.PluginsService     ] [node21] loaded module [reindex] 
[2018-06-07T23:50:15,006][INFO ][o.e.p.PluginsService     ] [node21] loaded module [repository-url] 
[2018-06-07T23:50:15,006][INFO ][o.e.p.PluginsService     ] [node21] loaded module [transport-netty4] 
[2018-06-07T23:50:15,006][INFO ][o.e.p.PluginsService     ] [node21] loaded module [tribe] 
[2018-06-07T23:50:15,007][INFO ][o.e.p.PluginsService     ] [node21] no plugins loaded 
[2018-06-07T23:50:22,618][INFO ][o.e.d.DiscoveryModule    ] [node21] using discovery type [zen] 
[2018-06-07T23:50:24,023][INFO ][o.e.n.Node               ] [node21] initialized 
[2018-06-07T23:50:24,024][INFO ][o.e.n.Node               ] [node21] starting ... 
[2018-06-07T23:50:24,561][INFO ][o.e.t.TransportService   ] [node21] publish_address {192.168.100.21:9300}, bound_addresses {192.168.100.21:9300} 
[2018-06-07T23:50:24,609][INFO ][o.e.b.BootstrapChecks    ] [node21] bound or publishing to a non-loopback address, enforcing bootstrap checks 
ERROR: [1] bootstrap checks failed 
[1]: memory locking requested for elasticsearch process but memory is not locked 
[2018-06-07T23:50:24,684][INFO ][o.e.n.Node               ] [node21] stopping ... 
[2018-06-07T23:50:24,811][INFO ][o.e.n.Node               ] [node21] stopped 
[2018-06-07T23:50:24,812][INFO ][o.e.n.Node               ] [node21] closing ... 
[2018-06-07T23:50:24,851][INFO ][o.e.n.Node               ] [node21] closed

vi /etc/sysctl.conf 
vm.swappiness=0 
vi /etc/security/limits.conf  
admin soft memlock unlimited 
admin hard memlock unlimited 
修改完重新登录生效

[[email protected] bin]$ ./elasticsearch 
Killed

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"analyzer [ik_max_word] not found for field [content]"}],"type":"mapper_parsing_exception","reason" 
:"analyzer [ik_max_word] not found for field [content]"},"status":400}

log   [16:11:15.912] [error][timelion] Error:  in cell #1: Elasticsearch index not found: _all: Error:  in cell #1: Elasticsearch index not found: _all