安装logstash-input-jdbc插件,实现数据的插入
1.安装gem
yum install gem -y
2.替换ruby库(两种国内源+淘宝源)
2.1.1替换为国内源
gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/
gem sources -l
2.1.2修改配置文件
cd /opt/inst/logstash722/
vi Gemfile
#修改Gemfile文件
#source https://rubygems.org/
httpsvi Gemfile.lockgems.ruby-china.com/
vi Gemfile.lock
#remote: https://rubygems.org/
https://gems.ruby-china.com/
2.2 同上替换ruby为淘宝源(如果上述替换国内源不成功,可以换此方法)
gem sources --add https://ruby.taobao.org/ --remove https://rubygems.org/
gem sources -l
由于修改ruby镜像库为淘宝库并没有成功,这里就需要修改Gemfile文件里面的数据源:
cd /opt/inst/logstash722/
vi Gemfile
#修改Gemfile文件
#source https://rubygems.org/
source "https://ruby.taobao.org"
#remote: https://rubygems.org/
remote: https://ruby.taobao.org
3.安装logstash-input-jdbc
3.1 到logstash的bin目录下运行下面的命令,查看可用的插件
cd bin
./logstash-plugin list --verbose
3.2 上述操作看到可以使用的logstash-input-jdbc版本是4.3.3,安装JDBC
unzip logstash-input-jdbc-4.3.3.zip
mv logstash-input-jdbc-4.3.3 /opt/inst/log_jdbc433
cd log_jdbc433/
3.3 配置Gemfile文件
vi Gemfile
source 'https://gems.ruby-china.com/'
4.测试是否成功
4.1 提供数据
需要一个mysql驱动包,sql文件,以及conf配置文件
input {
jdbc {
# mysql jdbc connection string to our backup databse
jdbc_connection_string => "jdbc:mysql://169.254.211.100:3306/test"
# the user we wish to excute our statement as
jdbc_user => "root"
jdbc_password => "root"
# the path to our downloaded jdbc driver
jdbc_driver_library => "/opt/inst/logstash722/sql/mysql-connector-java-5.1.40-bin.jar"
# the name of the driver class for mysql
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement_filepath => "/opt/inst/logstash722/sql/logs_20171206.sql"
schedule => "*/1 * * * *"
type => "jdbc"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
elasticsearch {
hosts => "169.254.211.100:9200"
index => "logs_20171206"
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}
4.2 实现logstash把文件写进mysq数据库l中
cd /opt/inst/logstash722/bin
./logstash -f /opt/logfiter/mysql.conf
http://169.254.211.100:5601/app/kibana#/dashboard/1e4f4310-c638-11e9-9ea7-f9ef9537f0f4?_g=(refreshInterval%3A(display%3AOff%2Cpause%3A!f%2Cvalue%3A0)%2Ctime%3A(from%3Anow-1y%2Cmode%3Aquick%2Cto%3Anow))
1566580763148
原创文章,作者:Maggie-Hunter,如若转载,请注明出处:https://blog.ytso.com/186692.html