使用elastalert进行告警

作者 uunnfly 日期 2019-03-24
ELK
使用elastalert进行告警

安装

git clone https://github.com/Yelp/elastalert.git
安装模块

1
2
$ pip install "setuptools>=11.3"
$ python setup.py install

安装elasticsearch-py,需要根据你的elasticsearch版本,比如我现在用是6.x
`pip install “elasticsearch>=6.0.0”``

配置

将 config.yaml.example 复制一份命名为 config.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
rules_folder: example_rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
seconds: 3 #每三秒向es请求数据
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
#日志会延迟进入es,这里是配置query的向前的时间范围,这是15分钟,即查询 time[now-15m, now]

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 188.88.88.88

# The Elasticsearch port
es_port: 9200

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 1

以下是各字段的解释:
Rules_folder:用来加载下一阶段rule的设置,默认是example_rules
Run_every:用来设置定时向elasticsearch发送请求
Buffer_time:用来设置请求里时间字段的范围,默认是45分钟
Es_host:elasticsearch的host地址
Es_port:elasticsearch 对应的端口号
Use_ssl:可选的,选择是否用SSL连接es,true或者false
Verify_certs:可选的,是否验证TLS证书,设置为true或者false,默认为- true
Es_username:es认证的username
Es_password:es认证的password
Es_url_prefix:可选的,es的url前缀(我的理解是https或者http)
Es_send_get_body_as:可选的,查询es的方式,默认的是GET
Writeback_index:elastalert产生的日志在elasticsearch中的创建的索引
Alert_time_limit:失败重试的时间限制

规则配置

example_rules 目录下每份文件都是一个规则,已经给我们提供了不少示例写法。现在打开example_frequency.yaml,这里有一修改过的示范配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
es_host: localhost

# (Optional)
# Elasticsearch port
es_port: 9200

# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
name: nginxerror

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency

# (Required)
# Index to search, wildcard supported
index: filebeat-nginxaccess-*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
#hours: 4
minutes: 1

# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- range:
response:
from: 400
to: 599

这是一份根据频率进行报警的配置,1分钟内有发生一次就会报警

使用邮箱进行报警

elastalert内置了十多种通知类型,这里介绍邮箱告警。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# (Required)
# The alert is use when a match is found
alert:
- "email"

# (required, email specific)
# a list of email addresses to send alerts to
email:
- "test@qq.com"

smtp_host: smtp.exmail.qq.com
smtp_prot: 465
smtp_ssl: true
smtp_auth_file: ./smtp_auth_file.yaml
from_addr: test@myemail.com

在这里你可以配置发件人与收件人。

smtp_auth_file存放账户名和密码

1
2
user: xxx@qq.com
password: password

运行

python -m elastalert.elastalert --verbose --rule example_rules/example_frequency.yaml
将会打出日志

使用短信告警

这里选择使用elastalert的command调用发短信的脚本来进行短信报警
在配置文件中:

1
2
3
4
alert:
- command

command: ["/root/dysms_python/sms_send.py","--host","%(host)s","--time","%(time)s","--path","%(source)s"]

/root/dysms_python/sms_send.py 是该脚本名称,后面是传入的参数

增强功能

现在想做到这一点:在nginx日志中发现500之后去查另一个type(表)中与之对应的traceback附在邮件中。你可以在enhancements中改变告警内容

1
2
3
4
$ mkdir elastalert_modules
$ cd elastalert_modules
$ touch __init__.py
$ vim my_enhancements.py

官方示例:

1
2
3
4
5
6
7
8
9
10
11
from elastalert.enhancements import BaseEnhancement

class MyEnhancement(BaseEnhancement):

# The enhancement is run against every match
# The match is passed to the process function where it can be modified in any way
# ElastAlert will do this for each enhancement linked to a rule
def process(self, match):
if 'domain' in match:
url = "http://who.is/whois/%s" % (match['domain'])
match['domain_whois_link'] = url

然后在你的规则配置文件中

1
2
match_enhancements:
- "elastalert_modules.my_enhancements.MyEnhancement"

此时就会多一个domain_whois_link的字段,并且该值是http://who.is/whois/+domain_whois_link

es的精确值匹配

使用term可以进行精确值匹配,比如

1
"term" : { "user" : "Kimchy" }

会查找到user为Kimchy的记录
但是你如果这样查
“term” : { “user” : “Kimchy Hello” }
你会找不到Kimchy Hello的记录!(已经存放了该条记录)
这是因为lucene的底层是倒排索引,会拆成Kimchy与Hello两个关键字,所以使用term是无法精确匹配到的。你必须删除索引并重新建立时将该字段标识为“not_analyzed”
具体可以看这里

Running ElastAlert for the First Time
ElastAlert监控日志告警Web攻击行为
查找准确值