安装
git clone https://github.com/Yelp/elastalert.git
安装模块1
2$ pip install "setuptools>=11.3"
$ python setup.py install
安装elasticsearch-py,需要根据你的elasticsearch版本,比如我现在用是6.x
`pip install “elasticsearch>=6.0.0”``
配置
将 config.yaml.example 复制一份命名为 config.yaml1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47rules_folder: example_rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
seconds: 3 #每三秒向es请求数据
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
#日志会延迟进入es,这里是配置query的向前的时间范围,这是15分钟,即查询 time[now-15m, now]
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 188.88.88.88
# The Elasticsearch port
es_port: 9200
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to Elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 1
以下是各字段的解释:
Rules_folder:用来加载下一阶段rule的设置,默认是example_rules
Run_every:用来设置定时向elasticsearch发送请求
Buffer_time:用来设置请求里时间字段的范围,默认是45分钟
Es_host:elasticsearch的host地址
Es_port:elasticsearch 对应的端口号
Use_ssl:可选的,选择是否用SSL连接es,true或者false
Verify_certs:可选的,是否验证TLS证书,设置为true或者false,默认为- true
Es_username:es认证的username
Es_password:es认证的password
Es_url_prefix:可选的,es的url前缀(我的理解是https或者http)
Es_send_get_body_as:可选的,查询es的方式,默认的是GET
Writeback_index:elastalert产生的日志在elasticsearch中的创建的索引
Alert_time_limit:失败重试的时间限制
规则配置
example_rules 目录下每份文件都是一个规则,已经给我们提供了不少示例写法。现在打开example_frequency.yaml,这里有一修改过的示范配置1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49# Alert when the rate of events exceeds a threshold
# (Optional)
# Elasticsearch host
es_host: localhost
# (Optional)
# Elasticsearch port
es_port: 9200
# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True
# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
# (Required)
# Rule name, must be unique
name: nginxerror
# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
# (Required)
# Index to search, wildcard supported
index: filebeat-nginxaccess-*
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1
# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
#hours: 4
minutes: 1
# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- range:
response:
from: 400
to: 599
这是一份根据频率进行报警的配置,1分钟内有发生一次就会报警
使用邮箱进行报警
elastalert内置了十多种通知类型,这里介绍邮箱告警。1
2
3
4
5
6
7
8
9
10
11
12
13
14
15# (Required)
# The alert is use when a match is found
alert:
- "email"
# (required, email specific)
# a list of email addresses to send alerts to
email:
- "test@qq.com"
smtp_host: smtp.exmail.qq.com
smtp_prot: 465
smtp_ssl: true
smtp_auth_file: ./smtp_auth_file.yaml
from_addr: test@myemail.com
在这里你可以配置发件人与收件人。
smtp_auth_file存放账户名和密码1
2user: xxx@qq.com
password: password
运行
python -m elastalert.elastalert --verbose --rule example_rules/example_frequency.yaml
将会打出日志
使用短信告警
这里选择使用elastalert的command调用发短信的脚本来进行短信报警
在配置文件中:1
2
3
4alert:
- command
command: ["/root/dysms_python/sms_send.py","--host","%(host)s","--time","%(time)s","--path","%(source)s"]
/root/dysms_python/sms_send.py 是该脚本名称,后面是传入的参数
增强功能
现在想做到这一点:在nginx日志中发现500之后去查另一个type(表)中与之对应的traceback附在邮件中。你可以在enhancements中改变告警内容1
2
3
4$ mkdir elastalert_modules
$ cd elastalert_modules
$ touch __init__.py
$ vim my_enhancements.py
官方示例:1
2
3
4
5
6
7
8
9
10
11from elastalert.enhancements import BaseEnhancement
class MyEnhancement(BaseEnhancement):
# The enhancement is run against every match
# The match is passed to the process function where it can be modified in any way
# ElastAlert will do this for each enhancement linked to a rule
def process(self, match):
if 'domain' in match:
url = "http://who.is/whois/%s" % (match['domain'])
match['domain_whois_link'] = url
然后在你的规则配置文件中1
2match_enhancements:
- "elastalert_modules.my_enhancements.MyEnhancement"
此时就会多一个domain_whois_link的字段,并且该值是http://who.is/whois/+domain_whois_link
es的精确值匹配
使用term可以进行精确值匹配,比如1
"term" : { "user" : "Kimchy" }
会查找到user为Kimchy的记录
但是你如果这样查
“term” : { “user” : “Kimchy Hello” }
你会找不到Kimchy Hello的记录!(已经存放了该条记录)
这是因为lucene的底层是倒排索引,会拆成Kimchy与Hello两个关键字,所以使用term是无法精确匹配到的。你必须删除索引并重新建立时将该字段标识为“not_analyzed”
具体可以看这里
Running ElastAlert for the First Time
ElastAlert监控日志告警Web攻击行为
查找准确值