В предыдущих статьях мы собирали о отправляли логи с помощью beats. Сегодня будем собирать, парсить и отправлять логи с помощью Fluentd в Elastic
Начнем с простого приложения на python, которое генерирует логи каждые 10 секунд:
|
1 2 3 4 5 6 7 8 9 10 11 |
import random import time user = ("John Sonner", "Adam Walker", "Frank Sinatra", "Alex Baldwin", "Stefany Frizzell") level = ("INFO", "WARNING", "FAIL", "CRITICAL", "DISASTER") message = ("User parameters updating fails", "Something goes wrong with server", "Payment disabled for user", "Connection lost", "API not availiable") while True: num = random.randrange(0,5) print(level[num] + ' \"' + user[num] + '\" \"' + message[num] + '\"') time.sleep(10) |
Вывод приложения выглядит так:
INFO «John Sonner» «User parameters updating fails»
INFO «John Sonner» «User parameters updating fails»
WARNING «Adam Walker» «Something goes wrong with server»
CRITICAL «Alex Baldwin» «Connection lost»
FAIL «Frank Sinatra» «Payment disabled for user»
Пишем Dockerfile:
|
1 2 3 4 5 6 |
FROM python:3.7.3-alpine3.8 RUN mkdir -p /opt/bin WORKDIR /opt/bin COPY ./codebase /opt/bin CMD [ "python", "-u", "logsGenerator.py" ] |
Собираем docker контейнер
|
1 |
docker build --no-cache --rm -t imagename:tag . |
Отправляем его в registry:
|
1 |
docker push imagename:tag |
Создаем файл деплоя для kubernetes:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
apiVersion: v1 kind: Namespace metadata: name: logs-generator-app --- apiVersion: apps/v1 kind: Deployment metadata: name: logs-generator namespace: logs-generator-app labels: app: logs version: beta spec: replicas: 3 selector: matchLabels: app: logs template: metadata: labels: app: logs version: beta spec: containers: - name: logs-generator image: imagename:tag |
Деплоим наше приложение в kubernetes:
|
1 |
kubectl apply -f deployment.yaml |
Настраиваем fluentd:
Создаем файл fluentd-cm-application.yaml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: default data: fluent.conf: | @include systemd.conf @include kubernetes.conf @include conf.d/*.conf <match **> @type elasticsearch @id out_es @log_level info include_tag_key true host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}" port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}" path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}" scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}" ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}" ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1'}" user "#{ENV['FLUENT_ELASTICSEARCH_USER']}" password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}" reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}" reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'false'}" reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'false'}" logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}" logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}" index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}" type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}" <buffer> flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}" flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}" chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}" queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}" retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}" retry_forever true </buffer> </match> systemd.conf: | # AUTOMATICALLY GENERATED # DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/systemd.conf.erb kubernetes.conf: | # AUTOMATICALLY GENERATED # DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/kubernetes.conf.erb <match fluent.**> @type null </match> <source> @type tail @id in_tail_container_logs path /var/log/containers/logs-generator-*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true <parse> @type json time_format %Y-%m-%dT%H:%M:%S.%NZ </parse> </source> <filter kubernetes.**> @type parser key_name log <parse> @type regexp expression /^(?<level>[A-Z]*) "(?<user>[^\"]*)" "(?<message>[^\"]*)"$/ time_format %d/%b/%Y:%H:%M:%S %z </parse> </filter> <filter kubernetes.**> @type kubernetes_metadata @id filter_kube_metadata </filter> disable.conf: | #empty |
и fluentd-deployment.yaml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
--- apiVersion: v1 kind: ServiceAccount metadata: name: fluentd namespace: default --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: fluentd namespace: default rules: - apiGroups: - "" resources: - pods - namespaces verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: fluentd roleRef: kind: ClusterRole name: fluentd apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluentd namespace: default --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd namespace: default labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: template: metadata: labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule initContainers: - name: copy-fluentd-config image: busybox command: ['sh', '-c', 'cp /config-volume/..data/* /fluentd/etc'] volumeMounts: - name: config-volume mountPath: /config-volume - name: fluentdconf mountPath: /fluentd/etc containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.3-debian-elasticsearch-1 env: - name: FLUENT_ELASTICSEARCH_HOST value: "" - name: FLUENT_ELASTICSEARCH_PORT value: "" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME value: pbs-app - name: FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX value: pbs-app resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: config-volume mountPath: /config-volume - name: fluentdconf mountPath: /fluentd/etc - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: config-volume configMap: name: fluentd-config - name: fluentdconf emptyDir: {} - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers |
Дописываем в переменные наши значения:
| Переменная | Описание |
| FLUENT_ELASTICSEARCH_HOST | Адрес elasticsearch сервера |
| FLUENT_ELASTICSEARCH_PORT | Порт elasticsearch сервера |
| FLUENT_ELASTICSEARCH_SCHEME | http,https |
| FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME | индекс elasticsearch |
Деплоим fluentd в kubernetes:
|
1 |
kubectl apply -f fluentd-cm-application.yaml -f fluentd-deployment.yaml |
Проверяем:
NAME READY STATUS RESTARTS AGE
fluentd-54v2f 1/1 Running 0 98m
fluentd-knvl5 1/1 Running 0 98m
fluentd-mw6fq 1/1 Running 0 98m
fluentd-nwrbc 1/1 Running 0 98m
Настраиваем kibana:
Переходим в Management -> Index Patterns и добавляем свой индекс
Переходим в discover и выбираем наш Index:
Тот же вывод только в json:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
{ "_index": "pbs-app-2019.06.20", "_type": "fluentd", "_id": "Pq1sdWsBIUk8DCg7Kilo", "_version": 1, "_score": null, "_source": { "level": "DISASTER", "user": "Stefany Frizzell", "message": "API not availiable", "docker": { "container_id": "41c0fadf4d0bd43ebb443316a82a838d5e543e4d53da94d2687cd501748fc514" }, "kubernetes": { "container_name": "logs-generator", "namespace_name": "logs-generator-app", "pod_name": "logs-generator-d489555f9-jwswb", "container_image": "wacken/logs-generator:beta-2", "container_image_id": "docker-pullable://wacken/logs-generator@sha256:a9eac59fac4d93e8273cdf16d05b1b90d3a7d8966bcd4b2c7967644e22827394", "pod_id": "8dcdca6c-934a-11e9-b7d8-96000028f204", "labels": { "app": "logs", "pod-template-hash": "d489555f9", "version": "beta" }, "host": "kc-fluentd-app-worker4", "master_url": "https://10.43.0.1:443/api", "namespace_id": "f61ff9e8-9346-11e9-b7d8-96000028f204", "namespace_labels": { "field_cattle_io/projectId": "p-tnvtx" } }, "@timestamp": "2019-06-20T15:05:46.561534572+00:00", "tag": "kubernetes.var.log.containers.logs-generator-d489555f9-jwswb_logs-generator-app_logs-generator-41c0fadf4d0bd43ebb443316a82a838d5e543e4d53da94d2687cd501748fc514.log" }, "fields": { "@timestamp": [ "2019-06-20T15:05:46.561Z" ] }, "sort": [ 1561043146561 ] } |
Для сбора логов NGINX напишем fluentd-cm-nginx.yaml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: default data: fluent.conf: | @include systemd.conf @include kubernetes.conf @include conf.d/*.conf <match **> @type elasticsearch @id out_es @log_level info include_tag_key true host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}" port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}" path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}" scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}" ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}" ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1'}" user "#{ENV['FLUENT_ELASTICSEARCH_USER']}" password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}" reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}" reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'false'}" reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'false'}" logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}" logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}" index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}" type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}" <buffer> flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}" flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}" chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}" queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}" retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}" retry_forever true </buffer> </match> systemd.conf: | # AUTOMATICALLY GENERATED # DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/systemd.conf.erb kubernetes.conf: | # AUTOMATICALLY GENERATED # DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/kubernetes.conf.erb <match fluent.**> @type null </match> <source> @type tail @id in_tail_container_logs path /var/log/containers/nginx-*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true <parse> @type json time_format %Y-%m-%dT%H:%M:%S.%NZ </parse> </source> <filter kubernetes.**> @type parser key_name log <parse> @type regexp expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/ time_format %d/%b/%Y:%H:%M:%S %z </parse> </filter> <filter kubernetes.**> @type kubernetes_metadata @id filter_kube_metadata </filter> disable.conf: | #empty |
В kibana увидим следующее:
Это все, спасибо за внимание!
Все тоже самое можете найти на моем github