{"id":292,"date":"2023-11-10T21:14:00","date_gmt":"2023-11-10T21:14:00","guid":{"rendered":"http:\/\/blog.firatyasar.com\/?p=292"},"modified":"2024-03-29T21:26:28","modified_gmt":"2024-03-29T21:26:28","slug":"fluentbit-opensearch-integration-in-k8s-environment","status":"publish","type":"post","link":"https:\/\/blog.firatyasar.com\/?p=292","title":{"rendered":"Fluentbit Opensearch Integration in K8s Environment"},"content":{"rendered":"\n
\"OpenSearch<\/figure>\n\n\n\n

Elastic Search’\u00fcn lisansl\u0131 olmas\u0131ndan dolay\u0131 log repositroy olarak opensearch kullan\u0131lmaya yayg\u0131n olarak ba\u015fkand\u0131. Daha \u00f6ncelerde EFK stack olarak yap\u0131land\u0131rd\u0131\u011f\u0131m\u0131z Elasticsearch-Fluentbit-Kibana \u00fc\u00e7l\u00fcs\u00fcn\u00fcn yerini fluentbit-opensearch almaya ba\u015flad\u0131 diyebiliriz. Tabi Elastich Search kullan\u0131p enterprise \u00f6zelliklerinden faydalanmak isteyenler lisans sat\u0131n al\u0131p ayn\u0131 davran\u0131\u015fta log management yapmaya devam ediyorlar. Ancak log management i\u015fini opensource teknolojilere b\u0131rakmak isteyenler i\u00e7in opensearch alternatifi de mevcut.<\/p>\n\n\n\n

Bu makalemde fluent-bit gibi light weight bir parser’\u0131 kubernetes \u00fczerine entegre ederek container loglar\u0131n\u0131 parse edip, opensearch’e aktarmay\u0131 sizlere anlataca\u011f\u0131m.<\/p>\n\n\n\n

Cluster taraf\u0131nda gerekli rbac konfig\u00fcrasyonlar\u0131 yap\u0131ld\u0131ktan sonra a\u015fa\u011f\u0131daki fluentbit definition file’\u0131 kubernetes \u00fczerinde apply edilerek gerekli bile\u015fenler olu\u015fturulmal\u0131d\u0131r.<\/p>\n\n\n\n

---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRole\nmetadata:\n  name: fluent-bit-read\nrules:\n- apiGroups: [\"\"]\n  resources:\n  - namespaces\n  - pods\n  verbs: [\"get\", \"list\", \"watch\"]\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: fluent-bit-read\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: fluent-bit-read\nsubjects:\n- kind: ServiceAccount\n  name: fluent-bit\n  namespace: logging\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: fluent-bit-config\n  namespace: logging\n  labels:\n    k8s-app: fluent-bit\ndata:\n  # Configuration files: server, input, filters and output\n  # ======================================================\n  fluent-bit.conf: |\n    [SERVICE]\n        Flush         1\n        Log_Level     info\n        Daemon        off\n        Parsers_File  parsers.conf\n        HTTP_Server   On\n        HTTP_Listen   0.0.0.0\n        HTTP_Port     2020\n\n    @INCLUDE input-kubernetes.conf\n    @INCLUDE filter-kubernetes.conf\n    @INCLUDE output-elasticsearch.conf\n\n  input-kubernetes.conf: |\n    [INPUT]\n        Name              tail\n        Tag               kube.*\n        Path              \/var\/log\/containers\/frt*.log\n        Parser            docker\n        DB                \/var\/log\/flb_kube.db\n        Mem_Buf_Limit     50MB\n        Skip_Long_Lines   On\n        Refresh_Interval  10\n\n  filter-kubernetes.conf: |\n    [FILTER]\n        Name                kubernetes\n        Match               kube.*\n        Kube_URL            https:\/\/kubernetes.default.svc:443\n        Kube_CA_File        \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt\n        Kube_Token_File     \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token\n        Kube_Tag_Prefix     kube.var.log.containers.\n        Merge_Log           On\n        Merge_Log_Key       log_processed\n        K8S-Logging.Parser  On\n        K8S-Logging.Exclude Off\n\n  output-elasticsearch.conf: |\n    [OUTPUT]\n        Name            es\n        Match           *\n        Host            opensearch-host<\/strong><\/span>\n        Port            443\n        TLS             On\n        TLS.Verify      Off\n        AWS_Auth        On\n        AWS_Region      eu-west-1\n        Retry_Limit     6\n        Index           prod-fluent-bit\n        Suppress_Type_Name On\n        Logstash_Format On\n        Logstash_Prefix prod\n        Retry_Limit     False\n        Buffer_Size     64KB\n\n  parsers.conf: |\n    [PARSER]\n        Name   apache\n        Format regex\n        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \\[(?<time>[^\\]]*)\\] \"(?<method>\\S+)(?: +(?<path>[^\\\"]*?)(?: +\\S*)?)?\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \"(?<referer>[^\\\"]*)\" \"(?<agent>[^\\\"]*)\")?$\n        Time_Key time\n        Time_Format %d\/%b\/%Y:%H:%M:%S %z\n\n    [PARSER]\n        Name   apache2\n        Format regex\n        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \\[(?<time>[^\\]]*)\\] \"(?<method>\\S+)(?: +(?<path>[^ ]*) +\\S*)?\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \"(?<referer>[^\\\"]*)\" \"(?<agent>[^\\\"]*)\")?$\n        Time_Key time\n        Time_Format %d\/%b\/%Y:%H:%M:%S %z\n\n    [PARSER]\n        Name   apache_error\n        Format regex\n        Regex  ^\\[[^ ]* (?<time>[^\\]]*)\\] \\[(?<level>[^\\]]*)\\](?: \\[pid (?<pid>[^\\]]*)\\])?( \\[client (?<client>[^\\]]*)\\])? (?<message>.*)$\n\n    [PARSER]\n        Name   nginx\n        Format regex\n        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \\[(?<time>[^\\]]*)\\] \"(?<method>\\S+)(?: +(?<path>[^\\\"]*?)(?: +\\S*)?)?\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \"(?<referer>[^\\\"]*)\" \"(?<agent>[^\\\"]*)\")?$\n        Time_Key time\n        Time_Format %d\/%b\/%Y:%H:%M:%S %z\n\n    [PARSER]\n        Name   json\n        Format json\n        Time_Key time\n        Time_Format %d\/%b\/%Y:%H:%M:%S %z\n\n    [PARSER]\n        Name        docker\n        Format      json\n        Time_Key    time\n        Time_Format %Y-%m-%dT%H:%M:%S.%L\n        Time_Keep   On\n\n    [PARSER]\n        Name        syslog\n        Format      regex\n        Regex       ^\\<(?<pri>[0-9]+)\\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\\\/\\.\\-]*)(?:\\[(?<pid>[0-9]+)\\])?(?:[^\\:]*\\:)? *(?<message>.*)$\n        Time_Key    time\n        Time_Format %b %d %H:%M:%S\n---\napiVersion: apps\/v1\nkind: DaemonSet\nmetadata:\n  name: fluent-bit\n  namespace: logging\n  labels:\n    k8s-app: fluent-bit-logging\n    version: v1\n    kubernetes.io\/cluster-service: \"true\"\n\nspec:\n  selector:\n    matchLabels:\n      k8s-app: fluent-bit-logging\n  template:\n    metadata:\n      labels:\n        k8s-app: fluent-bit-logging\n        version: v1\n        kubernetes.io\/cluster-service: \"true\"\n      annotations:\n        prometheus.io\/scrape: \"true\"\n        prometheus.io\/port: \"2020\"\n        prometheus.io\/path: \/api\/v1\/metrics\/prometheus\n    spec:\n      containers:\n      - name: fluent-bit\n        image: amazon\/aws-for-fluent-bit:latest\n        imagePullPolicy: Always\n        ports:\n          - containerPort: 2020\n        volumeMounts:\n        - name: varlog\n          mountPath: \/var\/log\n        - name: varlibdockercontainers\n          mountPath: \/var\/lib\/docker\/containers\n          readOnly: true\n        - name: fluent-bit-config\n          mountPath: \/fluent-bit\/etc\/\n      terminationGracePeriodSeconds: 10\n      volumes:\n      - name: varlog\n        hostPath:\n          path: \/var\/log\n      - name: varlibdockercontainers\n        hostPath:\n          path: \/var\/lib\/docker\/containers\n      - name: fluent-bit-config\n        configMap:\n          name: fluent-bit-config\n      serviceAccountName: fluent-bit<\/code><\/pre>\n\n\n\n
Bu  i\u015flem tamamland\u0131ktan sonra cluser \u00fczerinde \/var\/lib\/container dizininde bulunan frt ile ba\u015flayan container loglar\u0131 opensearch’e aktar\u0131lmaya ba\u015flanacakt\u0131r.<\/p>\n\n\n\n
\u0130lgili opensearch dashboard’una giderek gerekli index’i olu\u015fturup loglar\u0131n\u0131z\u0131 g\u00f6r\u00fcnt\u00fcleyebilirsiniz.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Elastic Search’\u00fcn lisansl\u0131 olmas\u0131ndan dolay\u0131 log repositroy olarak opensearch kullan\u0131lmaya yayg\u0131n olarak ba\u015fkand\u0131. Daha \u00f6ncelerde EFK stack olarak yap\u0131land\u0131rd\u0131\u011f\u0131m\u0131z Elasticsearch-Fluentbit-Kibana \u00fc\u00e7l\u00fcs\u00fcn\u00fcn yerini fluentbit-opensearch almaya ba\u015flad\u0131 diyebiliriz. Tabi Elastich Search kullan\u0131p enterprise \u00f6zelliklerinden faydalanmak isteyenler lisans sat\u0131n al\u0131p ayn\u0131 davran\u0131\u015fta log management yapmaya devam ediyorlar. Ancak log management i\u015fini opensource teknolojilere b\u0131rakmak isteyenler i\u00e7in opensearch alternatifi\u2026 Read More »<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":293,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[112,111,109,110],"_links":{"self":[{"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=\/wp\/v2\/posts\/292"}],"collection":[{"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=292"}],"version-history":[{"count":1,"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=\/wp\/v2\/posts\/292\/revisions"}],"predecessor-version":[{"id":294,"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=\/wp\/v2\/posts\/292\/revisions\/294"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=\/wp\/v2\/media\/293"}],"wp:attachment":[{"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=292"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.firatyasar.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}