loki问题

filetarget.fsnotify.NewWatcher: too many open files

报错信息：level=error ts=2024-01-04T07:02:47.637495247Z caller=filetargetmanager.go:265 msg="Failed to create target" key="{app="xxx", container="xxx", job="default/xxx", namespace="default", pod="xxx-57988fb6db-q59zz", pod_template_hash="57988fb6db"}" error="filetarget.fsnotify.NewWatcher: too many open files"

问题分析：loki部署在k8s集群中，集群里有一些也容器产生的日志文件特别多，导致loki需要watcher的文件数过多，inotify数量不足

网上搜了一下，这一篇比较靠谱，记录一下。原文链接：https://www.jianshu.com/p/337946a30013

问题描述

level=info ts=2021-01-21T01:56:05.301932147Z caller=server.go:225 http=[::]:9080 grpc=[::]:41303 msg="server listening on addresses"
level=info ts=2021-01-21T01:56:05.302278747Z caller=main.go:108 msg="Starting Promtail" version="(version=2.0.0, branch=HEAD, revision=6978ee5d7)"
level=info ts=2021-01-21T01:56:10.30212042Z caller=filetargetmanager.go:261 msg="Adding target" key="{log_from=\"static_pods\"}"
level=error ts=2021-01-21T01:56:10.302202345Z caller=filetargetmanager.go:265 msg="Failed to create target" key="{log_from=\"static_pods\"}" error="filetarget.fsnotify.NewWatcher: too many open files"

解决办法

# 在宿主机执行
sudo sysctl -w fs.inotify.max_user_instances=512

参考文档

• https://github.com/grafana/loki/issues/1153
• https://unix.stackexchange.com/questions/444998/how-to-set-and-understand-fs-notify-max-user-watches