服务器资源排查

OOM错误

现象，某后端服务早上发现状态是unhealthy。重启后，正常运行。

2025-06-06 09:27:52.847ERROR 50---[http-nio-9104-exec-29]c.g.c.s.p.e.DefaultExceptionHandleAdvice:Handler dispatch failed;nested exception is java.lang.OutOfMemoryError:GC overhead limit exceeded

        org.springframework.web.util.NestedServletException:Handler dispatch failed;nested exception is java.lang.OutOfMemoryError:GC overhead limit exceeded
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1087)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:555)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337)

• 原因：启动参数设置太小 -Xms256m -Xmx256m；然后free -h查看剩余内存空间足够，就修改为了1024m

• 深层原因：

  • 需要追查分析堆转储：使用工具（如 MAT、VisualVM）分析堆转储，定位占用内存的大对象或泄漏点

  • 遗憾的是，虽然配置了OOM时的堆转储路径，但是指定的**/home/log目录在容器内不存在，root用户也不会创建不存在的目录**，导致文件没有存储下来！

  java -server -Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/log/java_heapdump.hprof -jar /data/app.jar
  

spring服务引起CPU高占用

• 实时查看服务器资源占用

$top

• 实时查看docker容器资源占用

$ docker stats

• 进入容器查看

  $ docker exec -it 容器id sh
  
  # 获取到java服务的进程pid
  $ ps -ef | grep java
  
  # 
  $ jstat -gc pid号 1s 10