同事跟我說線上的一個(gè)dubbo provider服務(wù)啟動(dòng)不了了，然后發(fā)了一段報(bào)錯(cuò)信息，因?yàn)檫@個(gè)項(xiàng)目之前一直是我在跟，我就登上機(jī)器看了下

1.排查原因

// 這邊省略了很多，loadClass遞歸了導(dǎo)致了StackOverflowError
Exception in thread "main" java.lang.StackOverflowError
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)</font>
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethods(Class.java:1975)
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:612)
    at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
    at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:510)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(AutowiredAnnotationBeanPostProcessor.java:241)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineConstructorsFromBeanPostProcessors(AbstractAutowireCapableBeanFactory.java:1069)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1042)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
    at com.alibaba.dubbo.container.spring.SpringContainer.start(SpringContainer.java:50)
    at com.alibaba.dubbo.container.Main.main(Main.java:80)

整個(gè)異常的調(diào)用棧就是這樣。初步看起來是spring容器在創(chuàng)建bean的時(shí)候，加載這個(gè)bean相關(guān)的類時(shí)導(dǎo)致了棧溢出。
我們部署的dubbo provider使用的是 Dubbo 中的服務(wù)容器，dubbo的服務(wù)容器啟動(dòng)使用的是下圖標(biāo)紅的start.sh

1648997410(1).png

這個(gè)腳本中的設(shè)定是64位機(jī)器上線程的棧最大是256k，雖然256k的確有些小，但之前一直也都o(jì)k的，怎么現(xiàn)在不行了。

BITS=`java -version 2>&1 | grep -i 64-bit`
if [ -n "$BITS" ]; then
    # JAVA_MEM_OPTS=" -server -Xmx2g -Xms2g -Xmn256m -XX:PermSize=128m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
    JAVA_MEM_OPTS=" -server -Xmx2g -Xms2g -Xmn256m -XX:MetaspaceSize=128m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
    JAVA_MEM_OPTS=" -server -Xms1g -Xmx1g -XX:PermSize=128m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi

本地啟動(dòng)vm參數(shù)加入-Xss256k 果然是相同的報(bào)錯(cuò),加大到-Xss512k就能正常啟動(dòng)了。但我還是想看看到底是什么新的代碼導(dǎo)致了這個(gè)問題。
分析調(diào)用棧是spring容器初始化bean的時(shí)候需要獲取對(duì)象的構(gòu)造方法，遍歷了這個(gè)類的所有的方法，期間觸發(fā)了類的加載，導(dǎo)致的stackOverflow。可是這個(gè)代碼并沒有日志打印beanName，無從知曉是哪個(gè)類引起的問題，也不可能在這邊打斷點(diǎn)一個(gè)個(gè)看，畢竟那么多需要加載的bean呢，那有什么辦法呢。javaAgent加日志好像可行。

    @Override
    public Constructor<?>[] determineCandidateConstructors(Class<?> beanClass, final String beanName) throws BeansException {
        if (!this.lookupMethodsChecked.contains(beanName)) {
            ReflectionUtils.doWithMethods(beanClass, new ReflectionUtils.MethodCallback() {
                @Override
                public void doWith(Method method) throws IllegalArgumentException, IllegalAccessException {
                    Lookup lookup = method.getAnnotation(Lookup.class);
                    if (lookup != null) {
                        LookupOverride override = new LookupOverride(method, lookup.value());
                        try {
                            RootBeanDefinition mbd = (RootBeanDefinition) beanFactory.getMergedBeanDefinition(beanName);
                            mbd.getMethodOverrides().addOverride(override);
                        }
                        catch (NoSuchBeanDefinitionException ex) {
                            throw new BeanCreationException(beanName,
                                    "Cannot apply @Lookup to beans without corresponding bean definition");
                        }
                    }
                }
            });
            this.lookupMethodsChecked.add(beanName);
        }

2.寫一個(gè)javaagent

參考了如下博文
基于 Javassist 和 Javaagent 實(shí)現(xiàn)動(dòng)態(tài)切面
JVM插碼之三：javaagent介紹及javassist介紹
使用agent和javassist實(shí)現(xiàn)基于jvm的aop日志打印系統(tǒng)
完成了自己的實(shí)現(xiàn)，具體的實(shí)現(xiàn)可以查看傳入類名和方法名打印指定方法的入?yún)gent實(shí)現(xiàn)--GitHub這邊就不具體講了
jvm 參數(shù)加上

 -javaagent:D:\javaAgentDemo\javaAgent\target\javaAgent-1.0.0-SNAPSHOT.jar=AutowiredAnnotationBeanPostProcessor,determineCandidateConstructors

日志打印顯示是我們使用obs的bean,每次啟動(dòng)到這就會(huì)打印StackOverflowError

image.png

鎖定了一個(gè)方法，為什么鎖定這個(gè)方法呢
最近我們的對(duì)象存儲(chǔ)服務(wù)從阿里云遷移到了華為云，其他方法都跟阿里云一樣，就這個(gè)是我新增的一個(gè)方法。

public ObsClient getObsClient() {
        String accessKeyId = obsConf.getAccessKeyId();
        String accessKeySecret = obsConf.getAccessKeySecret();
        String endpoint = obsConf.getEndpoint();
        return new ObsClient(accessKeyId, accessKeySecret, endpoint);
    }

可看起來這個(gè)方法也沒什么特殊的啊，一看這個(gè)方法的返回嚇一跳。ObsClient 的繼承關(guān)系居然有20多層，spring獲取ObsOperateServiceImpl的方法時(shí)需要加載ObsClient這個(gè)類，由于這個(gè)類的繼承層級(jí)有20多層導(dǎo)致了StackOverflowError。把這個(gè)方法注釋掉就能正常啟動(dòng)了。
（華為云的代碼真心不怎么樣啊，這個(gè)ObsClient集合了一堆東西，jar包就有5m大）

image.png

可惜的是google了很久也沒有找到能準(zhǔn)確統(tǒng)計(jì)棧內(nèi)存增加的方法。