阿里內(nèi)部員工，排查Java問題常用的工具單

tail?-300f?shopbase.log?#倒數(shù)300行并進(jìn)入實時監(jiān)聽文件寫入模式

grep

grep?forest?f.txt?????#文件查找
grep?forest?f.txt?cpf.txt?#多文件查找
grep?'log'?/home/admin?-r?-n?#目錄下查找所有符合關(guān)鍵字的文件
cat?f.txt?|?grep?-i?shopbase????
grep?'shopbase'?/home/admin?-r?-n?--include?*.{vm,java}?#指定文件后綴
grep?'shopbase'?/home/admin?-r?-n?--exclude?*.{vm,java}?#反匹配
seq?10?|?grep?5?-A?3????#上匹配
seq?10?|?grep?5?-B?3????#下匹配
seq?10?|?grep?5?-C?3????#上下匹配，平時用這個就妥了
cat?f.txt?|?grep?-c?'SHOPBASE'

awk

awk?'{print?$4,$6}'?f.txt
awk?'{print?NR,$0}'?f.txt?cpf.txt????
awk?'{print?FNR,$0}'?f.txt?cpf.txt
awk?'{print?FNR,FILENAME,$0}'?f.txt?cpf.txt
awk?'{print?FILENAME,"NR="NR,"FNR="FNR,"$"NF"="$NF}'?f.txt?cpf.txt
echo?1:2:3:4?|?awk?-F:?'{print?$1,$2,$3,$4}'

awk?'/ldb/?{print}'?f.txt???#匹配ldb
awk?'!/ldb/?{print}'?f.txt??#不匹配ldb
awk?'/ldb/?&&?/LISTEN/?{print}'?f.txt???#匹配ldb和LISTEN
awk?'$5?~?/ldb/?{print}'?f.txt?#第五列匹配ldb

find

sudo?-u?admin?find?/home/admin?/tmp?/usr?-name?\*.log(多個目錄去找)
find?.?-iname?\*.txt(大小寫都匹配)
find?.?-type?d(當(dāng)前目錄下的所有子目錄)
find?/usr?-type?l(當(dāng)前目錄下所有的符號鏈接)
find?/usr?-type?l?-name?"z*"?-ls(符號鏈接的詳細(xì)信息?eg:inode,目錄)
find?/home/admin?-size?+250000k(超過250000k的文件，當(dāng)然+改成-就是小于了)
find?/home/admin?f?-perm?777?-exec?ls?-l?{}?\;?(按照權(quán)限查詢文件)
find?/home/admin?-atime?-1??1天內(nèi)訪問過的文件
find?/home/admin?-ctime?-1??1天內(nèi)狀態(tài)改變過的文件????
find?/home/admin?-mtime?-1??1天內(nèi)修改過的文件
find?/home/admin?-amin?-1??1分鐘內(nèi)訪問過的文件
find?/home/admin?-cmin?-1??1分鐘內(nèi)狀態(tài)改變過的文件????
find?/home/admin?-mmin?-1??1分鐘內(nèi)修改過的文件

pgm

pgm?-A?-f?vm-shopbase?'cat?/home/admin/shopbase/logs/shopbase.log.2017-01-17|grep?2069861630'

tsar

tsar??###可以查看最近一天的各項指標(biāo)

tsar?--live?###可以查看實時指標(biāo)，默認(rèn)五秒一刷

tsar?-d?20161218?###指定查看某天的數(shù)據(jù)，貌似最多只能看四個月的數(shù)據(jù)

tsar?--mem
tsar?--load
tsar?--cpu
###當(dāng)然這個也可以和-d參數(shù)配合來查詢某天的單個指標(biāo)的情況?

top

ps?-ef?|?grep?java
top?-H?-p?pid

其他

netstat?-nat|awk??'{print?$6}'|sort|uniq?-c|sort?-rn?
#查看當(dāng)前連接，注意close_wait偏高的情況，比如如下

@OnMethod(clazz?=?"java.util.ArrayList",?method="add",?location?=?@Location(value?=?Kind.CALL,?clazz?=?"/.*/",?method?=?"/.*/"))
public?static?void?m(@ProbeClassName?String?probeClass,?@ProbeMethodName?String?probeMethod,?@TargetInstance?Object?instance,?@TargetMethodOrField?String?method)?{
???if(getInt(field("java.util.ArrayList",?"size"),?instance)?>?479){
???????println("check?who?ArrayList.add?method:"?+?probeClass?+?"#"?+?probeMethod??+?",?method:"?+?method?+?",?size:"?+?getInt(field("java.util.ArrayList",?"size"),?instance));
???????jstack();
???????println();
???????println("===========================");
???????println();
???}
}

@OnMethod(clazz?=?"com.taobao.sellerhome.transfer.biz.impl.C2CApplyerServiceImpl",?method="nav",?location?=?@Location(value?=?Kind.RETURN))
public?static?void?mt(long?userId,?int?current,?int?relation,?String?check,?String?redirectUrl,?@Return?AnyType?result)?{
???println("parameter#?userId:"?+?userId?+?",?current:"?+?current?+?",?relation:"?+?relation?+?",?check:"?+?check?+?",?redirectUrl:"?+?redirectUrl?+?",?result:"?+?result);
}

1. 經(jīng)過觀察，1.3.9的release輸出不穩(wěn)定，要多觸發(fā)幾次才能看到正確的結(jié)果

2. 正則表達(dá)式匹配trace類時范圍一定要控制，否則極有可能出現(xiàn)跑滿CPU導(dǎo)致應(yīng)用卡死的情況

3. 由于是字節(jié)碼注入的原理，想要應(yīng)用恢復(fù)到正常情況，需要重啟應(yīng)用。

Greys

javOSize

JProfiler

sudo?-u?admin?/opt/taobao/java/bin/jps?-mlvV

jstack

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack?2815

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstack?-m?2815

jinfo

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jinfo?-flags?2815

jmap

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap?-heap?2815

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap?-dump:live,format=b,file=/tmp/heap2.bin?2815

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap?-dump:format=b,file=/tmp/heap3.bin?2815

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jmap?-histo?2815?|?head?-10

jstat

sudo?-u?admin?/opt/taobao/install/ajdk-8_1_1_fp1-b52/bin/jstat?-gcutil?2815?1000?

jdb

CHLSDB

sudo?-u?admin?/opt/taobao/java/bin/java?-classpath?/opt/taobao/java/lib/sa-jdi.jar?sun.jvm.hotspot.CLHSDB

maven helper

-XX:+TraceClassLoading
結(jié)果形如[Loaded?java.lang.invoke.MethodHandleImpl$Lazy?from?D:\programme\jdk\jdk8U74\jre\lib\rt.jar]

-XX:+HeapDumpOnOutOfMemoryError?-XX:HeapDumpPath=/home/admin/logs/java.hprof

mvn?dependency:tree?> ~/dependency.txt

mvn?dependency:tree -Dverbose -Dincludes=groupId:artifactId

-XX:+TraceClassLoading

-verbose

greys:sc

tomcat-classloader-locate

sudo?dmesg|grep?-i?kill|less

[6710782.021013]?java?invoked?oom-killer:?gfp_mask=0xd0,?order=0,?oom_adj=0,?oom_scoe_adj=0
[6710782.070639]?[]???oom_kill_process+0x68/0x140?
[6710782.257588]?Task?in?/LXC011175068174?killed?as?a?result?of?limit?of?/LXC011175068174?
[6710784.698347]?Memory?cgroup?out?of?memory:?Kill?process?215701?(java)?score?854?or?sacrifice?child?
[6710784.707978]?Killed?process?215701,?UID?679,?(java)?total-vm:11017300kB,?anon-rss:7152432kB,?file-rss:1232kB

date?-d?"1970-01-01?UTC?`echo?"$(date?+%s)-$(cat?/proc/uptime|cut?-f?1?-d'?')+12288812.926194"|bc?`?seconds"

感謝閱讀