14 Replies Latest reply: Mar 16, 2011 10:32 AM by Ramesh Reddy RSS

java.io.IOException: Too many open files

allen li Newbie

Hi all,

 

          I did a test about deploy VDB to Teiid. I used 20 thread to create VDB. Teiid server throw below log. I want to know why it is and how to avoid it.

java.io.IOException: Too many open files

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)

        at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:245)

        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

        at java.lang.Thread.run(Thread.java:636)

 

 

Thanks

Best Regards

  • 1. java.io.IOException: Too many open files
    Ramesh Reddy Master

    Allen,

     

    How many client connection do you have. What platform. Teiid certainly does not throw too many files for moderate level of usage. However if you are opening lot of connections and holding on to them and doing the large queries (millions of records) there slight possibility. But you would have describe more to about your use case.

     

    Ramesh..

  • 2. java.io.IOException: Too many open files
    Steven Hawkins Master

    It may help to roughly account for things that may use a file handle:

     

    -Each incoming socket connection

    -Each outgoing socket connection to a source

    -The buffermanager file cache, which is limited to 64 open files by default

    -Each read from a Teiid filestore over the previous limit.  There will never be more concurrent reads than maxActivePlans

    -Any file usage by your sources

     

    We have also seen a similar issue in the past with large jvm sizes (32gigs or more) and some vendor drivers.

     

    Avoiding the issue may include increasing your open file limit, limiting source connections, etc.

     

    Steve

  • 3. java.io.IOException: Too many open files
    allen li Newbie

    Thanks your help.

     

    platform: jboss soa 5.0.1   linux

     

    limit parameter

    core file size          (blocks, -c) unlimited

    data seg size           (kbytes, -d) unlimited

    scheduling priority             (-e) 0

    file size               (blocks, -f) unlimited

    pending signals                 (-i) 135167

    max locked memory       (kbytes, -l) 32

    max memory size         (kbytes, -m) unlimited

    open files                      (-n) 1024

    pipe size            (512 bytes, -p) 8

    POSIX message queues     (bytes, -q) 819200

    real-time priority              (-r) 0

    stack size              (kbytes, -s) 10240

    cpu time               (seconds, -t) unlimited

    max user processes              (-u) 135167

    virtual memory          (kbytes, -v) unlimited

    file locks                      (-x) unlimited

     

    JVM arguments:

     

    -Xms1303m -Xmx3303m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

     

    This case is used 20 concurrent thread to deploy and undeploy VDB

    Duration: 2 hours

     

    Thanks

    Best Regards

  • 4. java.io.IOException: Too many open files
    Ramesh Reddy Master

    Allen,

     

    Are you saying that you have 20 threads that are constantly deploying and undeploying the vdbs?

     

    Ramesh..

  • 6. java.io.IOException: Too many open files
    allen li Newbie

    Hi Ramesh,

             

              Our logic need deploy and undeploy VDB, So we have to pass this case. Do you have any advice ?

     

     

     

    Thanks

    Best Regards

    Allen

  • 7. java.io.IOException: Too many open files
    Ramesh Reddy Master

    Allen,

     

    This is a interesting test. Usually any typical applications just deploy a VDB and use it. The queries could be done using multiple threads. Anyway couple questions,

     

    1) Are you using 20 different vdbs? or a single vdb.

    2) Can provide the testcase you are using to test this.

     

    Thanks

     

    Ramesh..

  • 8. java.io.IOException: Too many open files
    allen li Newbie

    Ramesh,

     

              In our architecture, we will used many VDB. This VDB will be deployed and undeployed anytime.

     

     

    Thanks

    Best Regards

    Allen

  • 9. Re: java.io.IOException: Too many open files
    allen li Newbie

    Hello Ramesh,

     

              I have tested with deploying  and undelpoying of VDB.

    I found :

        When deploy 2 VDB and undeploy 2 VDB, then I execute "lsof" to check. I found teiid server will increase 6 "open file"

    They are:

    java    6780 root  395r  IPv4          117035708                TCP 192.168.3.3:59916 (LISTEN)

    java    6780 root  416u  IPv4          117035704                TCP 192.168.3.3:59915 (LISTEN)

    java    6780 root  424u   REG              253,0   749935  68519379 /opt/jboss-teiid-5.0.1/jboss-as/server/var/mwm/jboss/server/teiid-server/tmp/teiid/buffer/b_temp-stream_0_4310886018569254268.tmp

    java    6780 root  426u   REG              253,0   749933  68519380 /opt/jboss-teiid-5.0.1/jboss-as/server/var/mwm/jboss/server/teiid-server/tmp/teiid/buffer/b_temp-stream_0_5857253331971952634.tmp

    java    6780 root  427u  IPv4          117035721                TCP 192.168.3.3:59917 (LISTEN)

    java    6780 root  428u  IPv4          117035722                TCP 192.168.3.3:59918 (LISTEN)

     

     

    I found teiid add 2 open file  and 4 listen. Maybe it is caused this issue. Do you think it is leak for teiid?

    Thanks

  • 10. java.io.IOException: Too many open files
    allen li Newbie

    Hello Ramesh,

     

        

     

              I met many question about teiid in our endurance test. I want to know whether it is reasonable. In our architecture, we will used many VDB.Maybe the number will over 100. And under the each VDB, it will have another vdb and an oracle.  I think this architecture is so large for teiid.

    Do you have any advice? Or do you think how many should teiid in the jboss?

     

    Thanks your help.

     

    Allen

  • 11. java.io.IOException: Too many open files
    Ramesh Reddy Master

    Allen,

     

    This is more of a tuning question, please open another discussion thread for this and post the link here. Simple answer, possible depending upon your VM size and capabilities of the server.

     

    Thanks.

     

    Ramesh..

  • 12. java.io.IOException: Too many open files
    Ramesh Reddy Master

    Allen,

     

    I logged https://issues.jboss.org/browse/TEIID-1511

     

    Thanks

     

    Ramesh..

  • 13. java.io.IOException: Too many open files
    allen li Newbie

    Dear Ramesh,

        

                   I have tested our test case for one day.

     

              [root@charon test]# lsof -p 6780 | grep java|wc -l

              1546

     

    We can see. This value has increased to 1546. So I think it is bug.

     

     

    Thanks for your help.

     

    Best Regards

    Allen

  • 14. java.io.IOException: Too many open files
    Ramesh Reddy Master

    Allen,

     

    Yes, but I need to do further testing. Follow the above JIRA for the progress of the issue.

     

    Thanks

     

    Ramesh..