14 Replies Latest reply on Mar 16, 2011 10:32 AM by rareddy

    java.io.IOException: Too many open files

    tawalisa

      Hi all,

       

                I did a test about deploy VDB to Teiid. I used 20 thread to create VDB. Teiid server throw below log. I want to know why it is and how to avoid it.

      java.io.IOException: Too many open files

              at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)

              at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:245)

              at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

              at java.lang.Thread.run(Thread.java:636)

       

       

      Thanks

      Best Regards

        • 1. java.io.IOException: Too many open files
          rareddy

          Allen,

           

          How many client connection do you have. What platform. Teiid certainly does not throw too many files for moderate level of usage. However if you are opening lot of connections and holding on to them and doing the large queries (millions of records) there slight possibility. But you would have describe more to about your use case.

           

          Ramesh..

          • 2. java.io.IOException: Too many open files
            shawkins

            It may help to roughly account for things that may use a file handle:

             

            -Each incoming socket connection

            -Each outgoing socket connection to a source

            -The buffermanager file cache, which is limited to 64 open files by default

            -Each read from a Teiid filestore over the previous limit.  There will never be more concurrent reads than maxActivePlans

            -Any file usage by your sources

             

            We have also seen a similar issue in the past with large jvm sizes (32gigs or more) and some vendor drivers.

             

            Avoiding the issue may include increasing your open file limit, limiting source connections, etc.

             

            Steve

            • 3. java.io.IOException: Too many open files
              tawalisa

              Thanks your help.

               

              platform: jboss soa 5.0.1   linux

               

              limit parameter

              core file size          (blocks, -c) unlimited

              data seg size           (kbytes, -d) unlimited

              scheduling priority             (-e) 0

              file size               (blocks, -f) unlimited

              pending signals                 (-i) 135167

              max locked memory       (kbytes, -l) 32

              max memory size         (kbytes, -m) unlimited

              open files                      (-n) 1024

              pipe size            (512 bytes, -p) 8

              POSIX message queues     (bytes, -q) 819200

              real-time priority              (-r) 0

              stack size              (kbytes, -s) 10240

              cpu time               (seconds, -t) unlimited

              max user processes              (-u) 135167

              virtual memory          (kbytes, -v) unlimited

              file locks                      (-x) unlimited

               

              JVM arguments:

               

              -Xms1303m -Xmx3303m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

               

              This case is used 20 concurrent thread to deploy and undeploy VDB

              Duration: 2 hours

               

              Thanks

              Best Regards

              • 4. java.io.IOException: Too many open files
                rareddy

                Allen,

                 

                Are you saying that you have 20 threads that are constantly deploying and undeploying the vdbs?

                 

                Ramesh..

                • 5. java.io.IOException: Too many open files
                  tawalisa

                  Yes

                  • 6. java.io.IOException: Too many open files
                    tawalisa

                    Hi Ramesh,

                             

                              Our logic need deploy and undeploy VDB, So we have to pass this case. Do you have any advice ?

                     

                     

                     

                    Thanks

                    Best Regards

                    Allen

                    • 7. java.io.IOException: Too many open files
                      rareddy

                      Allen,

                       

                      This is a interesting test. Usually any typical applications just deploy a VDB and use it. The queries could be done using multiple threads. Anyway couple questions,

                       

                      1) Are you using 20 different vdbs? or a single vdb.

                      2) Can provide the testcase you are using to test this.

                       

                      Thanks

                       

                      Ramesh..

                      • 8. java.io.IOException: Too many open files
                        tawalisa

                        Ramesh,

                         

                                  In our architecture, we will used many VDB. This VDB will be deployed and undeployed anytime.

                         

                         

                        Thanks

                        Best Regards

                        Allen

                        • 9. Re: java.io.IOException: Too many open files
                          tawalisa

                          Hello Ramesh,

                           

                                    I have tested with deploying  and undelpoying of VDB.

                          I found :

                              When deploy 2 VDB and undeploy 2 VDB, then I execute "lsof" to check. I found teiid server will increase 6 "open file"

                          They are:

                          java    6780 root  395r  IPv4          117035708                TCP 192.168.3.3:59916 (LISTEN)

                          java    6780 root  416u  IPv4          117035704                TCP 192.168.3.3:59915 (LISTEN)

                          java    6780 root  424u   REG              253,0   749935  68519379 /opt/jboss-teiid-5.0.1/jboss-as/server/var/mwm/jboss/server/teiid-server/tmp/teiid/buffer/b_temp-stream_0_4310886018569254268.tmp

                          java    6780 root  426u   REG              253,0   749933  68519380 /opt/jboss-teiid-5.0.1/jboss-as/server/var/mwm/jboss/server/teiid-server/tmp/teiid/buffer/b_temp-stream_0_5857253331971952634.tmp

                          java    6780 root  427u  IPv4          117035721                TCP 192.168.3.3:59917 (LISTEN)

                          java    6780 root  428u  IPv4          117035722                TCP 192.168.3.3:59918 (LISTEN)

                           

                           

                          I found teiid add 2 open file  and 4 listen. Maybe it is caused this issue. Do you think it is leak for teiid?

                          Thanks

                          • 10. java.io.IOException: Too many open files
                            tawalisa

                            Hello Ramesh,

                             

                                

                             

                                      I met many question about teiid in our endurance test. I want to know whether it is reasonable. In our architecture, we will used many VDB.Maybe the number will over 100. And under the each VDB, it will have another vdb and an oracle.  I think this architecture is so large for teiid.

                            Do you have any advice? Or do you think how many should teiid in the jboss?

                             

                            Thanks your help.

                             

                            Allen

                            • 11. java.io.IOException: Too many open files
                              rareddy

                              Allen,

                               

                              This is more of a tuning question, please open another discussion thread for this and post the link here. Simple answer, possible depending upon your VM size and capabilities of the server.

                               

                              Thanks.

                               

                              Ramesh..

                              • 12. java.io.IOException: Too many open files
                                rareddy

                                Allen,

                                 

                                I logged https://issues.jboss.org/browse/TEIID-1511

                                 

                                Thanks

                                 

                                Ramesh..

                                • 13. java.io.IOException: Too many open files
                                  tawalisa

                                  Dear Ramesh,

                                      

                                                 I have tested our test case for one day.

                                   

                                            [root@charon test]# lsof -p 6780 | grep java|wc -l

                                            1546

                                   

                                  We can see. This value has increased to 1546. So I think it is bug.

                                   

                                   

                                  Thanks for your help.

                                   

                                  Best Regards

                                  Allen

                                  • 14. java.io.IOException: Too many open files
                                    rareddy

                                    Allen,

                                     

                                    Yes, but I need to do further testing. Follow the above JIRA for the progress of the issue.

                                     

                                    Thanks

                                     

                                    Ramesh..