1 2 Previous Next 17 Replies Latest reply: Jul 19, 2012 2:33 AM by wiktorowski maximilien RSS

Bad Gateway

Steve Chernyak Newbie

I have gone through the documentation on jboss.com and set up a basic proxy. Client requests are proxied over an ssl connection, but the connection to the cluster over port 6666 is unencrypted. I'm testing with the admin-console web app.

 

I've set the bindaddress on the 8443 connector to the host name of the server (which matches the common name on the certificate), however, according to the apache cluster manager, the name of the node is the ip address. This causes apache to throw up warnings about mismatches between the certificate CN and the server name. Is there a way to force the node to connect to the proxy using the hostname instead of the ip address?

 

I'm also experiencing intermitted 502 (Bad Gateway) problems. There is nothing logged on the app server side when these errors start happening, and on the apache side, the only thing logged is the 502 error:

 

[Fri Jan 13 09:56:57 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 1<jboss ip>:8443 (<jboss ip>)

[Fri Jan 13 09:56:57 2012] [error] proxy: pass request body failed to <jboss ip>:8443 (<jboss ip>) from <my ip> ()

 

Sometimes this problem goes away after refreshing the browser, and sometimes i have to restart apache. Is there additional logging that can be turned on to see what the problem is?

 

Environment:

Red Hat Enterprise Linux Server release 6.0 (Santiago)

Apache 2.2.15

JBoss EAP 5.1

 

Thanks

  • 1. Re: Bad Gateway
    Radoslav Husar Master

    Steve, yes, please turn on debug logging in Apache:

    LogLevel debug

  • 2. Re: Bad Gateway
    Steve Chernyak Newbie

    Turning debug on produced the following:

     

    [Fri Jan 13 10:23:52 2012] [info] [client <jboss ip>] SSL Proxy connect failed

    [Fri Jan 13 10:23:52 2012] [info] SSL Library Error: 336032754 error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message

    [Fri Jan 13 10:23:52 2012] [info] [client <jboss ip>] Connection closed to child 0 with abortive shutdown (server <apache host>:443)

    [Fri Jan 13 10:23:52 2012] [error] (502)Unknown error 502: proxy: pass request body failed to <jboss ip>:8443 (<jboss ip>)

     

    I'm guessing there is a mismatch between servers and the encryption... Wierd that it would be intermitten.

     

    Thanks.

     

    What bout using the jboss host name for the node, is there a way to make that happen?

  • 3. Re: Bad Gateway
    Radoslav Husar Master

    YW!

    What bout using the jboss host name for the node, is there a way to make that happen?

    Not sure what you mean, can you clarify? You should be able to use hostname or IP interchangeably anywhere in mod_cluser configs if thats the question.

  • 4. Re: Bad Gateway
    Steve Chernyak Newbie

    I have the binding address for https connector set to the host name of the server. The host name matches the common name on the ssl certificate. The problem is that when jboss connects to the apache proxy, it's still using the IP address for the node name. This causes apache to throw up warnings because the common name on the ssl certificate doesn't match the ip address. I'd like to force the jboss and apache to use the host name instead of the ip address.

     

    I think we've figured out the intermittent 502 errors. It seems iptables was configured to drop ack/fin packets from jboss. This would cause apache to attempt to reuse ssl sessions that where closed by jboss, which would cause iptables to reject connections from apache, which would cause apache to report bad gateways. We're continuing to test, but it's been working fine for a while now.

     

    Thanks again

  • 5. Re: Bad Gateway
    Jean-Frederic Clere Master

    that can't work. You need on certificate/key for each machine and you need to forward the SSL information httpd has received to jboss (See https://community.jboss.org/wiki/SSLModproxyForwarding).

  • 6. Re: Bad Gateway
    Steve Chernyak Newbie

    I'm not following... I'm not trying to do client side cert authentication. The issue I'm having, is jboss is configured to run https on port 8443. The server certificate for the jboss instance was issued for the host name as the CN. When this jboss node is added to the proxy on the apache httpd instance, the node is identified by the ip address. Whe apache establishes an https connection back to jboss on 8443, it throws up a warning because it's connecting using the ip address while the CN on the certificate jboss is presenting is the host name. I'm trying to figure out how to force the jboss node to be identified by the host name instead of the ip address in apache. Hope that made sense.

     

    Thanks

  • 7. Re: Bad Gateway
    Steve Chernyak Newbie

    Looks like the 502s are back... iptables is off. It seems like it happens most often after a few hours of idle time (first time testing in the morning), but I'm also seeing it right after apache restarts...

  • 8. Re: Bad Gateway
    Jean-Frederic Clere Master

    The certificat/key is valid from one CN you can't go around that... Well SSLProxyCheckPeerCN.

     

    Is it a permant or a interminant error?

  • 9. Re: Bad Gateway
    Steve Chernyak Newbie

    >The certificat/key is valid from one CN you can't go around that...

     

    I'd like apache to connect to jboss using the hostname. The certificate configured in jboss uses the hostname for the CN. The bind address on the connector for port 8443 specifies the hostname (that matches the CN). However, when the jboss node is added to the apache proxy, apache uses the ip address to connect back to jboss on port 8443. This causes a mismatch between ip and the CN. I'd like to force apache to use the hostname when proxying the requests.

     

    >Well SSLProxyCheckPeerCN

     

    I mistakenly assumed the cn check was off by default (seeing as how it works sometimes). I've turned it off now and will continue to monitor for 502 errors.

     

    >Is it a permant or a interminant error?

     

    It's not permanent. It seems to happen more often after an extended period of not being used (first time in the morning).

     

    Thanks

  • 10. Re: Bad Gateway
    wiktorowski maximilien Novice

    Hi Steve,

     

    I have the exact same problem. Intermittent 502 errors between apache and jboss AS 7.1.1.Final

     

     

    [Tue Apr 24 11:20:35 2012] [debug] ssl_engine_kernel.c(1881): OpenSSL: Read: SSLv2/v3 read server hello A

    [Tue Apr 24 11:20:35 2012] [debug] ssl_engine_kernel.c(1905): OpenSSL: Exit: error in SSLv2/v3 read server hello A

    [Tue Apr 24 11:20:35 2012] [info] [client 192.168.41.231] SSL Proxy connect failed

    [Tue Apr 24 11:20:35 2012] [info] SSL Library Error: 336032754 error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message

    [Tue Apr 24 11:20:35 2012] [info] [client 192.168.41.231] Connection closed to child 0 with abortive shutdown (server dev1.mycompany.com:443)

    [Tue Apr 24 11:20:35 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 192.168.41.231:8943 (192.168.41.231)

    [Tue Apr 24 11:20:35 2012] [error] proxy: pass request body failed to 192.168.41.231:8943 (192.168.41.231) from 90.82.78.215 ()

     

    Have you found a solution ?

     

    Best regards,

  • 11. Re: Bad Gateway
    Jean-Frederic Clere Master

    Any error message on AS7 side? what is in the web subsystem configuration?

  • 12. Re: Bad Gateway
    wiktorowski maximilien Novice

    Hi,

     

    I have no error on AS7 side.

     

    My web subsystem is

     

    <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

         <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">

              <ssl name="myssl" key-alias="1" password="xxxxx" certificate-key-file="C:/DigicashCA.p12" verify-client="want" verify-depth="1"

                                 ca-certificate-file="C:/DigicashCA.p12" ca-certificate-password="xxxxx" keystore-type="PKCS12" truststore-type="PKCS12"/>

         </connector>

         <virtual-server name="default-host" enable-welcome-root="false">

              <alias name="maximilien.digica.sh"/>

         </virtual-server>

    </subsystem>

     

    I'm also using proxy-list instead of advertising on modcluster.

     

    It works most of time but sometimes i got 502 errors. It seems to appear after a long time of inactivity.

     

    Best regards,

  • 13. Re: Bad Gateway
    Jean-Frederic Clere Master

    that is weird you have 2 problem:

    1 - httpd can't connect to the proxy like it speaks https to http back-end.

    2 - the client (browser) is already disconnected when http tries to send the response.

    Look to the access_log to see what was requested and check in the error_log that all the above error message are from the same request/response tuple.

  • 14. Re: Bad Gateway
    Darragh Sherwin Newbie

    I am getting similar intermittent issues with mod_cluster and ssl with AS 7.1.1

     

    The error in apache log is

    [Wed Jun 13 16:39:05 2012] [error] (502)Unknown error 502: proxy: pass request body failed to 10.152.20.99:31002 (10.152.20.99)

    [Wed Jun 13 16:39:05 2012] [error] [client 10.152.50.142] proxy: Error during SSL Handshake with remote server returned by /connect/com/company/mobile/api/auth/login

    [Wed Jun 13 16:39:05 2012] [error] proxy: pass request body failed to 10.152.20.99:31002 (10.152.20.99) from 10.152.50.142 ()

     

    My apache ssl config is

     

    AddType application/x-x509-ca-cert .crt

    AddType application/x-pkcs7-crl    .crl

    SSLPassPhraseDialog  builtin

    SSLSessionCache        "shmcb:/opt/jboss/httpd/httpd/logs/ssl_scache(512000)"

    SSLSessionCacheTimeout  36000

    SSLMutex  "file:/home/dev01/opt/jboss/httpd/httpd/logs/ssl_mutex"

     

    <VirtualHost myserver.mycompany.com:8090>

     

    DocumentRoot "/home/dev01/opt/jboss/httpd/htdocs/htdocs"

    ServerName myserver.mycompany.com:8090

    ServerAdmin you@example.com

    ErrorLog "/home/dev01/opt/jboss/httpd/httpd/logs/error_log"

    TransferLog "/home/dev01/opt/jboss/httpd/httpd/logs/access_log"

     

    SSLEngine on

            SSLProxyEngine on

            SSLProxyCheckPeerCN off

    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

     

    SSLCertificateFile "/home/dev01/opt/jboss/httpd/httpd/conf/fd-cert.pem"

    SSLCertificateKeyFile "/home/dev01/opt/jboss/httpd/httpd/conf/fd-key.pem"

     

    <FilesMatch "\.(cgi|shtml|phtml|php)$">

               SSLOptions +StdEnvVars

    </FilesMatch>

    <Directory "/home/dev01/opt/jboss/httpd/htdocs/cgi-bin">

    SSLOptions +StdEnvVars

    </Directory>

     

    BrowserMatch ".*MSIE.*" \

    nokeepalive ssl-unclean-shutdown \

                downgrade-1.0 force-response-1.0

     

    CustomLog "/home/dev01/opt/jboss/httpd/httpd/logs/ssl_request_log" \

    "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

     

    </VirtualHost>

     

     

    Web subsystem config for AS 7.1.1 is

     

    <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

                    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">

                        <ssl key-alias="0" password="FD-CERTS" certificate-key-file="${jboss.domain.config.dir}/fd-jboss.keystore" cipher-suite="ALL" protocol="SSL"/>

                    </connector>

                    <virtual-server name="default-host" enable-welcome-root="true">

                        <alias name="localhost"/>

                        <access-log/>

                    </virtual-server>

                </subsystem>

1 2 Previous Next