Mod cluster Installation: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(32 intermediate revisions by the same user not shown)
Line 60: Line 60:
==Insure that Module Dependencies are Present and Loaded==
==Insure that Module Dependencies are Present and Loaded==


mod_cluster needs mod_proxy to work. [[mod_cluster Concepts#Dependency_on_mod_proxy|More details about mod_cluster dependency on mod_proxy are found here]].  
mod_cluster needs mod_proxy to work. [[mod_cluster Concepts#mod_proxy|More details about mod_cluster dependency on mod_proxy are found here]].  


Make sure <tt>mod_proxy.so</tt> and <tt>mod_proxy_ajp.so</tt> are loaded. For a standard httpd installation, those modules are loaded in <tt>/etc/httpd/conf.modules.d/00-proxy.conf</tt>:
Make sure <tt>mod_proxy.so</tt> and <tt>mod_proxy_ajp.so</tt> are loaded. For a standard httpd installation, those modules are loaded in <tt>/etc/httpd/conf.modules.d/00-proxy.conf</tt>:
Line 95: Line 95:
         type httpd_t;
         type httpd_t;
         type unreserved_port_t;
         type unreserved_port_t;
         class tcp_socket name_bind;
         class tcp_socket { name_bind name_connect };
         class dir remove_name;
         class dir remove_name;
         class file { write unlink };
         class file { write unlink };
Line 103: Line 103:
#============= httpd_t ==============
#============= httpd_t ==============
allow httpd_t httpd_log_t:dir remove_name;
allow httpd_t httpd_log_t:dir remove_name;
allow httpd_t httpd_log_t:file unlink;
allow httpd_t httpd_log_t:file { write unlink };
allow httpd_t httpd_log_t:file write;
allow httpd_t unreserved_port_t:tcp_socket { name_bind name_connect };
allow httpd_t unreserved_port_t:udp_socket name_bind;
allow httpd_t unreserved_port_t:udp_socket name_bind;
allow httpd_t unreserved_port_t:tcp_socket name_bind;
</pre>
</pre>


Line 134: Line 133:
==Basic httpd.conf Configuration==
==Basic httpd.conf Configuration==


This is a simple configuration that can be used to conduct smoke tests. It consists in just one mod_cluster-enabled virtual host. Note that access is allowed to all, you may want to configure stricter security in production. Also note that /mod_cluster-manager access is enabled. More complex configurations are of course possible. For more details on how to configure mod_cluster, go to [[mod_cluster Configuration]].  
This is a simple httpd configuration that can be used to validate the mod_cluster installation.  
 
The configuration declares just one virtual node that gives access to one mod_cluster manager for one balancer. For a discussion on mod_cluster balancers, managers and virtual nodes, see [[mod_cluster Concepts]].
 
===Listen Directive===
 
Note that the manager is exposed on port 8088, so httpd needs to be made to listen on that port:


<pre>
<pre>
...
Listen ...
Listen 1.2.3.4:8088
</pre>
 
If the host that runs httpd has iptables enabled, you may want to open access to port 8088. Go here for more [[#iptables_Configuration|iptables configuration details]].
 
===Manager Virtual Host===
 
Declare the mod_cluster manager virtual host. It is critical to enable MCPM receive with <tt>EnableMCPMReceive</tt> otherwise nodes won't be able to register themselves with httpd.
 
Note that multicast advertising is turned off and the application server nodes will have to be explicitly configured to connect to this manager.
 
Also note that access is allowed to all, you may want to configure stricter access rules in production.
 
In case you need to troubleshoot mod_cluster problems, you may want to uncomment <tt>LogLevel debug</tt>.


<pre>
<VirtualHost 1.2.3.4:8088>
<VirtualHost 1.2.3.4:8088>


Line 146: Line 166:
         Allow from all
         Allow from all
     </Directory>
     </Directory>
    # this is critical - if not enabled, JBoss nodes won't be able to talk to the httpd node
    EnableMCPMReceive


     KeepAliveTimeout 60
     KeepAliveTimeout 60
Line 154: Line 171:


     ManagerBalancerName mycluster
     ManagerBalancerName mycluster
 
     EnableMCPMReceive
     # multicast advertising
    #ServerAdvertise On
    #AdvertiseGroup 224.0.1.105:23364
    #AdvertiseFrequency 5
 
    # JBoss nodes will connect to a fixed list of httpd hosts
     ServerAdvertise Off
     ServerAdvertise Off


Line 169: Line 180:


</VirtualHost>
</VirtualHost>
</pre>


...
===<Directory /> Permissions===
 
https://access.redhat.com/solutions/318283
 
httpd:
 
<pre>
[Sat Jan 09 19:12:10.564490 2016] [:debug] [pid 1715] mod_manager.c(2271): manager_trans INFO (/)
[Sat Jan 09 19:12:10.564590 2016] [authz_core:debug] [pid 1715] mod_authz_core.c(809): [client 172.20.2.43:34545] AH01626: authorization result of Require all denied: denied
[Sat Jan 09 19:12:10.564599 2016] [authz_core:debug] [pid 1715] mod_authz_core.c(809): [client 172.20.2.43:34545] AH01626: authorization result of <RequireAny>: denied
[Sat Jan 09 19:12:10.564621 2016] [authz_core:error] [pid 1715] [client 172.20.2.43:34545] AH01630: client denied by server configuration: /
</pre>
</pre>


client:


'''Important'''
<pre>
: the default httpd configuration restricts access to the root directory, so you may want to comment out the following. <font color=red>Can do better than that.</font>
19:12:20,564 ERROR [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000042: Error null sending INFO command to 172.20.2.41/172.20.2.41:8088, configuration will be reset: null
</pre>
 
<font color=red>
Temporarily circumvented by enabling access on the "/" directory as shown below. However, this is not a viable solution. Research the proper solution and move the above failures in a "Troubleshooting" section.


<pre>
<pre>
Line 182: Line 209:
#    Require all denied
#    Require all denied
#</Directory>
#</Directory>
<Directory />
    AllowOverride none
    Require all granted
</Directory>
</pre>
</pre>
</font>
===Further Configuration===


'''AdvertiseGroup'''
More complex configurations are of course possible. For more details on how to configure mod_cluster, go to [[mod_cluster Configuration]].
: The multicast address and port used by <tt>AdvertiseGroup</tt> must be synchronized with the values used by JBoss nodes. See [[mod_cluster mod_advertise Configuration#AdvertiseGroup|AdvertiseGroup]].


=Native Component Installation on Windows=
=Native Component Installation on Windows=


<font color=red>TODO, manual page 6, section 2.2.4 Install on Windows.</font>
mod_cluster Manual page 6, Section 2.2.4. Install on Windows.


=iptables Configuration=
=iptables Configuration=


If iptables is enabled, make sure the multicast, MCPM, HTTP and AJP traffic are allowed:
If iptables is enabled, make sure the multicast, MCPM, HTTP and AJP traffic are allowed on various mod_cluster hosts that need to talk to each other:


==MCPM==
==MCPM==
Line 201: Line 234:
<pre>
<pre>
-A INPUT -i eth0 -p tcp -m tcp --dport 8088 -m state --state NEW -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8088 -m state --state NEW -j ACCEPT
</pre>
==mod_cluster Manager Port==
If you chose to expose the mod_cluster manager on non-standard port, make sure the nodes have access to that port. You could open the port to everyone (non-recommended) or restrict access to that port only to the hosts that are supposed to be able to access it (recommended):
<pre>
-A INPUT -i eth0 -p tcp -m tcp --dport 8088 -m state --state NEW -s 172.20.0.0/16 -j ACCEPT
</pre>
</pre>


Line 222: Line 263:
More details on configuring iptables are available here: [[iptables Command Line Tool Examples#Allow_Multicast_Traffic_to_Come_In|iptables configuration examples]].
More details on configuring iptables are available here: [[iptables Command Line Tool Examples#Allow_Multicast_Traffic_to_Come_In|iptables configuration examples]].


=JBoss Components Installation=
=JBoss Component Installation=


<tt>standard-ha</tt> and <tt>standard-full-ha</tt> WildFly 9 profiles come with mod_cluster enabled.  
<tt>standard-ha</tt> and <tt>standard-full-ha</tt> WildFly 9 profiles come with mod_cluster enabled.  


Make sure the advertise multicast address and port as the same on the httpd nodes and JBoss nodes.
==Static Proxy List==


Also make sure multicast propagates between httpd and the JBoss nodes (see [[#Multicast_Traffic]]).
If the operational environment does not support multicast traffic, the JBoss nodes can be configured with a static httpd server list, as shown below. Unlike mod_jk, where the httpd configuration needs to be changed every time a new JBoss node is brought on-line, in mod_cluster's case JBoss nodes register themselves dynamically with httpd. This is a considerable advantage in environments with a large number of JBoss nodes.


==JBoss mod_cluster Configuration==
===WildFly 9===
 
<pre>
    ...
    <subsystem xmlns="urn:jboss:domain:modcluster:2.0">
        <mod-cluster-config proxies="mod-cluster-server-1" advertise="false" connector="ajp">
          <dynamic-load-provider>
              <load-metric type="cpu"/>
          </dynamic-load-provider>
      </mod-cluster-config>
    </subsystem>
    ...
    <socket-binding-group name="standard-sockets"...>
        ...
        <outbound-socket-binding name="mod-cluster-server-1">
            <remote-destination host="172.20.2.41" port="8088"/>
        </outbound-socket-binding>
        ...
    </socket-binding-group>
    ...
</pre>


===WildFly 8===
===WildFly 8===
<font color=red>TODO</font>
This example uses uses a static proxy server list, instead of multicast httpd server advertising.


<pre>
<pre>
         ...
         ...
         <subsystem xmlns="urn:jboss:domain:modcluster:1.2">
         <subsystem xmlns="urn:jboss:domain:modcluster:1.2">
             <mod-cluster-config proxy-list="10.10.10.10:6666"  advertise="false" connector="ajp">
             <mod-cluster-config proxy-list="1.2.3.4:8088"  advertise="false" connector="ajp">
                 <dynamic-load-provider>
                 <dynamic-load-provider>
                     <load-metric type="busyness"/>
                     <load-metric type="busyness"/>
Line 250: Line 307:
</pre>
</pre>


Dynamic:
===EAP 6.4===


<pre>
<pre>
         ...
         ...
         <subsystem xmlns="urn:jboss:domain:modcluster:2.0">
         <subsystem xmlns="urn:jboss:domain:modcluster:1.2">
             <mod-cluster-config advertise-socket="modcluster" connector="ajp">
             <mod-cluster-config proxy-list="1.2.3.4:8088"  advertise="false" connector="ajp">
                 <dynamic-load-provider>
                 <dynamic-load-provider>
                     <load-metric type="cpu"/>
                     <load-metric type="busyness"/>
                 </dynamic-load-provider>
                 </dynamic-load-provider>
             </mod-cluster-config>
             </mod-cluster-config>
Line 264: Line 321:
</pre>
</pre>


<font color=red>TODO: dynamic server advertising</font>
==Multicast Advertise==
 
===WildFly 9===
 
<font color=red>TODO</font>
 
This example uses uses a static proxy server list, instead of multicast httpd server advertising.
 
<pre>
    ...
 
    <subsystem xmlns="urn:jboss:domain:modcluster:2.0">
        <mod-cluster-config proxies="mod-cluster-server-1" advertise="false" connector="ajp">
          <dynamic-load-provider>
              <load-metric type="cpu"/>
          </dynamic-load-provider>
      </mod-cluster-config>
    </subsystem>
 
    ...


    <socket-binding-group name="standard-sockets"...>
See [[mod_cluster JBoss Configuration#Multicast_Advertise]]
 
        ...
        <outbound-socket-binding name="mod-cluster-server-1">
            <remote-destination host="172.20.2.41" port="8088"/>
        </outbound-socket-binding>
 
    </socket-binding-group>
 
    ...
</pre>
 
<font color=red>TODO: dynamic server advertising</font>


=Test Initial Installation=
=Test Initial Installation=
Line 305: Line 331:
mod_cluster will create a set of new files in (by default) <tt>/var/log/httpd</tt>: <tt>manager.*.slotmem</tt> and <tt>manager.*.lock</tt>.
mod_cluster will create a set of new files in (by default) <tt>/var/log/httpd</tt>: <tt>manager.*.slotmem</tt> and <tt>manager.*.lock</tt>.


Start all JBoss nodes.
JBoss nodes should start without warning or errors.
 
They should register with httpd servers they were configured with.
 
After JBoss nodes have started, the httpd manager console available at http://1.2.3.4:8088/mod_cluster-manager should look similar to the image below. Note that the node list should appear even if there are no deployed web applications.


Try http://httpd-virtual-host:httpd-virtual-post/mod_cluster-manager
[[Image:Mod cluster-manager.png|left|frame|Successful mod_cluster installation]]

Latest revision as of 19:16, 10 October 2016

External

Internal

Download

Compiled bundles are available here: http://mod-cluster.jboss.org/downloads

Native Components Installation on Linux

This procedure assumes httpd was already installed, configured, and it works well without mod_cluster. Details on httpd installation and configuration are available here: httpd installation, httpd configuration.

At the time of the writing, the pre-compiled native components to be installed within an existing httpd are published on the download page as "mod_cluster modules for httpd". For example, the mod_cluster binaries for Linux x86 httpd are listed as "linux-x86_64 mod_cluster binaries". The file name is mod_cluster-1.3.1.Final-linux2-x64-so.tar.gz. It contains the pre-compiled httpd dynamic libraries (mod_advertise.so, mod_cluster_slotmem.so, mod_manager.so and mod_proxy_cluster.so).

Place the Dynamic Libraries in the modules Directory

Identify the httpd instance's module directory, by looking at the existing LoadModule directives specified in the configuration. It is usually /etc/httpd/modules.

It is good practice to include the mod_cluster version in the name of the dynamic library files, to make it obvious what version is used.

cp mod_advertise.so /etc/httpd/modules/mod_advertise-1.3.1.Final.so 
cp mod_cluster_slotmem.so /etc/httpd/modules/mod_cluster_slotmem-1.3.1.Final.so 
cp mod_manager.so /etc/httpd/modules/mod_manager-1.3.1.Final.so 
cp mod_proxy_cluster.so /etc/httpd/modules/mod_proxy_cluster-1.3.1.Final.so 

Make sure the newly copied files have the same ownership and permissions as the existing modules.

Load Modules at Startup

A standard RHEL httpd installation has its modules configured in /etc/httpd/conf.modules.d. All files present in that directory are automatically included from httpd.conf:

...
Include conf.modules.d/*.conf
...

Follow the same pattern to configure mod_cluster.

Add a new 99-mod_cluster.conf file in /etc/httpd/conf.modules.d with the following content:

#
# This file configures mod_cluster
#

LoadModule proxy_cluster_module modules/mod_proxy_cluster-1.3.1.Final.so
LoadModule cluster_slotmem_module modules/mod_cluster_slotmem-1.3.1.Final.so
LoadModule manager_module modules/mod_manager-1.3.1.Final.so
LoadModule advertise_module modules/mod_advertise-1.3.1.Final.so

Insure that Module Dependencies are Present and Loaded

mod_cluster needs mod_proxy to work. More details about mod_cluster dependency on mod_proxy are found here.

Make sure mod_proxy.so and mod_proxy_ajp.so are loaded. For a standard httpd installation, those modules are loaded in /etc/httpd/conf.modules.d/00-proxy.conf:

...
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
...

Remove Undesired Modules

mod_cluster's mod_proxy_cluster cannot work correctly if mod_proxy's mod_proxy_balancer, so mod_proxy_balancer must be removed from the httpd configuration.

For a standard httpd deployment, mod_proxy_balancer is loaded in /etc/httpd/conf.modules.d/00-proxy.conf. Comment out the following line:

#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

SELinux Configuration

If SELinux it is being enforced, we will need to install additional policies to allow httpd to write mod_cluster-related files in /var/log/httpd, to perform multicast socket operations and bind on non-standard ports.

Start with a mod_cluster_1.te policy file:


module mod_cluster_1 1.0;

require {
        type httpd_log_t;
        type httpd_t;
        type unreserved_port_t;
        class tcp_socket { name_bind name_connect };
        class dir remove_name;
        class file { write unlink };
        class udp_socket name_bind;
}

#============= httpd_t ==============
allow httpd_t httpd_log_t:dir remove_name;
allow httpd_t httpd_log_t:file { write unlink };
allow httpd_t unreserved_port_t:tcp_socket { name_bind name_connect };
allow httpd_t unreserved_port_t:udp_socket name_bind;

Compile the policy:

checkmodule -M -m -o mod_cluster_1.mod mod_cluster_1.te

Create the module package:

semodule_package -o  mod_cluster_1.pp -m mod_cluster_1.mod 

Install the policy:

semodule -i mod_cluster_1.pp

Verify that the policy was installed:

semodule -l | grep mod_cluster_1

More details about managing SELinux policies are available here: SELinux Operations.

Basic httpd.conf Configuration

This is a simple httpd configuration that can be used to validate the mod_cluster installation.

The configuration declares just one virtual node that gives access to one mod_cluster manager for one balancer. For a discussion on mod_cluster balancers, managers and virtual nodes, see mod_cluster Concepts.

Listen Directive

Note that the manager is exposed on port 8088, so httpd needs to be made to listen on that port:

Listen ...
Listen 1.2.3.4:8088

If the host that runs httpd has iptables enabled, you may want to open access to port 8088. Go here for more iptables configuration details.

Manager Virtual Host

Declare the mod_cluster manager virtual host. It is critical to enable MCPM receive with EnableMCPMReceive otherwise nodes won't be able to register themselves with httpd.

Note that multicast advertising is turned off and the application server nodes will have to be explicitly configured to connect to this manager.

Also note that access is allowed to all, you may want to configure stricter access rules in production.

In case you need to troubleshoot mod_cluster problems, you may want to uncomment LogLevel debug.

<VirtualHost 1.2.3.4:8088>

    # LogLevel debug

    <Directory />
        Allow from all
    </Directory>

    KeepAliveTimeout 60
    MaxKeepAliveRequests 0

    ManagerBalancerName mycluster
    EnableMCPMReceive
    ServerAdvertise Off

    <Location /mod_cluster-manager>
        SetHandler mod_cluster-manager
        Allow from all
    </Location>

</VirtualHost>

<Directory /> Permissions

https://access.redhat.com/solutions/318283

httpd:

[Sat Jan 09 19:12:10.564490 2016] [:debug] [pid 1715] mod_manager.c(2271): manager_trans INFO (/)
[Sat Jan 09 19:12:10.564590 2016] [authz_core:debug] [pid 1715] mod_authz_core.c(809): [client 172.20.2.43:34545] AH01626: authorization result of Require all denied: denied
[Sat Jan 09 19:12:10.564599 2016] [authz_core:debug] [pid 1715] mod_authz_core.c(809): [client 172.20.2.43:34545] AH01626: authorization result of <RequireAny>: denied
[Sat Jan 09 19:12:10.564621 2016] [authz_core:error] [pid 1715] [client 172.20.2.43:34545] AH01630: client denied by server configuration: /

client:

19:12:20,564 ERROR [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000042: Error null sending INFO command to 172.20.2.41/172.20.2.41:8088, configuration will be reset: null

Temporarily circumvented by enabling access on the "/" directory as shown below. However, this is not a viable solution. Research the proper solution and move the above failures in a "Troubleshooting" section.

#<Directory />
#    AllowOverride none
#    Require all denied
#</Directory>
<Directory />
    AllowOverride none
    Require all granted
</Directory>

Further Configuration

More complex configurations are of course possible. For more details on how to configure mod_cluster, go to mod_cluster Configuration.

Native Component Installation on Windows

mod_cluster Manual page 6, Section 2.2.4. Install on Windows.

iptables Configuration

If iptables is enabled, make sure the multicast, MCPM, HTTP and AJP traffic are allowed on various mod_cluster hosts that need to talk to each other:

MCPM

On the httpd node, make sure MCPM inbound traffic is allowed:

-A INPUT -i eth0 -p tcp -m tcp --dport 8088 -m state --state NEW -j ACCEPT

mod_cluster Manager Port

If you chose to expose the mod_cluster manager on non-standard port, make sure the nodes have access to that port. You could open the port to everyone (non-recommended) or restrict access to that port only to the hosts that are supposed to be able to access it (recommended):

-A INPUT -i eth0 -p tcp -m tcp --dport 8088 -m state --state NEW -s 172.20.0.0/16 -j ACCEPT

HTTP and AJP Traffic

On the JBoss nodes, add this to /etc/sysconfig/iptables:

-A INPUT -i eth0 -p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8009 -m state --state NEW -j ACCEPT

Multicast Traffic

Multicast traffic should propagate from the httpd nodes to the JBoss node and maybe back? Are JBoss nodes multicasting to httpd? Research that.. Normally iptables is configured to leave all traffic out by default, so the httpd nodes should be fine. However, JBoss nodes must be enabled to allow the multicast traffic in, so add this to /etc/sysconfig/iptables:

-A INPUT -m pkttype --pkt-type multicast -j ACCEPT

More details on configuring iptables are available here: iptables configuration examples.

JBoss Component Installation

standard-ha and standard-full-ha WildFly 9 profiles come with mod_cluster enabled.

Static Proxy List

If the operational environment does not support multicast traffic, the JBoss nodes can be configured with a static httpd server list, as shown below. Unlike mod_jk, where the httpd configuration needs to be changed every time a new JBoss node is brought on-line, in mod_cluster's case JBoss nodes register themselves dynamically with httpd. This is a considerable advantage in environments with a large number of JBoss nodes.

WildFly 9

    ...
    <subsystem xmlns="urn:jboss:domain:modcluster:2.0">
        <mod-cluster-config proxies="mod-cluster-server-1" advertise="false" connector="ajp">
           <dynamic-load-provider>
              <load-metric type="cpu"/>
           </dynamic-load-provider>
       </mod-cluster-config>
    </subsystem>
     ...
    <socket-binding-group name="standard-sockets"...>
        ...
        <outbound-socket-binding name="mod-cluster-server-1">
            <remote-destination host="172.20.2.41" port="8088"/>
        </outbound-socket-binding>
        ...
    </socket-binding-group>
    ...

WildFly 8

        ...
        <subsystem xmlns="urn:jboss:domain:modcluster:1.2">
            <mod-cluster-config proxy-list="1.2.3.4:8088"  advertise="false" connector="ajp">
                <dynamic-load-provider>
                    <load-metric type="busyness"/>
                </dynamic-load-provider>
            </mod-cluster-config>
        </subsystem>
        ...

EAP 6.4

        ...
        <subsystem xmlns="urn:jboss:domain:modcluster:1.2">
            <mod-cluster-config proxy-list="1.2.3.4:8088"  advertise="false" connector="ajp">
                <dynamic-load-provider>
                    <load-metric type="busyness"/>
                </dynamic-load-provider>
            </mod-cluster-config>
        </subsystem>
        ...

Multicast Advertise

See mod_cluster JBoss Configuration#Multicast_Advertise

Test Initial Installation

Start httpd, it should start without warnings or errors.

mod_cluster will create a set of new files in (by default) /var/log/httpd: manager.*.slotmem and manager.*.lock.

JBoss nodes should start without warning or errors.

They should register with httpd servers they were configured with.

After JBoss nodes have started, the httpd manager console available at http://1.2.3.4:8088/mod_cluster-manager should look similar to the image below. Note that the node list should appear even if there are no deployed web applications.

Successful mod_cluster installation