This section is work in progress; we strive to update the documentation as we make changes to the code.
The most important properties are described on the wiki. The idea is that users take one of the predefined configurations (shipped with JGroups) and make only minor changes to it.
For each protocol define:
Properties provided
Required services
Provided services
Behavior
Table 7.1. Properties
| Name | Description |
|---|---|
| bind_addr | The bind address which should be used by this transport. The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| bind_interface_str | The interface (NIC) which should be used by this transport |
| bind_port | The port to which the transport binds. Default of 0 binds to any (ephemeral) port |
| bundler_capacity | The max number of elements in a bundler if the bundler supports size limitations |
| bundler_type | The type of bundler used. Has to be "old" (default) or "new" |
| diagnostics_addr | Address for diagnostic probing. Default is 224.0.75.75 |
| diagnostics_port | Port for diagnostic probing. Default is 7500 |
| disable_loopback | |
| discard_incompatible_packets | Discard packets with a different version if true. Default is false |
| enable_bundling | Enable bundling of smaller messages into bigger ones. Default is true |
| enable_diagnostics | Switch to enable diagnostic probing. Default is true |
| enable_unicast_bundling | Enable bundling of smaller messages into bigger ones for unicast messages. Default is false |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| ip_mcast | Multicast toggle. If false multiple unicast datagrams are sent instead of one multicast. Default is true |
| ip_ttl | The time-to-live (TTL) for multicast datagram packets. Default is 8 |
| level | Sets the logger level (see javadocs) |
| log_discard_msgs | whether or not warnings about messages from different groups are logged |
| logical_addr_cache_expiration | Time (in ms) after which entries in the logical address cache marked as removable are removed |
| logical_addr_cache_max_size | Max number of elements in the logical address cache before eviction starts |
| loopback | Messages to self are looped back immediately if true |
| max_bundle_size | Maximum number of bytes for messages to be queued until they are sent |
| max_bundle_timeout | Max number of milliseconds until queued messages are sent |
| mcast_group_addr | The multicast address used for sending and receiving packets. Default is 228.8.8.8 |
| mcast_port | The multicast port used for sending and receiving packets. Default is 7600 |
| mcast_recv_buf_size | Receive buffer size of the multicast datagram socket. Default is 500'000 bytes |
| mcast_send_buf_size | Send buffer size of the multicast datagram socket. Default is 100'000 bytes |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| oob_thread_pool.keep_alive_time | Timeout in ms to remove idle threads from the OOB pool |
| oob_thread_pool.max_threads | Max thread pool size for the OOB thread pool |
| oob_thread_pool.min_threads | Minimum thread pool size for the OOB thread pool |
| oob_thread_pool_enabled | Switch for enabling thread pool for OOB messages. Default=true |
| oob_thread_pool_queue_enabled | Use queue to enqueue incoming OOB messages |
| oob_thread_pool_queue_max_size | Maximum queue size for incoming OOB messages. Default is 500 |
| oob_thread_pool_rejection_policy | Thread rejection policy. Possible values are Abort, Discard, DiscardOldest and Run. Default is Discard |
| port_range | The range of valid ports, from bind_port to end_port. Infinite if 0 |
| receive_interfaces | Comma delimited list of interfaces (IP addresses or interface names) to receive multicasts on |
| receive_on_all_interfaces | If true, the transport should use all available interfaces to receive multicast messages |
| singleton_name | If assigned enable this transport to be a singleton (shared) transport |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| thread_naming_pattern | Thread naming pattern for threads in this channel. Default is cl |
| thread_pool.keep_alive_time | Timeout in milliseconds to remove idle thread from regular pool |
| thread_pool.max_threads | Maximum thread pool size for the regular thread pool |
| thread_pool.min_threads | Minimum thread pool size for the regular thread pool |
| thread_pool_enabled | Switch for enabling thread pool for regular messages. Default true |
| thread_pool_queue_enabled | Use queue to enqueue incoming regular messages. Default is true |
| thread_pool_queue_max_size | Maximum queue size for incoming OOB messages. Default is 500 |
| thread_pool_rejection_policy | Thread rejection policy. Possible values are Abort, Discard, DiscardOldest and Run. Default is Discard |
| tick_time | Tick duration in the HashedTimingWheel timer. Only applicable if timer_type is "wheel" |
| timer.keep_alive_time | Timeout in ms to remove idle threads from the timer pool |
| timer.max_threads | Max thread pool size for the timer thread pool |
| timer.min_threads | Minimum thread pool size for the timer thread pool |
| timer_queue_max_size | Max number of elements on a timer queue |
| timer_type | Type of timer to be used. Valid values are "old" (DefaultTimeScheduler, used up to 2.10), "new" (TimeScheduler2) and "wheel". Note that this property might disappear in future releases, if one of the 3 timers is chosen as default timer |
| tos | Traffic class for sending unicast and multicast datagrams. Default is 8 |
| ucast_recv_buf_size | Receive buffer size of the unicast datagram socket. Default is 64'000 bytes |
| ucast_send_buf_size | Send buffer size of the unicast datagram socket. Default is 100'000 bytes |
| wheel_size | Number of ticks in the HashedTimingWheel timer. Only applicable if timer_type is "wheel" |
Table 7.2. Properties
| Name | Description |
|---|---|
| bind_addr | The bind address which should be used by this transport. The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| bind_interface_str | The interface (NIC) which should be used by this transport |
| bind_port | The port to which the transport binds. Default of 0 binds to any (ephemeral) port |
| bundler_capacity | The max number of elements in a bundler if the bundler supports size limitations |
| bundler_type | The type of bundler used. Has to be "old" (default) or "new" |
| conn_expire_time | Max time connection can be idle before being reaped (in ms) |
| diagnostics_addr | Address for diagnostic probing. Default is 224.0.75.75 |
| diagnostics_port | Port for diagnostic probing. Default is 7500 |
| discard_incompatible_packets | Discard packets with a different version if true. Default is false |
| enable_bundling | Enable bundling of smaller messages into bigger ones. Default is true |
| enable_diagnostics | Switch to enable diagnostic probing. Default is true |
| enable_unicast_bundling | Enable bundling of smaller messages into bigger ones for unicast messages. Default is false |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| external_addr | Use "external_addr" if you have hosts on different networks, behind firewalls. On each firewall, set up a port forwarding rule (sometimes called "virtual server") to the local IP (e.g. 192.168.1.100) of the host then on each host, set "external_addr" TCP transport parameter to the external (public IP) address of the firewall. |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| linger | SO_LINGER in msec. Default of -1 disables it |
| log_discard_msgs | whether or not warnings about messages from different groups are logged |
| logical_addr_cache_expiration | Time (in ms) after which entries in the logical address cache marked as removable are removed |
| logical_addr_cache_max_size | Max number of elements in the logical address cache before eviction starts |
| loopback | Messages to self are looped back immediately if true |
| max_bundle_size | Maximum number of bytes for messages to be queued until they are sent |
| max_bundle_timeout | Max number of milliseconds until queued messages are sent |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| oob_thread_pool.keep_alive_time | Timeout in ms to remove idle threads from the OOB pool |
| oob_thread_pool.max_threads | Max thread pool size for the OOB thread pool |
| oob_thread_pool.min_threads | Minimum thread pool size for the OOB thread pool |
| oob_thread_pool_enabled | Switch for enabling thread pool for OOB messages. Default=true |
| oob_thread_pool_queue_enabled | Use queue to enqueue incoming OOB messages |
| oob_thread_pool_queue_max_size | Maximum queue size for incoming OOB messages. Default is 500 |
| oob_thread_pool_rejection_policy | Thread rejection policy. Possible values are Abort, Discard, DiscardOldest and Run. Default is Discard |
| peer_addr_read_timeout | Max time to block on reading of peer address |
| port_range | The range of valid ports, from bind_port to end_port. Infinite if 0 |
| reaper_interval | Reaper interval in msec. Default is 0 (no reaping) |
| receive_interfaces | Comma delimited list of interfaces (IP addresses or interface names) to receive multicasts on |
| receive_on_all_interfaces | If true, the transport should use all available interfaces to receive multicast messages |
| recv_buf_size | Receiver buffer size in bytes |
| send_buf_size | Send buffer size in bytes |
| send_queue_size | Max number of messages in a send queue |
| singleton_name | If assigned enable this transport to be a singleton (shared) transport |
| sock_conn_timeout | Max time allowed for a socket creation in connection table |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| tcp_nodelay | Should TCP no delay flag be turned on |
| thread_naming_pattern | Thread naming pattern for threads in this channel. Default is cl |
| thread_pool.keep_alive_time | Timeout in milliseconds to remove idle thread from regular pool |
| thread_pool.max_threads | Maximum thread pool size for the regular thread pool |
| thread_pool.min_threads | Minimum thread pool size for the regular thread pool |
| thread_pool_enabled | Switch for enabling thread pool for regular messages. Default true |
| thread_pool_queue_enabled | Use queue to enqueue incoming regular messages. Default is true |
| thread_pool_queue_max_size | Maximum queue size for incoming OOB messages. Default is 500 |
| thread_pool_rejection_policy | Thread rejection policy. Possible values are Abort, Discard, DiscardOldest and Run. Default is Discard |
| tick_time | Tick duration in the HashedTimingWheel timer. Only applicable if timer_type is "wheel" |
| timer.keep_alive_time | Timeout in ms to remove idle threads from the timer pool |
| timer.max_threads | Max thread pool size for the timer thread pool |
| timer.min_threads | Minimum thread pool size for the timer thread pool |
| timer_queue_max_size | Max number of elements on a timer queue |
| timer_type | Type of timer to be used. Valid values are "old" (DefaultTimeScheduler, used up to 2.10), "new" (TimeScheduler2) and "wheel". Note that this property might disappear in future releases, if one of the 3 timers is chosen as default timer |
| use_send_queues | Should separate send queues be used for each connection |
| wheel_size | Number of ticks in the HashedTimingWheel timer. Only applicable if timer_type is "wheel" |
Table 7.3. Properties (experimental)
| Name | Description |
|---|---|
| bind_addr | The bind address which should be used by this transport. The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| bind_interface_str | The interface (NIC) which should be used by this transport |
| bind_port | The port to which the transport binds. Default of 0 binds to any (ephemeral) port |
| bundler_capacity | The max number of elements in a bundler if the bundler supports size limitations |
| bundler_type | The type of bundler used. Has to be "old" (default) or "new" |
| diagnostics_addr | Address for diagnostic probing. Default is 224.0.75.75 |
| diagnostics_port | Port for diagnostic probing. Default is 7500 |
| discard_incompatible_packets | Discard packets with a different version if true. Default is false |
| enable_bundling | Enable bundling of smaller messages into bigger ones. Default is true |
| enable_diagnostics | Switch to enable diagnostic probing. Default is true |
| enable_unicast_bundling | Enable bundling of smaller messages into bigger ones for unicast messages. Default is false |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| log_discard_msgs | whether or not warnings about messages from different groups are logged |
| logical_addr_cache_expiration | Time (in ms) after which entries in the logical address cache marked as removable are removed |
| logical_addr_cache_max_size | Max number of elements in the logical address cache before eviction starts |
| loopback | Messages to self are looped back immediately if true |
| max_bundle_size | Maximum number of bytes for messages to be queued until they are sent |
| max_bundle_timeout | Max number of milliseconds until queued messages are sent |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| oob_thread_pool.keep_alive_time | Timeout in ms to remove idle threads from the OOB pool |
| oob_thread_pool.max_threads | Max thread pool size for the OOB thread pool |
| oob_thread_pool.min_threads | Minimum thread pool size for the OOB thread pool |
| oob_thread_pool_enabled | Switch for enabling thread pool for OOB messages. Default=true |
| oob_thread_pool_queue_enabled | Use queue to enqueue incoming OOB messages |
| oob_thread_pool_queue_max_size | Maximum queue size for incoming OOB messages. Default is 500 |
| oob_thread_pool_rejection_policy | Thread rejection policy. Possible values are Abort, Discard, DiscardOldest and Run. Default is Discard |
| port_range | The range of valid ports, from bind_port to end_port. Infinite if 0 |
| receive_interfaces | Comma delimited list of interfaces (IP addresses or interface names) to receive multicasts on |
| receive_on_all_interfaces | If true, the transport should use all available interfaces to receive multicast messages |
| reconnect_interval | Interval in msec to attempt connecting back to router in case of torn connection. Default is 5000 msec |
| router_host | Router host address |
| router_port | Router port |
| singleton_name | If assigned enable this transport to be a singleton (shared) transport |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| tcp_nodelay | Should TCP no delay flag be turned on |
| thread_naming_pattern | Thread naming pattern for threads in this channel. Default is cl |
| thread_pool.keep_alive_time | Timeout in milliseconds to remove idle thread from regular pool |
| thread_pool.max_threads | Maximum thread pool size for the regular thread pool |
| thread_pool.min_threads | Minimum thread pool size for the regular thread pool |
| thread_pool_enabled | Switch for enabling thread pool for regular messages. Default true |
| thread_pool_queue_enabled | Use queue to enqueue incoming regular messages. Default is true |
| thread_pool_queue_max_size | Maximum queue size for incoming OOB messages. Default is 500 |
| thread_pool_rejection_policy | Thread rejection policy. Possible values are Abort, Discard, DiscardOldest and Run. Default is Discard |
| tick_time | Tick duration in the HashedTimingWheel timer. Only applicable if timer_type is "wheel" |
| timer.keep_alive_time | Timeout in ms to remove idle threads from the timer pool |
| timer.max_threads | Max thread pool size for the timer thread pool |
| timer.min_threads | Minimum thread pool size for the timer thread pool |
| timer_queue_max_size | Max number of elements on a timer queue |
| timer_type | Type of timer to be used. Valid values are "old" (DefaultTimeScheduler, used up to 2.10), "new" (TimeScheduler2) and "wheel". Note that this property might disappear in future releases, if one of the 3 timers is chosen as default timer |
| wheel_size | Number of ticks in the HashedTimingWheel timer. Only applicable if timer_type is "wheel" |
Table 7.4. Properties
| Name | Description |
|---|---|
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| discovery_timeout | Time (in ms) to wait for our own discovery message to be received. 0 means don't wait. If the discovery message is not received within discovery_timeout ms, a warning will be logged |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
FILE_PING can be used instead of GossipRouter in cases where no external process is desired.
Table 7.5. Properties
| Name | Description |
|---|---|
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| discovery_timeout | Time (in ms) to wait for our own discovery message to be received. 0 means don't wait. If the discovery message is not received within discovery_timeout ms, a warning will be logged |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
JDBC_PING is an alternative to S3_PING by using Amazon RDS instead of S3.
Table 7.6. Properties
| Name | Description |
|---|---|
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| connection_driver | The JDBC connection driver name |
| connection_password | The JDBC connection password |
| connection_url | The JDBC connection URL |
| connection_username | The JDBC connection username |
| datasource_jndi_name | To use a DataSource registered in JNDI, specify the JNDI name here. This is an alternative to all connection_* configuration options: if this property is not empty, then all connection relatedproperties must be empty. |
| delete_single_sql | SQL used to delete a row. Customizable, but keep the order of parameters and pick compatible types: 1)Own Address, as String 2)Cluster name, as String |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| initialize_sql | If not empty, this SQL statement will be performed at startup.Customize it to create the needed table on those databases which permit table creation attempt without loosing data, such as PostgreSQL and MySQL (using IF NOT EXISTS). To allow for creation attempts, errors performing this statement will be loggedbut not considered fatal. To avoid any DDL operation, set this to an empty string. |
| insert_single_sql | SQL used to insert a new row. Customizable, but keep the order of parameters and pick compatible types: 1)Own Address, as String 2)Cluster name, as String 3)Serialized PingData as byte[] |
| interval | Interval (in milliseconds) at which the own Address is written. 0 disables it. |
| level | Sets the logger level (see javadocs) |
| location | The absolute path of the shared file |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| select_all_pingdata_sql | SQL used to fetch all node's PingData. Customizable, but keep the order of parameters and pick compatible types: only one parameter needed, String compatible, representing the Cluster name. Must return a byte[], the Serialized PingData as it was stored by the insert_single_sql statement |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
Table 7.7. Properties
| Name | Description |
|---|---|
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| initial_hosts | Comma delimited list of hosts to be contacted for initial membership |
| level | Sets the logger level (see javadocs) |
| max_dynamic_hosts | max number of hosts to keep beyond the ones in initial_hosts |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| port_range | Number of ports to be probed for initial membership. Default is 1 |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
Table 7.8. Properties
| Name | Description |
|---|---|
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| initial_hosts | Comma delimited list of hosts to be contacted for initial membership |
| level | Sets the logger level (see javadocs) |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| reconnect_interval | Interval (ms) by which a disconnected stub attempts to reconnect to the GossipRouter |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| sock_conn_timeout | Max time for socket creation. Default is 1000 msec |
| sock_read_timeout | Max time in milliseconds to block on a read. 0 blocks forever |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
Table 7.9. Properties
| Name | Description |
|---|---|
| bind_addr | Bind address for multicast socket. The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| bind_interface_str | The interface (NIC) which should be used by this transport |
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| discovery_timeout | Time (in ms) to wait for our own discovery message to be received. 0 means don't wait. If the discovery message is not received within discovery_timeout ms, a warning will be logged |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| ip_ttl | Time to live for discovery packets. Default is 8 |
| level | Sets the logger level (see javadocs) |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| mcast_addr | |
| mcast_port | Multicast port for discovery packets. Default is 7555 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| receive_interfaces | List of interfaces to receive multicasts on |
| receive_on_all_interfaces | If true, the transport should use all available interfaces to receive multicast messages. Default is false |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| send_interfaces | List of interfaces to send multicasts on |
| send_on_all_interfaces | Whether send messages are sent on all interfaces. Default is false |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
BPING uses UDP broadcasts to discover other nodes. The default broadcast address (dest) is 255.255.255.255, and should be replaced with a subnet specific broadcast, e.g. 192.168.1.255.
Table 7.10. Properties (experimental)
| Name | Description |
|---|---|
| bind_port | Port for discovery packets |
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| dest | Target address for broadcasts. This should be restricted to the local subnet, e.g. 192.168.1.255 |
| discovery_timeout | Time (in ms) to wait for our own discovery message to be received. 0 means don't wait. If the discovery message is not received within discovery_timeout ms, a warning will be logged |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| port_range | Sends discovery packets to ports 8555 to (8555+port_range) |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
S3_PING is primarily meant to be used on Amazon EC2 where multicast traffic is not allowed and no external process (GossipRouter) is desired. When Amazon RDS is preferred over S3, or if a shared database is used, an alternative is to use JDBC_PING.
Table 7.11. Properties (experimental)
| Name | Description |
|---|---|
| access_key | The access key to AWS (S3) |
| break_on_coord_rsp | Return from the discovery phase as soon as we have 1 coordinator response |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| interval | Interval (in milliseconds) at which the own Address is written. 0 disables it. |
| level | Sets the logger level (see javadocs) |
| location | The absolute path of the shared file |
| max_rank | Only members with a rank <= max_rank will send a discovery response. 1 means only the coordinator will reply. 0 disables this; everyone replies. JIRA: https://jira.jboss.org/browse/JGRP-1181 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_initial_members | Minimum number of initial members to get a response from. Default is 2 |
| num_initial_srv_members | Minimum number of server responses (PingData.isServer()=true). If this value is greater than 0, we'll ignore num_initial_members |
| num_ping_requests | Number of discovery requests to be sent distributed over timeout. Default is 2 |
| pre_signed_delete_url | When non-null, we use this pre-signed URL for DELETEs |
| pre_signed_put_url | When non-null, we use this pre-signed URL for PUTs |
| prefix | When non-null, we set location to prefix-UUID |
| return_entire_cache | Whether or not to return the entire logical-physical address cache mappings on a discovery request, or not. Default is false, except for TCPPING |
| secret_access_key | The secret access key to AWS (S3) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to wait for the initial members. Default is 3000 msec |
Table 7.12. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| inconsistent_view_threshold | Number of inconsistent views with only 1 coord after a MERGE event is sent up |
| level | Sets the logger level (see javadocs) |
| max_interval | Maximum time in ms between runs to discover other clusters |
| merge_fast | When receiving a multicast message, checks if the sender is member of the cluster. If not, initiates a merge |
| merge_fast_delay | The delay (in milliseconds) after which a merge fast execution is started |
| min_interval | Minimum time in msbetween runs to discover other clusters |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
Failure detection based on heartbeat messages. If reply is not received without timeout ms, max_tries times, a member is declared suspected, and will be excluded by GMS
Each member send a message containing a "FD" - HEARTBEAT header to its neighbor to the right (identified by the ping_dest address). The heartbeats are sent by the inner class Monitor. When the neighbor receives the HEARTBEAT, it replies with a message containing a "FD" - HEARTBEAT_ACK header. The first member watches for "FD" - HEARTBEAT_ACK replies from its neigbor. For each received reply, it resets the last_ack timestamp (sets it to current time) and num_tries counter (sets it to 0). The same Monitor instance that sends heartbeats whatches the difference between current time and last_ack. If this difference grows over timeout, the Monitor cycles several more times (until max_tries) is reached) and then sends a SUSPECT message for the neighbor's address. The SUSPECT message is sent down the stack, is addressed to all members, and is as a regular message with a FdHeader.SUSPECT header.
Table 7.13. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_tries | Number of times to send an are-you-alive message |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout to suspect a node P if neither a heartbeat nor data were received from P. Default is 3000 msec |
Failure detection based on simple heartbeat protocol. Every member periodically multicasts a heartbeat. Every member also maintains a table of all members (minus itself). When data or a heartbeat from P are received, we reset the timestamp for P to the current time. Periodically, we check for expired members, and suspect those.
Example: <FD_ALL interval="3000" timeout="10000"/>
In the example above, we send a heartbeat every 3 seconds and suspect members if we haven't received a heartbeat (or traffic) for more than 10 seconds. Note that since we check the timestamps every 'interval' milliseconds, we will suspect a member after roughly 4 * 3s == 12 seconds. If we set the timeout to 8500, then we would suspect a member after 3 * 3 secs == 9 seconds.
Table 7.14. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| interval | Interval in which a HEARTBEAT is sent to the cluster |
| level | Sets the logger level (see javadocs) |
| msg_counts_as_heartbeat | Treat messages received from members as heartbeats. Note that this means we're updating a value in a hashmap every time a message is passing up the stack through FD_ALL, which is costly. Default is false |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Timeout after which a node P is suspected if neither a heartbeat nor data were received from P |
Failure detection protocol based on a ring of TCP sockets created between group members. Each member in a group connects to its neighbor (last member connects to first) thus forming a ring. Member B is suspected when its neighbor A detects abnormally closed TCP socket (presumably due to a node B crash). However, if a member B is about to leave gracefully, it lets its neighbor A know, so that it does not become suspected.
If you are using a multi NIC machine note that JGroups versions prior to 2.2.8 have FD_SOCK implementation that does not assume this possibility. Therefore JVM can possibly select NIC unreachable to its neighbor and setup FD_SOCK server socket on it. Neighbor would be unable to connect to that server socket thus resulting in immediate suspecting of a member. Suspected member is kicked out of the group, tries to rejoin, and thus goes into join/leave loop. JGroups version 2.2.8 introduces srv_sock_bind_addr property so you can specify network interface where FD_SOCK TCP server socket should be bound. This network interface is most likely the same interface used for other JGroups traffic. JGroups versions 2.2.9 and newer consult bind.address system property or you can specify network interface directly as FD_SOCK bind_addr property.
Table 7.16. Properties
| Name | Description |
|---|---|
| bind_addr | The NIC on which the ServerSocket should listen on. The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| bind_interface_str | The interface (NIC) which should be used by this transport |
| client_bind_port | Start port for client socket. Default value of 0 picks a random port |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| get_cache_timeout | Timeout for getting socket cache from coordinator. Default is 1000 msec |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| keep_alive | Whether to use KEEP_ALIVE on the ping socket or not. Default is true |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_tries | Number of attempts coordinator is solicited for socket cache until we give up. Default is 3 |
| port_range | Number of ports to probe for start_port and client_bind_port |
| sock_conn_timeout | Max time in millis to wait for ping Socket.connect() to return |
| start_port | Start port for server socket. Default value of 0 picks a random port |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| suspect_msg_interval | Interval for broadcasting suspect messages. Default is 5000 msec |
Table 7.17. Properties
| Name | Description |
|---|---|
| bind_addr | Interface for ICMP pings. Used if use_icmp is true The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| bind_interface_str | The interface (NIC) which should be used by this transport |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_msgs | Number of verify heartbeats sent to a suspected member |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Number of millisecs to wait for a response from a suspected member |
| use_icmp | Use InetAddress.isReachable() to verify suspected member instead of regular messages |
NAKACK provides reliable delivery and FIFO (= First In First Out) properties for messages sent to all nodes in a cluster.
Reliable delivery means that no message sent by a sender will ever be lost, as all messages are numbered with sequence numbers (by sender) and retransmission requests are sent to the sender of a message[14] if that sequence number is not received.
FIFO order means that all messages from a given sender are received in exactly the order in which they were sent.
Table 7.18. Properties
| Name | Description |
|---|---|
| discard_delivered_msgs | Should messages delivered to application be discarded |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| exponential_backoff | The first value (in milliseconds) to use in the exponential backoff. Enabled if greater than 0. Default is 0 |
| gc_lag | Garbage collection lag |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| log_discard_msgs | discards warnings about promiscuous traffic |
| log_not_found_msgs | If true, trashes warnings about retransmission messages not found in the xmit_table (used for testing) |
| max_msg_batch_size | Max number of messages to be removed from a NakReceiverWindow. This property might get removed anytime, so don't use it ! |
| max_rebroadcast_timeout | Timeout to rebroadcast messages. Default is 2000 msec |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| print_stability_history_on_failed_xmit | Should stability history be printed if we fail in retransmission. Default is false |
| retransmit_timeouts | Timeout before requesting retransmissions. Default is 600, 1200, 2400, 4800 |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| use_mcast_xmit | Retransmit messages using multicast rather than unicast |
| use_mcast_xmit_req | Use a multicast to request retransmission of missing messages. Default is false |
| use_range_based_retransmitter | Whether to use the old retransmitter which retransmits individual messages or the new one which uses ranges of retransmitted messages. Default is true. Note that this property will be removed in 3.0; it is only used to switch back to the old (and proven) retransmitter mechanism if issues occur |
| use_stats_for_retransmission | Use statistics gathered from actual retransmission times to compute new retransmission times. Default is false |
| xmit_from_random_member | Ask a random member for retransmission of a missing message. Default is false |
| xmit_history_max_size | Size of retransmission history. Default is 50 entries |
| xmit_table_max_compaction_time | Number of milliseconds after which the matrix in the retransmission table is compacted (only for experts) |
| xmit_table_msgs_per_row | Number of elements of a row of the matrix in the retransmission table (only for experts). The capacity of the matrix is xmit_table_num_rows * xmit_table_msgs_per_row |
| xmit_table_num_rows | Number of rows of the matrix in the retransmission table (only for experts) |
| xmit_table_resize_factor | Resize factor of the matrix in the retransmission table (only for experts) |
UNICAST provides reliable delivery and FIFO (= First In First Out) properties for point-to-point messages between one sender and one receiver.
Reliable delivery means that no message sent by a sender will ever be lost, as all messages are numbered with sequence numbers (by sender) and retransmission requests are sent to the sender of a message[15] if that sequence number is not received.
FIFO order means that all messages from a given sender are received in exactly the order in which they were sent.
On top of a reliable transport, such as TCP, UNICAST is not really needed. However, concurrent delivery of messages from the same sender is prevented by UNICAST by acquiring a lock on the sender's retransmission table, so unless concurrent delivery is desired, UNICAST should not be removed from the stack even if TCP is used.
Table 7.19. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| loopback | Whether to loop back messages sent to self. Default is false |
| max_msg_batch_size | Max number of messages to be removed from the AckReceiverWindow. This property might get removed anytime, so don't use it ! |
| max_retransmit_time | Max number of milliseconds we try to retransmit a message to any given member. After that, the connection is removed. Any new connection to that member will start with seqno #1 again. 0 disables this |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
UNICAST2 provides lossless, ordered, communication between 2 members. Contrary to UNICAST, it uses negative acks (similar to NAKACK) rather than positive acks. This reduces the communication overhead required for sending an ack for every message.
Table 7.20. Properties (experimental)
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_bytes | Max number of bytes before a stability message is sent to the sender |
| max_msg_batch_size | Max number of messages to be removed from a NakReceiverWindow. This property might get removed anytime, so don't use it ! |
| max_retransmit_time | Max number of milliseconds we try to retransmit a message to any given member. After that, the connection is removed. Any new connection to that member will start with seqno #1 again. 0 disables this |
| max_stable_msgs | Max number of STABLE messages sent for the same highest_received seqno. A value < 1 is invalid |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stable_interval | Max number of milliseconds before a stability message is sent to the sender(s) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| use_range_based_retransmitter | Whether to use the old retransmitter which retransmits individual messages or the new one which uses ranges of retransmitted messages. Default is true. Note that this property will be removed in 3.0; it is only used to switch back to the old (and proven) retransmitter mechanism if issues occur |
| xmit_table_automatic_purging | If enabled, the removal of a message from the retransmission table causes an automatic purge (only for experts) |
| xmit_table_max_compaction_time | Number of milliseconds after which the matrix in the retransmission table is compacted (only for experts) |
| xmit_table_msgs_per_row | Number of elements of a row of the matrix in the retransmission table (only for experts). The capacity of the matrix is xmit_table_num_rows * xmit_table_msgs_per_row |
| xmit_table_num_rows | Number of rows of the matrix in the retransmission table (only for experts) |
| xmit_table_resize_factor | Resize factor of the matrix in the retransmission table (only for experts) |
Table 7.21. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| frag_size | The max number of bytes in a message. Larger messages will be fragmented. Default is 8192 bytes |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_retained_buffer | The max size in bytes for the byte array output buffer |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
SEQUENCER provider total order for multicast (=group) messages by forwarding messages to the current coordinator, which then sends the messages to the cluster on behalf of the original sender. Because it is always the same sender (whose messages are delivered in FIFO order), a global (or total) order is established.
Sending members add every forwarded message M to a buffer and remove M when they receive it. Should the current coordinator crash, all buffered messages are forwarded to the new coordinator.
Note that retransmissions go to the original sender, not to the coordinator.
Table 7.22. Properties (experimental)
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
Group membership takes care of joining new members, handling leave requests by existing members, and handling SUSPECT messages for crashed members, as emitted by failure detection protocols. The algorithm for joining a new member is essentially:
- loop
- find initial members (discovery)
- if no responses:
- become singleton group and break out of the loop
- else:
- determine the coordinator (oldest member) from the responses
- send JOIN request to coordinator
- wait for JOIN response
- if JOIN response received:
- install view and break out of the loop
- else
- sleep for 5 seconds and continue the loop
Table 7.23. Properties
| Name | Description |
|---|---|
| disable_initial_coord | If true this member can never become coordinator. Default is false |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| flushInvokerClass | |
| handle_concurrent_startup | Temporary switch. Default is true and should not be changed |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| join_timeout | Join timeout |
| leave_timeout | Leave timeout |
| level | Sets the logger level (see javadocs) |
| log_collect_msgs | Logs failures for collecting all view acks if true |
| max_bundling_time | Max view bundling timeout if view bundling is turned on. Default is 50 msec |
| merge_timeout | Timeout to complete merge |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_prev_mbrs | Max number of old members to keep in history. Default is 50 |
| print_local_addr | Print local address of this member after connect. Default is true |
| print_physical_addrs | Print physical address(es) on startup |
| resume_task_timeout | Timeout to resume ViewHandler. Default is 10000 msec |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| use_flush_if_present | Use flush for view changes. Default is true |
| view_ack_collection_timeout | Time in ms to wait for all VIEW acks (0 == wait forever. Default is 2000 msec |
| view_bundling | View bundling toggle |
Consider the following situation: a new member wants to join a group. The prodedure to do so is:
Multicast an (unreliable) discovery request (ping)
Wait for n responses or m milliseconds (whichever is first)
Every member responds with the address of the coordinator
If the initial responses are > 0: determine the coordinator and start the JOIN protocolg
If the initial response are 0: become coordinator, assuming that no one else is out there
However, the problem is that the initial mcast discovery request might get lost, e.g. when multiple members start at the same time, the outgoing network buffer might overflow, and the mcast packet might get dropped. Nobody receives it and thus the sender will not receive any responses, resulting in an initial membership of 0. This could result in multiple coordinators, and multiple subgroups forming. How can we overcome this problem ? There are 3 solutions:
Increase the timeout, or number of responses received. This will only help if the reason of the empty membership was a slow host. If the mcast packet was dropped, this solution won't help
Add the MERGE(2) protocol. This doesn't actually prevent multiple initial cordinators, but rectifies the problem by merging different subgroups back into one. Note that this involves state merging which needs to be done by the application.
(new) Prevent members from becoming coordinator on initial startup. This solution is applicable when we know which member is going to be the initial coordinator of a fresh group. We don't care about afterwards, then coordinatorship can migrate to another member. In this case, we configure the member that is always supposed to be started first with disable_initial_coord=false (the default) and all other members with disable_initial_coord=true.This works as described below.
When the initial membership is received, and is null, and the property disable_initial_coord is true, then we just continue in the loop and retry receving the initial membership (until it is non-null). If the property is false, we are allowed to become coordinator, and will do so. Note that - if a member is started as first member of a group - but its property is set to true, then it will loop until another member whose disable_initial_coord property is set to false, is started.
Table 7.24. Properties
| Name | Description |
|---|---|
| alias | Alias used for recovering the key. Change the default |
| asymAlgorithm | Cipher engine transformation for asymmetric algorithm. Default is RSA |
| asymInit | Initial public/private key length. Default is 512 |
| asymProvider | Cryptographic Service Provider. Default is Bouncy Castle Provider |
| encrypt_entire_message | |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| keyPassword | Password for recovering the key. Change the default |
| keyStoreName | File on classpath that contains keystore repository |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| storePassword | Password used to check the integrity/unlock the keystore. Change the default |
| symAlgorithm | Cipher engine transformation for symmetric algorithm. Default is AES |
| symInit | Initial key length for matching symmetric algorithm. Default is 128 |
| symProvider | Cryptographic Service Provider. Default is Bouncy Castle Provider |
In order to transfer application state to a joining member of a group pbcast.STATE_TRANSFER has to load entire state into memory and send it to a joining member. Major limitation of this approach is that the state transfer that is very large (>1Gb) would likely result in OutOfMemoryException. In order to alleviate this problem a new state transfer methodology, based on a streaming state transfer, was introduced in JGroups 2.4
Streaming state transfer supports both partial and full state transfer.
Streaming state transfer provides an InputStream to a state reader and an OutputStream to a state writer. OutputStream and InputStream abstractions enable state transfer in byte chunks thus resulting in smaller memory requirements. For example, if application state consists a huge DOM tree, whose aggregate size is 2GB (and which has partly been passivated to disk), then the state provider (ie. the coordinator) can simply iterate over the DOM tree (activating the parts which have been passivated out to disk), and write to the OutputStream as it traverses the tree. The state receiver will simply read from the InputStream and reconstruct the tree on its side, possibly again passivating parts to disk.
Rather than having to provide a 2GB byte[] buffer, streaming state transfer transfers the state in chunks of N bytes where N is user configurable.
Prior to 2.6.9 and 2.8 releases streaming state transfer relied exclusively on its own tcp sockets to transfer state between members. The downside of tcp socket approach is that it is not firewall friendly. If use_default_transport property of pbcast.STREAMING_STATE_TRANSFER is set to true streaming state transfer will use normal messages to transfer state. This approach besides being completely transparent to application is also firewall friendly. However, as expected, tcp sockets have better performance.
Streaming state transfer, just as regular byte based state transfer, can be used in both pull and push mode. Similarly to the current getState and setState methods of org.jgroups.MessageListener, application interested in streaming state transfer in a push mode would implement streaming getState method(s) by sending/writing state through a provided OutputStream reference and setState method(s) by receiving/reading state through a provided InputStream reference. In order to use streaming state transfer in a push mode, existing ExtendedMessageListener has been expanded to include additional four methods:
public interface ExtendedMessageListener
{
/*non-streaming callback methods ommitted for clarity*/
/**
* Allows an application to write a state through a provided OutputStream.
* An application is obligated to always close the given OutputStream reference.
*
* @param ostream the OutputStream
* @see OutputStream#close()
*/
public void getState(OutputStream ostream);
/**
* Allows an application to write a partial state through a provided OutputStream.
* An application is obligated to always close the given OutputStream reference.
*
* @param state_id id of the partial state requested
* @param ostream the OutputStream
*
* @see OutputStream#close()
*/
public void getState(String state_id, OutputStream ostream);
/**
* Allows an application to read a state through a provided InputStream.
* An application is obligated to always close the given InputStream reference.
*
* @param istream the InputStream
* @see InputStream#close()
*/
public void setState(InputStream istream);
/**
* Allows an application to read a partial state through a provided InputStream.
* An application is obligated to always close the given InputStream reference.
*
* @param state_id id of the partial state requested
* @param istream the InputStream
*
* @see InputStream#close()
*/
public void setState(String state_id, InputStream istream);
}
For a pull mode (when application uses channel.receive() to fetch events) two new event classes will be introduced:
StreamingGetStateEvent
StreamingSetStateEvent
These two events/classes are very similar to existing GetStateEvent and SetStateEvent but introduce a new field; StreamingGetStateEvent has an OutputStream and StreamingSetStateEvent has an InputStream.
The following code snippet demonstrates how to pull events from a channel, processing StreamingGetStateEvent and sending hypothetical state through a provided OutputStream reference. Handling of StreamingSetStateEvent is analogous to this example:
...
Object obj=channel.receive(0);
if(obj instanceof StreamingGetStateEvent) {
StreamingGetStateEvent evt=(StreamingGetStateEvent)obj;
OutputStream oos = null;
try {
oos = new ObjectOutputStream(evt.getArg());
oos.writeObject(state);
oos.flush();
} catch (Exception e) {}
finally{
try {
oos.close();
} catch (IOException e) {
System.err.println(e);
}
}
}
...
API that initiates state transfer on a JChannel level has the following methods:
public boolean getState(Address target,long timeout)throws
ChannelNotConnectedException,ChannelClosedException;
public boolean getState(Address target,String state_id,long timeout)throws
ChannelNotConnectedException,ChannelClosedException;
Introduction of STREAMING_STATE_TRANSFER does not change the current API.
State transfer type choice is static, implicit and mutually exclusive. JChannel cannot use both STREAMING_STATE_TRANSFER and STATE_TRANSFER in one JChannel configuration.
STREAMING_STATE_TRANSFER allows the following confguration parameters:
Table 7.25. Properties
| Name | Description |
|---|---|
| bind_addr | The interface (NIC) used to accept state requests. The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| bind_interface_str | The interface (NIC) which should be used by this transport |
| bind_port | The port listening for state requests. Default value of 0 binds to any (ephemeral) port |
| buffer_queue_size | If default transport is used the total state buffer size before state producer is blocked. Default is 81920 bytes |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_pool | Maximum number of pool threads serving state requests. Default is 5 |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| pool_thread_keep_alive | Keep alive for pool threads serving state requests. Default is 20000 msec |
| socket_buffer_size | Buffer size for state transfer. Default is 8192 bytes |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| use_default_transport | If true default transport is used for state transfer rather than seperate TCP sockets. Default is false |
Threading model used for state writing in a member providing state and state reading in a member receiving a state is tunable. For state provider thread pool is used to spawn threads providing state. Thus member providing state, in a push mode, will be able to concurrently serve N state requests where N is max_threads configuration parameter of the thread pool. If there are no further state transfer requests pool threads will be automatically reaped after configurable "pool_thread_keep_alive" timeout expires. For a channel operating in the push mode state reader channel can read state by piggybacking on jgroups protocol stack thread or optionally use a separate thread. State reader should use a separate thread if state reading is expensive (eg. large state, serialization) thus potentially affecting liveness of jgroups protocol thread. Since most state transfers are very short (<2-3 sec) by default we do not use a separate thread.
Flow control takes care of adjusting the rate of a message sender to the rate of the slowest receiver over time. If a sender continuously sends messages at a rate that is faster than the receiver(s), the receivers will either queue up messages, or the messages will get discarded by the receiver(s), triggering costly retransmissions. In addition, there is spurious traffic on the cluster, causing even more retransmissions.
Flow control throttles the sender so the receivers are not overrun with messages.
FC uses a credit based system, where each sender has max_credits credits and decrements them whenever a message is sent. The sender blocks when the credits fall below 0, and only resumes sending messages when it receives a replenishment message from the receivers.
The receivers maintain a table of credits for all senders and decrement the given sender's credits as well, when a message is received.
When a sender's credits drops below a threshold, the receiver will send a replenishment message to the sender. The threshold is defined by min_bytes or min_threshold.
Table 7.26. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| ignore_synchronous_response | Does not block a down message if it is a result of handling an up message in thesame thread. Fixes JGRP-928 |
| level | Sets the logger level (see javadocs) |
| max_block_time | Max time (in milliseconds) to block. Default is 5000 msec |
| max_block_times | Max times to block for the listed messages sizes (Message.getLength()). Example: "1000:10,5000:30,10000:500" |
| max_credits | Max number of bytes to send per receiver until an ack must be received to proceed. Default is 500000 bytes |
| min_credits | Computed as max_credits x min_theshold unless explicitly set |
| min_threshold | The threshold (as a percentage of max_credits) at which a receiver sends more credits to a sender. Example: if max_credits is 1'000'000, and min_threshold 0.25, then we send ca. 250'000 credits to P once we've received 250'000 bytes from P |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
A simplified version of FC. FC can actually still overrun receivers when the transport's latency is very small. SFC is a simple flow control protocol for group (= multipoint) messages.
Every sender has max_credits bytes for sending multicast messages to the group.
Every multicast message (we don't consider unicast messages) decrements max_credits by its size. When max_credits falls below 0, the sender asks all receivers for new credits and blocks until *all* credits have been received from all members.
When the receiver receives a credit request, it checks whether it has received max_credits bytes from the requester since the last credit request. If yes, it sends new credits to the requester and resets the max_credits for the requester. Else, it takes a note of the credit request from P and - when max_credits bytes have finally been received from P - it sends the credits to P and resets max_credits for P.
The maximum amount of memory for received messages is therefore <number of senders> * max_credits.
The relationship with STABLE is as follows: when a member Q is slow, it will prevent STABLE from collecting messages above the ones seen by Q (everybody else has seen more messages). However, because Q will *not* send credits back to the senders until it has processed all messages worth max_credits bytes, the senders will block. This in turn allows STABLE to progress and eventually garbage collect most messages from all senders. Therefore, SFC and STABLE complement each other, with SFC blocking senders so that STABLE can catch up.
SFC is currently experimental, we recommend to use MFC and UFC (see below) instead.
Table 7.27. Properties (experimental)
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_block_time | Max time (in milliseconds) to block. Default is 5000 msec |
| max_credits | Max number of bytes to send per receiver until an ack must be received to proceed. Default is 2000000 bytes |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
In 2.10, FC was separated into MFC (Multicast Flow Control) and Unicast Flow Control (UFC). The reason was that multicast flow control should not be impeded by unicast flow control, and vice versa. Also, performance for the separate implementations could be increased, plus they can be individually omitted. For example, if no unicast flow control is needed, UFC can be left out of the stack configuration.
Table 7.28. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| ignore_synchronous_response | Does not block a down message if it is a result of handling an up message in thesame thread. Fixes JGRP-928 |
| level | Sets the logger level (see javadocs) |
| max_block_time | Max time (in milliseconds) to block. Default is 5000 msec |
| max_block_times | Max times to block for the listed messages sizes (Message.getLength()). Example: "1000:10,5000:30,10000:500" |
| max_credits | Max number of bytes to send per receiver until an ack must be received to proceed |
| min_credits | Computed as max_credits x min_theshold unless explicitly set |
| min_threshold | The threshold (as a percentage of max_credits) at which a receiver sends more credits to a sender. Example: if max_credits is 1'000'000, and min_threshold 0.25, then we send ca. 250'000 credits to P once we've got only 250'000 credits left for P (we've received 750'000 bytes from P) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
Table 7.29. Properties
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| ignore_synchronous_response | Does not block a down message if it is a result of handling an up message in thesame thread. Fixes JGRP-928 |
| level | Sets the logger level (see javadocs) |
| max_block_time | Max time (in milliseconds) to block. Default is 5000 msec |
| max_block_times | Max times to block for the listed messages sizes (Message.getLength()). Example: "1000:10,5000:30,10000:500" |
| max_credits | Max number of bytes to send per receiver until an ack must be received to proceed |
| min_credits | Computed as max_credits x min_theshold unless explicitly set |
| min_threshold | The threshold (as a percentage of max_credits) at which a receiver sends more credits to a sender. Example: if max_credits is 1'000'000, and min_threshold 0.25, then we send ca. 250'000 credits to P once we've got only 250'000 credits left for P (we've received 750'000 bytes from P) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
To serve potential retransmission requests, a member has to store received messages until it is known that every member in the cluster has received them. Message stability for a given message M means that M has been seen by everyone in the cluster.
The stability protocol periodically (or when a certain number of bytes have been received) initiates a consensus protocol, which multicasts a stable message containing the highest message numbers for a given member. This is called a digest.
When everyone has received everybody else's stable messages, a digest is computed which consists of the minimum sequence numbers of all received digests so far. This is the stability vector, and contain only message sequence numbers that have been seen by everyone.
This stability vector is the broadcast to the group and everyone can remove messages from their retransmission tables whose sequence numbers are smaller than the ones received in the stability vector. These messages can then be garbage collected.
Table 7.30. Properties
| Name | Description |
|---|---|
| cap | Max percentage of the max heap (-Xmx) to be used for max_bytes. Only used if ergonomics is enabled. 0 disables setting max_bytes dynamically. |
| desired_avg_gossip | Average time to send a STABLE message. Default is 20000 msec |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_bytes | Maximum number of bytes received in all messages before sending a STABLE message is triggered .If ergonomics is enabled, this value is computed as max(MAX_HEAP * cap, N * max_bytes) where N = number of members |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stability_delay | Delay before stability message is sent. Default is 6000 msec |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
Table 7.31. Properties
| Name | Description |
|---|---|
| compression_level | Compression level 0-9 (0=no compression, 9=best compression). Default is 9 |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| min_size | Minimal payload size of a message (in bytes) for compression to kick in. Default is 500 bytes |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| pool_size | Number of inflaters/deflaters for concurrent processing. Default is 2 |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
Flushing forces group members to send all their pending messages prior to a certain event. The process of flushing acquiesces the cluster so that state transfer or a join can be done. It is also called the stop-the-world model as nobody will be able to send messages while a flush is in process. Flush is used:
State transfer
When a member requests state transfer it tells everyone to stop sending messages and waits for everyone's ack. Then it asks the application for its state and ships it back to the requester. After the requester has received and set the state successfully, the requester tells everyone to resume sending messages.
View changes (e.g.a join). Before installing a new view V2, flushing would ensure that all messages *sent* in the current view V1 are indeed *delivered* in V1, rather than in V2 (in all non-faulty members). This is essentially Virtual Synchrony.
FLUSH is designed as another protocol positioned just below the channel, e.g. above STATE_TRANSFER and FC. STATE_TRANSFER and GMS protocol request flush by sending a SUSPEND event up the stack, where it is handled by the FLUSH protcol. The SUSPEND_OK ack sent back by the FLUSH protocol let's the caller know that the flush has completed. When done (e.g. view was installed or state transferred), the protocol sends up a RESUME event, which will allow everyone in the cluster to resume sending.
Channel can be notified that FLUSH phase has been started by turning channel block option on. By default it is turned off. If channel blocking is turned on FLUSH notifies application layer that channel has been blocked by sending EVENT.BLOCK event. Channel responds by sending EVENT.BLOCK_OK event down to FLUSH protocol. We recommend turning on channel block notification only if channel is used in push mode. In push mode application that uses channel can perform block logic by implementing MembershipListener.block() callback method.
Table 7.32. Properties
| Name | Description |
|---|---|
| enable_reconciliation | Reconciliation phase toggle. Default is true |
| end_flush_timeout | Timeout to wait for UNBLOCK after STOP_FLUSH is issued. Default is 2000 msec |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| retry_timeout | Retry timeout after an unsuccessful attempt to quiet the cluster (first flush phase). Default is 3000 msec |
| start_flush_timeout | Timeout (per atttempt) to quiet the cluster during the first flush phase. Default is 2000 msec |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| timeout | Max time to keep channel blocked in flush. Default is 8000 msec |
As discussed in Section 5.4.4, “Scopes: concurrent message delivery for messages from the same sender”, the SCOPE protocol is used to deliver updates to different scopes concurrently. It has to be placed somewhere above UNICAST and NAKACK.
SCOPE has a separate thread pool. The reason why the default thread pool from the transport wasn't used is that the default thread pool has a different purpose. For example, it can use a queue to which all incoming messages are added, which would defy the purpose of concurrent delivery in SCOPE. As a matter of fact, using a queue would most likely delay messages get sent up into SCOPE !
Also, the default pool's rejection policy might not be "run", so the SCOPE implementation would have to catch rejection exceptions and engage in a retry protocol, which is complex and wastes resources.
The configuration of the thread pool is shown below. If you expect concurrent messages to N different scopes, then the max pool size would ideally be set to N. However, in most cases, this is not necessary as (a) the messages might not be to different scopes or (b) not all N scopes might get messages at the same time. So even if the max pool size is a bit smaller, the cost of this is slight delays, in the sense that a message for scope Y might wait until the thread processing message for scope X is available.
To remove unused scopes, an expiry policy is provided: expiration_time is the number of milliseconds after which an idle scope is removed. An idle scope is a scope which hasn't seen any messages for expiration_time milliseconds. The expiration_interval value defines the number of milliseconds at which the expiry task runs. Setting both values to 0 disables expiration; it would then have to be done manually (see Section 5.4.4, “Scopes: concurrent message delivery for messages from the same sender” for details).
Table 7.33. Properties (experimental)
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| expiration_interval | Interval in milliseconds at which the expiry task tries to remove expired scopes |
| expiration_time | Time in milliseconds after which an expired scope will get removed. An expired scope is one to which no messages have been added in max_expiration_time milliseconds. 0 never expires scopes |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| thread_naming_pattern | Thread naming pattern for threads in this channel. Default is cl |
| thread_pool.keep_alive_time | Timeout in milliseconds to remove idle thread from regular pool |
| thread_pool.max_threads | Maximum thread pool size for the regular thread pool |
| thread_pool.min_threads | Minimum thread pool size for the regular thread pool |
RELAY bridges traffic between seperate clusters, see Section 5.9, “Bridging between remote clusters” for details.
Table 7.34. Properties (experimental)
| Name | Description |
|---|---|
| bridge_name | Name of the bridge cluster |
| bridge_props | Properties of the bridge cluster (e.g. tcp.xml) |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| present_global_views | Drops views received from below and instead generates global views and passes them up. A global view consists of the local view and the remote view, ordered by view ID. If true, no protocolwhich requires (local) views can sit on top of RELAY |
| relay | If set to false, don't perform relaying. Used e.g. for backup clusters; unidirectional replication from one cluster to another, but not back. Can be changed at runtime |
| site | Description of the local cluster, e.g. "nyc". This is added to every address, so itshould be short. This is a mandatory property and must be set |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
STOMP is a JGroups protocol which implements the STOMP protocol. Currently (as of Nov 2010), transactions and acks are not implemented.
The location of a STOMP protocol in a stack is shown in Figure 7.1, “STOMP in a protocol stack”.
The STOMP protocol should be near the top of the stack.
A STOMP instance listens on a TCP socket for client connections. The port and bind address of the server socket can be defined via properties.
A client can send SUBSCRIBE commands for various destinations. When a SEND for a given destination is received, STOMP adds a header to the message and broadcasts it to all cluster nodes. Every node then in turn forwards the message to all of its connected clients which have subscribed to the same destination. When a destination is not given, STOMP simply forwards the message to all connected clients.
Traffic can be generated by clients and by servers. In the latter case, we could for example have code executing in the address space of a JGroups (server) node. In the former case, clients use the SEND command to send messages to a JGroups server and receive messages via the MESSAGE command. If there is code on the server which generates messages, it is important that both client and server code agree on a marshalling format, e.g. JSON, so that they understand each other's messages.
Clients can be written in any language, as long as they understand the STOMP protocol. Note that the JGroups STOMP protocol implementation sends additional information (e.g. INFO) to clients; non-JGroups STOMP clients should simply ignore them.
JGroups comes with a STOMP client (org.jgroups.client.StompConnection) and a demo (StompDraw). Both need to be started with the address and port of a JGroups cluster node. Once they have been started, the JGroups STOMP protocol will notify clients of cluster changes, which is needed so client can failover to another JGroups server node when a node is shut down. E.g. when a client connects to C, after connection, it'll get a list of endpoints (e.g. A,B,C,D). When C is terminated, or crashes, the client automatically reconnects to any of the remaining nodes, e.g. A, B, or D. When this happens, a client is also re-subscribed to the destinations it registered for.
The JGroups STOMP protocol can be used when we have clients, which are either not in the same network segment as the JGroups server nodes, or which don't want to become full-blown JGroups server nodes. Figure 7.2, “STOMP architecture” shows a typical setup.
There are 4 nodes in a cluster. Say the cluster is in a LAN, and communication is via IP multicasting (UDP as transport). We now have clients which do not want to be part of the cluster themselves, e.g. because they're in a different geographic location (and we don't want to switch the main cluster to TCP), or because clients are frequently started and stopped, and therefore the cost of startup and joining wouldn't be amortized over the lifetime of a client. Another reason could be that clients are written in a different language, or perhaps, we don't want a large cluster, which could be the case if we for example have 10 JGroups server nodes and 1000 clients connected to them.
In the example, we see 9 clients connected to every JGroups cluster node. If a client connected to node A sends a message to destination /topics/chat, then the message is multicast from node A to all other nodes (B, C and D). Every node then forwards the message to those clients which have previously subscribed to /topics/chat.
When node A crashes (or leaves) the JGroups STOMP clients (org.jgroups.client.StompConnection) simply pick another server node and connect to it.
The properties for STOMP are shown below:
Table 7.35. Properties (experimental)
| Name | Description |
|---|---|
| bind_addr | The bind address which should be used by the server socket. The following special values are also recognized: GLOBAL, SITE_LOCAL, LINK_LOCAL and NON_LOOPBACK |
| endpoint_addr | If set, then endpoint will be set to this address |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| exact_destination_match | If set to false, then a destination of /a/b match /a/b/c, a/b/d, a/b/c/d etc |
| forward_non_client_generated_msgs | Forward received messages which don't have a StompHeader to clients |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| port | Port on which the STOMP protocol listens for requests |
| send_info | If true, information such as a list of endpoints, or views, will be sent to all clients (via the INFO command). This allows for example intelligent clients to connect to a different server should a connection be closed. |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
The DAISYCHAIN protocol is discussed in Section 5.10, “Daisychaining”.
Table 7.36. Properties (experimental)
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| forward_queue_size | The number of messages in the forward queue. This queue is used to host messages that need to be forwarded by us on behalf of our neighbor |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| loopback | Loop back multicast messages |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| send_queue_size | The number of messages in the send queue. This queue is used to host messages that need to be sent |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
RATE_LIMITER can be used to set a limit on the data sent per time unit. When sending data, only max_bytes can be sent per time_period milliseconds. E.g. if max_bytes="50M" and time_period="1000", then a sender can only send 50MBytes / sec max.
Table 7.37. Properties (experimental)
| Name | Description |
|---|---|
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| max_bytes | Max number of bytes to be sent in time_period ms. Blocks the sender if exceeded until a new time period has started |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
| time_period | Number of milliseconds during which max_bytes bytes can be sent |
There are currently 2 locking protocols: org.jgroups.protocols.CENTRAL_LOCK and org.jgroups.protocols.PEER_LOCK.
CENTRAL_LOCK has the current coordinator of a cluster grants locks, so every node has to communicate with the coordinator to acquire or release a lock. Lock requests by different nodes for the same lock are processed in the order in which they are received.
A coordinator maintains a lock table. To prevent losing the knowledge of who holds which locks, the coordinator can push lock information to a number of backups defined by num_backups. If num_backups is 0, no replication of lock information happens. If num_backups is greater than 0, then the coordinator pushes information about acquired and released locks to all backup nodes. Topology changes might create new backup nodes, and lock information is pushed to those on becoming a new backup node.
The advantage of CENTRAL_LOCK is that all lock requests are granted in the same order across the cluster, which is not the case with PEER_LOCK.
Table 7.38. Properties (experimental)
| Name | Description |
|---|---|
| bypass_bundling | bypasses message bundling if set |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_backups | Number of backups to the coordinator. Server locks get replicated to these nodes as well |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
PEER_LOCK acquires a lock by contacting all cluster nodes, and lock acquisition is only successful if all non-faulty cluster nodes (peers) grant it.
Unless a total order configuration is used (e.g. org.jgroups.protocols.SEQUENCER based), lock requests for the same resource from different senders may be received in different order, so deadlocks can occur. Example:
To acquire a lock, we need lock grants from both A and B, but this will never happen here. To fix this, either add SEQUENCER to the configuration, so that all lock requests are received in the same global order at both A and B, or use java.util.concurrent.locks.Lock.tryLock(long,javaTimeUnit) with retries if a lock cannot be acquired.
Table 7.39. Properties (experimental)
| Name | Description |
|---|---|
| bypass_bundling | bypasses message bundling if set |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
CENTRAL_EXECUTOR is an implementation of Executing which is needed by the ExecutionService.
Table 7.40. Properties (experimental)
| Name | Description |
|---|---|
| bypass_bundling | bypasses message bundling if set |
| ergonomics | Enables ergonomics: dynamically find the best values for properties at runtime |
| id | Give the protocol a different ID if needed so we can have multiple instances of it in the same stack |
| level | Sets the logger level (see javadocs) |
| name | Give the protocol a different name if needed so we can have multiple instances of it in the same stack (also change ID) |
| num_backups | Number of backups to the coordinator. Queue State gets replicated to these nodes as well |
| stats | Determines whether to collect statistics (and expose them via JMX). Default is true |
[14] Note that NAKACK can also be configured to send retransmission requests for M to anyone in the cluster, rather than only to the sender of M.