[Top] [All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] [HELP!]GFS2 in the xen 4.1.2 does not work!

To: xen-users@xxxxxxxxxxxxx
From: 任我飞 <renwofei423@xxxxxxxxx>
Date: Wed, 7 Mar 2012 11:06:51 +0800
Delivery-date: Fri, 16 Mar 2012 15:35:10 +0000
List-id: Xen user discussion <xen-users.lists.xen.org>

Hardware environment

CPU: Intel E5410,8 core

MEM: 24G

Fibre disk array:IBM 5.7T (mkfs.gfs2 this)

3 NIC: 1000M

-------------------------------------------------------------

Software environment

2 nodes,hostname: vm1, vm2 .

Both have 2 kernels:

XEN:

Linux vm1 2.6.32.46-xen #2 SMP Mon Mar 5 16:05:01 CST 2012 x86_64 x86_64 x86_64 GNU/Linux

RHEL6.0:

Linux vm1 2.6.32-71.29.1.el6.x86_64 #1 SMP Thu May 19 13:15:41 CST 2011 x86_64 x86_64 x86_64 GNU/Linux

RHEL6.0 kernel just for test.

kernel's .config you can see Attachment.

XEN :4.1.2

ifconfig infomation:

eth0 Link encap:Ethernet HWaddr 00:1E:90:66:8E:F8

inet addr:172.19.0.81 Bcast:172.19.255.255 Mask:255.255.0.0

inet6 addr: fe80::21e:90ff:fe66:8ef8/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1124960 errors:0 dropped:0 overruns:0 frame:0

TX packets:618733 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:140496518 (133.9 MiB) TX bytes:115003293 (109.6 MiB)

Memory:fd780000-fd7a0000

eth1 Link encap:Ethernet HWaddr 00:1E:90:66:8E:F9

inet addr:172.19.0.91 Bcast:172.19.255.255 Mask:255.255.0.0

inet6 addr: fe80::21e:90ff:fe66:8ef9/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:527352 errors:0 dropped:0 overruns:0 frame:0

TX packets:27 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:34900470 (33.2 MiB) TX bytes:4015 (3.9 KiB)

Memory:fd7c0000-fd7e0000

eth2 Link encap:Ethernet HWaddr 00:1B:21:14:A0:21

inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.255.255.0

inet6 addr: fe80::21b:21ff:fe14:a021/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:527232 errors:0 dropped:0 overruns:0 frame:0

TX packets:27 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:34892627 (33.2 MiB) TX bytes:4000 (3.9 KiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:239 errors:0 dropped:0 overruns:0 frame:0

TX packets:239 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:36123 (35.2 KiB) TX bytes:36123 (35.2 KiB)

cat /etc/cluster/cluster.conf

<?xml version="1.0"?>

<fence_daemon post_fail_delay="0" post_join_delay="3"/>

<fence>

</method>

</fence>

</clusternode>

<fence>

</method>

</fence>

</clusternode>

</clusternodes>

</fencedevices>

<rm>

</rm>

<gfs_controld plock_rate_limit="0"/>

</cluster>

cat /etc/xen/xend-config.sxp

# -*- sh -*-

# Xend configuration file.

# This example configuration is appropriate for an installation that

# utilizes a bridged network configuration. Access to xend via http

# is disabled.

# Commented out entries show the default for that entry, unless otherwise

# specified.

#(logfile /var/log/xen/xend.log)

#(loglevel DEBUG)

# Uncomment the line below. Set the value to flask, acm, or dummy to

# select a security module.

#(xsm_module_name dummy)

# The Xen-API server configuration.

# This value configures the ports, interfaces, and access controls for the

# Xen-API server. Each entry in the list starts with either unix, a port

# number, or an address:port pair. If this is "unix", then a UDP socket is

# opened, and this entry applies to that. If it is a port, then Xend will

# listen on all interfaces on that TCP port, and if it is an address:port

# pair, then Xend will listen on the specified port, using the interface with

# the specified address.

# The subsequent string configures the user-based access control for the

# listener in question. This can be one of "none" or "pam", indicating either

# that users should be allowed access unconditionally, or that the local

# Pluggable Authentication Modules configuration should be used. If this

# string is missing or empty, then "pam" is used.

# The final string gives the host-based access control for that listener. If

# this is missing or empty, then all connections are accepted. Otherwise,

# this should be a space-separated sequence of regular expressions; any host

# with a fully-qualified domain name or an IP address that matches one of

# these regular expressions will be accepted.

# Example: listen on TCP port 9363 on all interfaces, accepting connections

# only from machines in example.com or localhost, and allow access through

# the unix domain socket unconditionally:

# (xen-api-server ((9363 pam '^localhost$ example\\.com$')

# (unix none)))

# Optionally, the TCP Xen-API server can use SSL by specifying the private

# key and certificate location:

# (9367 pam '' xen-api.key xen-api.crt)

# Default:

# (xen-api-server ((unix)))

(xend-http-server yes)

(xend-unix-server yes)

(xend-tcp-xmlrpc-server yes)

(xend-unix-xmlrpc-server yes)

#(xend-relocation-server no)

(xend-relocation-server yes)

#(xend-relocation-ssl-server no)

#(xend-udev-event-server no)

(xend-unix-path /var/lib/xend/xend-socket)

# Address and port xend should use for the legacy TCP XMLRPC interface,

# if xend-tcp-xmlrpc-server is set.

#(xend-tcp-xmlrpc-server-address 'localhost')

#(xend-tcp-xmlrpc-server-port 8006)

# SSL key and certificate to use for the legacy TCP XMLRPC interface.

# Setting these will mean that this port serves only SSL connections as

# opposed to plaintext ones.

#(xend-tcp-xmlrpc-server-ssl-key-file xmlrpc.key)

#(xend-tcp-xmlrpc-server-ssl-cert-file xmlrpc.crt)

# Port xend should use for the HTTP interface, if xend-http-server is set.

(xend-port 8000)

# Port xend should use for the relocation interface, if xend-relocation-server

# is set.

(xend-relocation-port 8002)

# Port xend should use for the ssl relocation interface, if

# xend-relocation-ssl-server is set.

#(xend-relocation-ssl-port 8003)

# SSL key and certificate to use for the ssl relocation interface, if

# xend-relocation-ssl-server is set.

#(xend-relocation-server-ssl-key-file xmlrpc.key)

#(xend-relocation-server-ssl-cert-file xmlrpc.crt)

# Whether to use ssl as default when relocating.

#(xend-relocation-ssl no)

# Address xend should listen on for HTTP connections, if xend-http-server is

# set.

# Specifying 'localhost' prevents remote connections.

# Specifying the empty string '' (the default) allows all connections.

(xend-address '')

#(xend-address localhost)

# Address xend should listen on for relocation-socket connections, if

# xend-relocation-server is set.

# Meaning and default as for xend-address above.

# Also, interface name is allowed (e.g. eth0) there to get the

# relocation address to be bound on.

(xend-relocation-address '')

# The hosts allowed to talk to the relocation port. If this is empty (the

# default), then all connections are allowed (assuming that the connection

# arrives on a port and interface on which we are listening; see

# xend-relocation-port and xend-relocation-address above). Otherwise, this

# should be a space-separated sequence of regular expressions. Any host with

# a fully-qualified domain name or an IP address that matches one of these

# regular expressions will be accepted.

# For example:

# (xend-relocation-hosts-allow '^localhost$ ^.*\\.example\\.org$')

(xend-relocation-server yes)

(xend-relocation-hosts-allow '')

#(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')

# The limit (in kilobytes) on the size of the console buffer

#(console-limit 1024)

# To bridge network traffic, like this:

# dom0: ----------------- bridge -> real eth0 -> the network

# |

# domU: fake eth0 -> vifN.0 -+

# use

# (network-script network-bridge)

# Your default ethernet device is used as the outgoing interface, by default.

# To use a different one (e.g. eth1) use

# (network-script 'network-bridge netdev=eth1')

# The bridge is named eth0, by default (yes, really!)

# It is normally much better to create the bridge yourself in

# /etc/network/interfaces. network-bridge start does nothing if you

# already have a bridge, and network-bridge stop does nothing if the

# default bridge name (normally eth0) is not a bridge. See

# bridge-utils-interfaces(5) for full information on the syntax in

# /etc/network/interfaces, but you probably want something like this:

# iface xenbr0 inet static

# address [etc]

# netmask [etc]

# [etc]

# bridge_ports eth0

# To have network-bridge create a differently-named bridge, use:

# (network-script 'network-bridge bridge=<name>')

# It is possible to use the network-bridge script in more complicated

# scenarios, such as having two outgoing interfaces, with two bridges, and

# two fake interfaces per guest domain. To do things like this, write

# yourself a wrapper script, and call network-bridge from it, as appropriate.

#(network-script network-bridge)

#(network-script 'network-bridge netdev=bond0')

#(network-script my-network-script)

(network-script 'network-bridge netdev=eth0')

#(network-script 'network-bridge netdev=eth1')

#(network-script 'network-bridge netdev=eth2')

#(network-script /bin/true)

#(network-script muti-2-network-script)

# The script used to control virtual interfaces. This can be overridden on a

# per-vif basis when creating a domain or a configuring a new vif. The

# vif-bridge script is designed for use with the network-bridge script, or

# similar configurations.

# If you have overridden the bridge name using

# (network-script 'network-bridge bridge=<name>') then you may wish to do the

# same here. The bridge name can also be set when creating a domain or

# configuring a new vif, but a value specified here would act as a default.

# If you are using only one bridge, the vif-bridge script will discover that,

# so there is no need to specify it explicitly. The default is to use

# the bridge which is listed first in the output from brctl.

(vif-script vif-bridge)

## Use the following if network traffic is routed, as an alternative to the

# settings for bridged networking given above.

#(network-script network-route)

#(vif-script vif-route)

## Use the following if network traffic is routed with NAT, as an alternative

# to the settings for bridged networking given above.

#(network-script network-nat)

#(vif-script vif-nat)

# dom0-min-mem is the lowest permissible memory level (in MB) for dom0.

# This is a minimum both for auto-ballooning (as enabled by

# enable-dom0-ballooning below) and for xm mem-set when applied to dom0.

(dom0-min-mem 196)

# Whether to enable auto-ballooning of dom0 to allow domUs to be created.

# If enable-dom0-ballooning = no, dom0 will never balloon out.

(enable-dom0-ballooning yes)

# 32-bit paravirtual domains can only consume physical

# memory below 168GB. On systems with memory beyond that address,

# they'll be confined to memory below 128GB.

# Using total_available_memory (in GB) to specify the amount of memory reserved

# in the memory pool exclusively for 32-bit paravirtual domains.

# Additionally you should use dom0_mem = <-Value> as a parameter in

# xen kernel to reserve the memory for 32-bit paravirtual domains, default

# is "0" (0GB).

(total_available_memory 0)

# In SMP system, dom0 will use dom0-cpus # of CPUS

# If dom0-cpus = 0, dom0 will take all cpus available

(dom0-cpus 0)

# Whether to enable core-dumps when domains crash.

#(enable-dump no)

# The tool used for initiating virtual TPM migration

#(external-migration-tool '')

# The interface for VNC servers to listen on. Defaults

# to 127.0.0.1 To restore old 'listen everywhere' behaviour

# set this to 0.0.0.0

#(vnc-listen '127.0.0.1')

# The default password for VNC console on HVM domain.

# Empty string is no authentication.

(vncpasswd '')

# The VNC server can be told to negotiate a TLS session

# to encryption all traffic, and provide x509 cert to

# clients enabling them to verify server identity. The

# GTK-VNC widget, virt-viewer, virt-manager and VeNCrypt

# all support the VNC extension for TLS used in QEMU. The

# TightVNC/RealVNC/UltraVNC clients do not.

# To enable this create x509 certificates / keys in the

# directory ${XEN_CONFIG_DIR} + vnc

# ca-cert.pem - The CA certificate

# server-cert.pem - The Server certificate signed by the CA

# server-key.pem - The server private key

# and then uncomment this next line

# (vnc-tls 1)

# The certificate dir can be pointed elsewhere..

# (vnc-x509-cert-dir vnc)

# The server can be told to request & validate an x509

# certificate from the client. Only clients with a cert

# signed by the trusted CA will be able to connect. This

# is more secure the password auth alone. Passwd auth can

# used at the same time if desired. To enable client cert

# checking uncomment this:

# (vnc-x509-verify 1)

# The default keymap to use for the VM's virtual keyboard

# when not specififed in VM's configuration

#(keymap 'en-us')

# Script to run when the label of a resource has changed.

#(resource-label-change-script '')

# Rotation count of qemu-dm log file.

#(qemu-dm-logrotate-count 10)

# Path where persistent domain configuration is stored.

# Default is /var/lib/xend/domains/

#(xend-domains-path /var/lib/xend/domains)

# Number of seconds xend will wait for device creation and

# destruction

#(device-create-timeout 100)

#(device-destroy-timeout 100)

# When assigning device to HVM guest, we use the strict check for HVM guest by

# default. (For PV guest, we use loose check automatically if necessary.)

# When we assign device to HVM guest, if we meet with the co-assignment

# issues or the ACS issue, we could try changing the option to 'no' -- however,

# we have to realize this may incur security issue and we can't make sure the

# device assignment could really work properly even after we do this.

#(pci-passthrough-strict-check yes)

# If we have a very big scsi device configuration, start of xend is slow,

# because xend scans all the device paths to build its internal PSCSI device

# list. If we need only a few devices for assigning to a guest, we can reduce

# the scan to this device. Set list list of device paths in same syntax like in

# command lsscsi, e.g. ('16:0:0:0' '15:0')

# (pscsi-device-mask ('*'))

------------------------------------------------------------------------------------------------------------

I made the GFS2 shared fiber disk array, but each time you run 1-2 hours, the directory corresponding to the GFS2 can not view, can not be accessed,But the two hosts is normal operation.View the log information, error messages as follows(/var/log/messages):

Mar 7 06:23:09 vm1 corosync[2152]: [QUORUM] Members[1]: 1

Mar 7 06:23:09 vm1 corosync[2152]: [TOTEM ] A processor joined or left the membership and a new membership was formed.

Mar 7 06:23:09 vm1 kernel: dlm: closing connection to node 2

Mar 7 06:23:09 vm1 corosync[2152]: [CPG ] downlist received left_list: 1

Mar 7 06:23:09 vm1 corosync[2152]: [CPG ] chosen downlist from node r(0) ip(172.19.0.81)

Mar 7 06:23:09 vm1 corosync[2152]: [MAIN ] Completed service synchronization, ready to provide service.

Mar 7 06:23:09 vm1 rgmanager[3108]: State change: vm2 DOWN

Mar 7 06:23:09 vm1 kernel: GFS2: fsid=gfscluster:data1.1: jid=0: Trying to acquire journal lock...

Mar 7 06:23:09 vm1 fenced[2206]: fencing node vm2

Mar 7 06:23:09 vm1 fenced[2206]: fence vm2 dev 0.0 agent fence_manual result: error from agent

Mar 7 06:23:09 vm1 fenced[2206]: fence vm2 failed

Mar 7 06:23:09 vm1 corosync[2152]: [TOTEM ] A processor joined or left the membership and a new membership was formed.

Mar 7 06:23:09 vm1 corosync[2152]: [QUORUM] Members[2]: 1 2

Mar 7 06:23:09 vm1 corosync[2152]: [CPG ] downlist received left_list: 0

Mar 7 06:23:09 vm1 corosync[2152]: [CPG ] chosen downlist from node r(0) ip(172.19.0.81)

Mar 7 06:23:09 vm1 corosync[2152]: [MAIN ] Completed service synchronization, ready to provide service.

Mar 7 06:23:09 vm1 gfs_controld[2247]: receive_start 2:4 add node with started_count 3

Mar 7 06:23:09 vm1 rgmanager[3108]: State change: vm2 UP

Mar 7 06:23:12 vm1 fenced[2206]: fencing node vm2

Mar 7 06:23:12 vm1 corosync[2152]: cman killed by node 2 because we were killed by cman_tool or other application

Mar 7 06:23:12 vm1 gfs_controld[2247]: daemon cpg_dispatch error 2

Mar 7 06:23:12 vm1 gfs_controld[2247]: cluster is down, exiting

Mar 7 06:23:12 vm1 rgmanager[3108]: #67: Shutting down uncleanly

Mar 7 06:23:12 vm1 dlm_controld[2222]: cluster is down, exiting

Mar 7 06:23:12 vm1 dlm_controld[2222]: daemon cpg_dispatch error 2

Mar 7 06:23:12 vm1 fenced[2206]: fence vm2 dev 0.0 agent none result: error from ccs

Mar 7 06:23:12 vm1 fenced[2206]: fence vm2 failed

Mar 7 06:23:20 vm1 kernel: dlm: closing connection to node 2

Mar 7 06:23:20 vm1 kernel: dlm: closing connection to node 1

Mar 7 06:25:14 vm1 kernel: INFO: task clvmd:2376 blocked for more than 120 seconds.

Mar 7 06:25:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Mar 7 06:25:14 vm1 kernel: clvmd D ffff8800280de180 0 2376 1 0x00000080

Mar 7 06:25:14 vm1 kernel: ffff8805c8413ce0 0000000000000282 0000000000000000 ffff8800280841e8

Mar 7 06:25:14 vm1 kernel: 00001e756d9fc134 ffff8805c318d460 0000000000000001 0000000101fc1a63

Mar 7 06:25:14 vm1 kernel: ffff8805c8411a10 ffff8805c8413fd8 000000000000fc08 ffff8805c8411a10

Mar 7 06:25:14 vm1 kernel: Call Trace:

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149ead6>] ? _spin_unlock_irqrestore+0x16/0x20

Mar 7 06:25:14 vm1 kernel: [<ffffffff8105d639>] ? try_to_wake_up+0x2f9/0x480

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010

Mar 7 06:25:14 vm1 kernel: [<ffffffffa0422fdd>] dlm_clear_proc_locks+0x3d/0x260 [dlm]

Mar 7 06:25:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10

Mar 7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149e9b0>] ? _spin_lock_irq+0x10/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffffa042e556>] device_close+0x66/0xc0 [dlm]

Mar 7 06:25:14 vm1 kernel: [<ffffffff811533a5>] __fput+0xf5/0x210

Mar 7 06:25:14 vm1 kernel: [<ffffffff811534e5>] fput+0x25/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffff8114eabd>] filp_close+0x5d/0x90

Mar 7 06:25:14 vm1 kernel: [<ffffffff8114eb8a>] sys_close+0x9a/0xf0

Mar 7 06:25:14 vm1 kernel: [<ffffffff810140f2>] system_call_fastpath+0x16/0x1b

Mar 7 06:25:14 vm1 kernel: INFO: task kslowd001:2523 blocked for more than 120 seconds.

Mar 7 06:25:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Mar 7 06:25:14 vm1 kernel: kslowd001 D ffff880028066180 0 2523 2 0x00000080

Mar 7 06:25:14 vm1 kernel: ffff8805c7ecf998 0000000000000246 0000210000002923 0001f5052c320000

Mar 7 06:25:14 vm1 kernel: 3200000000211000 211800000113052d 51052e3200000000 0000002120000000

Mar 7 06:25:14 vm1 kernel: ffff8805c7ec86b0 ffff8805c7ecffd8 000000000000fc08 ffff8805c7ec86b0

Mar 7 06:25:14 vm1 kernel: Call Trace:

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffffa04240d2>] dlm_lock+0x62/0x1e0 [dlm]

Mar 7 06:25:14 vm1 kernel: [<ffffffff81238c94>] ? vsnprintf+0x484/0x5f0

Mar 7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20

Mar 7 06:25:14 vm1 kernel: [<ffffffffa0471de2>] gdlm_lock+0xe2/0x120 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffffa0471ee0>] ? gdlm_ast+0x0/0xe0 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffffa0471e20>] ? gdlm_bast+0x0/0x50 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffffa0456ca3>] do_xmote+0x163/0x250 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffffa045715a>] run_queue+0xfa/0x170 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffffa045735a>] gfs2_glock_nq+0x11a/0x330 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffffa0457d79>] gfs2_glock_nq_num+0x69/0x90 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20

Mar 7 06:25:14 vm1 kernel: [<ffffffffa046a753>] gfs2_recover_work+0x93/0x7b0 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffff81010b0f>] ? xen_restore_fl_direct_end+0x0/0x1

Mar 7 06:25:14 vm1 kernel: [<ffffffff8100c676>] ? xen_mc_flush+0x106/0x250

Mar 7 06:25:14 vm1 kernel: [<ffffffff8100c2dd>] ? xen_write_cr0+0x4d/0xa0

Mar 7 06:25:14 vm1 kernel: [<ffffffff8100b4be>] ? xen_end_context_switch+0x1e/0x30

Mar 7 06:25:14 vm1 kernel: [<ffffffff81012776>] ? __switch_to+0x166/0x320

Mar 7 06:25:14 vm1 kernel: [<ffffffff810584b3>] ? finish_task_switch+0x53/0xe0

Mar 7 06:25:14 vm1 kernel: [<ffffffffa0457d71>] ? gfs2_glock_nq_num+0x61/0x90 [gfs2]

Mar 7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20

Mar 7 06:25:14 vm1 kernel: [<ffffffff810f87d2>] slow_work_execute+0x232/0x310

Mar 7 06:25:14 vm1 kernel: [<ffffffff810f8a07>] slow_work_thread+0x157/0x370

Mar 7 06:25:14 vm1 kernel: [<ffffffff8108deb0>] ? autoremove_wake_function+0x0/0x40

Mar 7 06:25:14 vm1 kernel: [<ffffffff810f88b0>] ? slow_work_thread+0x0/0x370

Mar 7 06:25:14 vm1 kernel: [<ffffffff8108db56>] kthread+0x96/0xa0

Mar 7 06:25:14 vm1 kernel: [<ffffffff8101514a>] child_rip+0xa/0x20

Mar 7 06:25:14 vm1 kernel: [<ffffffff81014311>] ? int_ret_from_sys_call+0x7/0x1b

Mar 7 06:25:14 vm1 kernel: [<ffffffff81014a9d>] ? retint_restore_args+0x5/0x6

Mar 7 06:25:14 vm1 kernel: [<ffffffff81015140>] ? child_rip+0x0/0x20

Mar 7 06:25:14 vm1 kernel: INFO: task rgmanager:3108 blocked for more than 120 seconds.

Mar 7 06:25:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Mar 7 06:25:14 vm1 kernel: rgmanager D ffff8800280a2180 0 3108 3106 0x00000080

Mar 7 06:25:14 vm1 kernel: ffff8805c65d9ce0 0000000000000286 ffff8805c65d9ca8 ffff8805c65d9ca4

Mar 7 06:25:14 vm1 kernel: ffffffff81b52fa8 ffff8805e6c24500 ffff880028066180 0000000101fc1a6d

Mar 7 06:25:14 vm1 kernel: ffff8805c8f2c6b0 ffff8805c65d9fd8 000000000000fc08 ffff8805c8f2c6b0