[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] [HELP!]GFS2 in the xen 4.1.2 does not work!



I made the GFS2 shared fiber disk array, but each time you run 1-2 hours, the directory corresponding to the GFS2 can not view, can not be accessed,But the two hosts is normal operation.
Before running the OCFS2 problems, the replacement became the GFS2.
I replaced the Red Hat kernel, test a week no problem, the above problems using XEN kernel, how can I fix it? The cause of some kernel parameters?


2012/3/7 任我飞 <renwofei423@xxxxxxxxx>
Hardware environment
CPU: Intel E5410,8 core
MEM: 24G
Fibre disk array:IBM 5.7T (mkfs.gfs2 this)
3 NIC: 1000M


-------------------------------------------------------------
Software environment
2 nodes,hostname:   vm1,   vm2 .
Both have 2 kernels: 
XEN:
    Linux vm1 2.6.32.46-xen #2 SMP Mon Mar 5 16:05:01 CST 2012 x86_64 x86_64 x86_64 GNU/Linux
RHEL6.0:
    Linux vm1 2.6.32-71.29.1.el6.x86_64 #1 SMP Thu May 19 13:15:41 CST 2011 x86_64 x86_64 x86_64 GNU/Linux
RHEL6.0 kernel just for test.
kernel's .config you can see  Attachment.
XEN :4.1.2
ifconfig infomation:
eth0      Link encap:Ethernet  HWaddr 00:1E:90:66:8E:F8  
          inet addr:172.19.0.81  Bcast:172.19.255.255  Mask:255.255.0.0
          inet6 addr: fe80::21e:90ff:fe66:8ef8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1124960 errors:0 dropped:0 overruns:0 frame:0
          TX packets:618733 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:140496518 (133.9 MiB)  TX bytes:115003293 (109.6 MiB)
          Memory:fd780000-fd7a0000 

eth1      Link encap:Ethernet  HWaddr 00:1E:90:66:8E:F9  
          inet addr:172.19.0.91  Bcast:172.19.255.255  Mask:255.255.0.0
          inet6 addr: fe80::21e:90ff:fe66:8ef9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:527352 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:34900470 (33.2 MiB)  TX bytes:4015 (3.9 KiB)
          Memory:fd7c0000-fd7e0000 

eth2      Link encap:Ethernet  HWaddr 00:1B:21:14:A0:21  
          inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:fe14:a021/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:527232 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:34892627 (33.2 MiB)  TX bytes:4000 (3.9 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:239 errors:0 dropped:0 overruns:0 frame:0
          TX packets:239 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:36123 (35.2 KiB)  TX bytes:36123 (35.2 KiB)

cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="3" name="gfscluster">
<fence_daemon  post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="vm1" nodeid="1" votes="1">
<fence>
<method name="single" >
<device name="human" nodename="vm1" />
</method>
</fence>
</clusternode>
<clusternode name="vm2" nodeid="2" votes="1">
<fence>
<method name="single" >
<device name="human" nodename="vm2" />
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/> 
<fencedevices>
<fencedevice name="human" agent="fence_manual" />
</fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
        <dlm plock_ownership="1" plock_rate_limit="0"/>
        <gfs_controld plock_rate_limit="0"/>
</cluster>

cat /etc/xen/xend-config.sxp
# -*- sh -*-

#
# Xend configuration file.
#

# This example configuration is appropriate for an installation that 
# utilizes a bridged network configuration. Access to xend via http
# is disabled.  

# Commented out entries show the default for that entry, unless otherwise
# specified.

#(logfile /var/log/xen/xend.log)
#(loglevel DEBUG)

# Uncomment the line below.  Set the value to flask, acm, or dummy to 
# select a security module.

#(xsm_module_name dummy)

# The Xen-API server configuration.
#
# This value configures the ports, interfaces, and access controls for the
# Xen-API server.  Each entry in the list starts with either unix, a port
# number, or an address:port pair.  If this is "unix", then a UDP socket is
# opened, and this entry applies to that.  If it is a port, then Xend will
# listen on all interfaces on that TCP port, and if it is an address:port
# pair, then Xend will listen on the specified port, using the interface with
# the specified address.
#
# The subsequent string configures the user-based access control for the
# listener in question.  This can be one of "none" or "pam", indicating either
# that users should be allowed access unconditionally, or that the local
# Pluggable Authentication Modules configuration should be used.  If this
# string is missing or empty, then "pam" is used.
#
# The final string gives the host-based access control for that listener. If
# this is missing or empty, then all connections are accepted.  Otherwise,
# this should be a space-separated sequence of regular expressions; any host
# with a fully-qualified domain name or an IP address that matches one of
# these regular expressions will be accepted.
#
# Example: listen on TCP port 9363 on all interfaces, accepting connections
# only from machines in example.com or localhost, and allow access through
# the unix domain socket unconditionally:
#
#   (xen-api-server ((9363 pam '^localhost$ example\\.com$')
#                    (unix none)))
#
# Optionally, the TCP Xen-API server can use SSL by specifying the private
# key and certificate location:
#
#                    (9367 pam '' xen-api.key xen-api.crt)
#
# Default:
#   (xen-api-server ((unix)))


(xend-http-server yes)
(xend-unix-server yes)
(xend-tcp-xmlrpc-server yes)
(xend-unix-xmlrpc-server yes)
#(xend-relocation-server no)
(xend-relocation-server yes)
#(xend-relocation-ssl-server no)
#(xend-udev-event-server no)

(xend-unix-path /var/lib/xend/xend-socket)


# Address and port xend should use for the legacy TCP XMLRPC interface, 
# if xend-tcp-xmlrpc-server is set.
#(xend-tcp-xmlrpc-server-address 'localhost')
#(xend-tcp-xmlrpc-server-port 8006)

# SSL key and certificate to use for the legacy TCP XMLRPC interface.
# Setting these will mean that this port serves only SSL connections as
# opposed to plaintext ones.
#(xend-tcp-xmlrpc-server-ssl-key-file  xmlrpc.key)
#(xend-tcp-xmlrpc-server-ssl-cert-file xmlrpc.crt)


# Port xend should use for the HTTP interface, if xend-http-server is set.
(xend-port            8000)

# Port xend should use for the relocation interface, if xend-relocation-server
# is set.
(xend-relocation-port 8002)

# Port xend should use for the ssl relocation interface, if
# xend-relocation-ssl-server is set.
#(xend-relocation-ssl-port 8003)

# SSL key and certificate to use for the ssl relocation interface, if
# xend-relocation-ssl-server is set.
#(xend-relocation-server-ssl-key-file   xmlrpc.key)
#(xend-relocation-server-ssl-cert-file  xmlrpc.crt)

# Whether to use ssl as default when relocating.
#(xend-relocation-ssl no)

# Address xend should listen on for HTTP connections, if xend-http-server is
# set.
# Specifying 'localhost' prevents remote connections.
# Specifying the empty string '' (the default) allows all connections.
(xend-address '')
#(xend-address localhost)

# Address xend should listen on for relocation-socket connections, if
# xend-relocation-server is set.
# Meaning and default as for xend-address above.
# Also, interface name is allowed (e.g. eth0) there to get the
# relocation address to be bound on.
(xend-relocation-address '')

# The hosts allowed to talk to the relocation port.  If this is empty (the
# default), then all connections are allowed (assuming that the connection
# arrives on a port and interface on which we are listening; see
# xend-relocation-port and xend-relocation-address above).  Otherwise, this
# should be a space-separated sequence of regular expressions.  Any host with
# a fully-qualified domain name or an IP address that matches one of these
# regular expressions will be accepted.
#
# For example:
#  (xend-relocation-hosts-allow '^localhost$ ^.*\\.example\\.org$')
(xend-relocation-server yes)
(xend-relocation-hosts-allow '')
#(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')

# The limit (in kilobytes) on the size of the console buffer
#(console-limit 1024)

##
# To bridge network traffic, like this:
#
# dom0: ----------------- bridge -> real eth0 -> the network
#                            |
# domU: fake eth0 -> vifN.0 -+
#
# use
#
# (network-script network-bridge)
#
# Your default ethernet device is used as the outgoing interface, by default. 
# To use a different one (e.g. eth1) use
#
# (network-script 'network-bridge netdev=eth1')
#
# The bridge is named eth0, by default (yes, really!)
#

# It is normally much better to create the bridge yourself in
# /etc/network/interfaces.  network-bridge start does nothing if you
# already have a bridge, and network-bridge stop does nothing if the
# default bridge name (normally eth0) is not a bridge.  See
# bridge-utils-interfaces(5) for full information on the syntax in
# /etc/network/interfaces, but you probably want something like this:
#    iface xenbr0 inet static
#        address [etc]
#        netmask [etc]
#        [etc]
#        bridge_ports eth0
#
# To have network-bridge create a differently-named bridge, use:
# (network-script 'network-bridge bridge=<name>')
#
# It is possible to use the network-bridge script in more complicated
# scenarios, such as having two outgoing interfaces, with two bridges, and
# two fake interfaces per guest domain.  To do things like this, write
# yourself a wrapper script, and call network-bridge from it, as appropriate.
#
#(network-script network-bridge)
#(network-script 'network-bridge netdev=bond0')
#(network-script my-network-script)
(network-script 'network-bridge netdev=eth0')
#(network-script 'network-bridge netdev=eth1')
#(network-script 'network-bridge netdev=eth2')
#(network-script /bin/true)
#(network-script muti-2-network-script)

# The script used to control virtual interfaces.  This can be overridden on a
# per-vif basis when creating a domain or a configuring a new vif.  The
# vif-bridge script is designed for use with the network-bridge script, or
# similar configurations.
#
# If you have overridden the bridge name using
# (network-script 'network-bridge bridge=<name>') then you may wish to do the
# same here.  The bridge name can also be set when creating a domain or
# configuring a new vif, but a value specified here would act as a default.
#
# If you are using only one bridge, the vif-bridge script will discover that,
# so there is no need to specify it explicitly.  The default is to use
# the bridge which is listed first in the output from brctl.
#
(vif-script vif-bridge)


## Use the following if network traffic is routed, as an alternative to the
# settings for bridged networking given above.
#(network-script network-route)
#(vif-script     vif-route)


## Use the following if network traffic is routed with NAT, as an alternative
# to the settings for bridged networking given above.
#(network-script network-nat)
#(vif-script     vif-nat)

# dom0-min-mem is the lowest permissible memory level (in MB) for dom0.
# This is a minimum both for auto-ballooning (as enabled by
# enable-dom0-ballooning below) and for xm mem-set when applied to dom0.
(dom0-min-mem 196)

# Whether to enable auto-ballooning of dom0 to allow domUs to be created.
# If enable-dom0-ballooning = no, dom0 will never balloon out.
(enable-dom0-ballooning yes)

# 32-bit paravirtual domains can only consume physical
# memory below 168GB. On systems with memory beyond that address,
# they'll be confined to memory below 128GB.
# Using total_available_memory (in GB) to specify the amount of memory reserved
# in the memory pool exclusively for 32-bit paravirtual domains.
# Additionally you should use dom0_mem = <-Value> as a parameter in 
# xen kernel to reserve the memory for 32-bit paravirtual domains, default 
# is "0" (0GB).  
(total_available_memory 0) 

# In SMP system, dom0 will use dom0-cpus # of CPUS
# If dom0-cpus = 0, dom0 will take all cpus available
(dom0-cpus 0)

# Whether to enable core-dumps when domains crash.
#(enable-dump no)

# The tool used for initiating virtual TPM migration
#(external-migration-tool '')

# The interface for VNC servers to listen on. Defaults
# to 127.0.0.1  To restore old 'listen everywhere' behaviour
# set this to 0.0.0.0
#(vnc-listen '127.0.0.1')

# The default password for VNC console on HVM domain.
# Empty string is no authentication.
(vncpasswd '')

# The VNC server can be told to negotiate a TLS session
# to encryption all traffic, and provide x509 cert to
# clients enabling them to verify server identity. The
# GTK-VNC widget, virt-viewer, virt-manager and VeNCrypt
# all support the VNC extension for TLS used in QEMU. The
# TightVNC/RealVNC/UltraVNC clients do not.
#
# To enable this create x509 certificates / keys in the
# directory ${XEN_CONFIG_DIR} + vnc
#
#  ca-cert.pem       - The CA certificate
#  server-cert.pem   - The Server certificate signed by the CA
#  server-key.pem    - The server private key
#
# and then uncomment this next line
# (vnc-tls 1)

# The certificate dir can be pointed elsewhere..
#
# (vnc-x509-cert-dir vnc)

# The server can be told to request & validate an x509
# certificate from the client. Only clients with a cert
# signed by the trusted CA will be able to connect. This
# is more secure the password auth alone. Passwd auth can
# used at the same time if desired. To enable client cert
# checking uncomment this:
#
# (vnc-x509-verify 1)

# The default keymap to use for the VM's virtual keyboard
# when not specififed in VM's configuration
#(keymap 'en-us')

# Script to run when the label of a resource has changed.
#(resource-label-change-script '')

# Rotation count of qemu-dm log file.
#(qemu-dm-logrotate-count 10)

# Path where persistent domain configuration is stored.
# Default is /var/lib/xend/domains/
#(xend-domains-path /var/lib/xend/domains)

# Number of seconds xend will wait for device creation and
# destruction
#(device-create-timeout 100)
#(device-destroy-timeout 100)

# When assigning device to HVM guest, we use the strict check for HVM guest by
# default. (For PV guest, we use loose check automatically if necessary.)
# When we assign device to HVM guest, if we meet with the co-assignment
# issues or the ACS issue, we could try changing the option to 'no' -- however,
# we have to realize this may incur security issue and we can't make sure the
# device assignment could really work properly even after we do this.
#(pci-passthrough-strict-check yes)

# If we have a very big scsi device configuration, start of xend is slow,
# because xend scans all the device paths to build its internal PSCSI device
# list.  If we need only a few devices for assigning to a guest, we can reduce
# the scan to this device. Set list list of device paths in same syntax like in
# command lsscsi, e.g. ('16:0:0:0' '15:0') 
# (pscsi-device-mask ('*'))

------------------------------------------------------------------------------------------------------------
I made the GFS2 shared fiber disk array, but each time you run 1-2 hours, the directory corresponding to the GFS2 can not view, can not be accessed,But the two hosts is normal operation.View the log information, error messages as follows(/var/log/messages):

Mar  7 06:23:09 vm1 corosync[2152]:   [QUORUM] Members[1]: 1
Mar  7 06:23:09 vm1 corosync[2152]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar  7 06:23:09 vm1 kernel: dlm: closing connection to node 2
Mar  7 06:23:09 vm1 corosync[2152]:   [CPG   ] downlist received left_list: 1
Mar  7 06:23:09 vm1 corosync[2152]:   [CPG   ] chosen downlist from node r(0) ip(172.19.0.81) 
Mar  7 06:23:09 vm1 corosync[2152]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar  7 06:23:09 vm1 rgmanager[3108]: State change: vm2 DOWN
Mar  7 06:23:09 vm1 kernel: GFS2: fsid=gfscluster:data1.1: jid=0: Trying to acquire journal lock...
Mar  7 06:23:09 vm1 fenced[2206]: fencing node vm2
Mar  7 06:23:09 vm1 fenced[2206]: fence vm2 dev 0.0 agent fence_manual result: error from agent
Mar  7 06:23:09 vm1 fenced[2206]: fence vm2 failed
Mar  7 06:23:09 vm1 corosync[2152]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar  7 06:23:09 vm1 corosync[2152]:   [QUORUM] Members[2]: 1 2
Mar  7 06:23:09 vm1 corosync[2152]:   [QUORUM] Members[2]: 1 2
Mar  7 06:23:09 vm1 corosync[2152]:   [CPG   ] downlist received left_list: 0
Mar  7 06:23:09 vm1 corosync[2152]:   [CPG   ] downlist received left_list: 0
Mar  7 06:23:09 vm1 corosync[2152]:   [CPG   ] chosen downlist from node r(0) ip(172.19.0.81) 
Mar  7 06:23:09 vm1 corosync[2152]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar  7 06:23:09 vm1 gfs_controld[2247]: receive_start 2:4 add node with started_count 3
Mar  7 06:23:09 vm1 rgmanager[3108]: State change: vm2 UP
Mar  7 06:23:12 vm1 fenced[2206]: fencing node vm2
Mar  7 06:23:12 vm1 corosync[2152]: cman killed by node 2 because we were killed by cman_tool or other application
Mar  7 06:23:12 vm1 gfs_controld[2247]: daemon cpg_dispatch error 2
Mar  7 06:23:12 vm1 gfs_controld[2247]: cluster is down, exiting
Mar  7 06:23:12 vm1 rgmanager[3108]: #67: Shutting down uncleanly
Mar  7 06:23:12 vm1 dlm_controld[2222]: cluster is down, exiting
Mar  7 06:23:12 vm1 dlm_controld[2222]: daemon cpg_dispatch error 2
Mar  7 06:23:12 vm1 fenced[2206]: fence vm2 dev 0.0 agent none result: error from ccs
Mar  7 06:23:12 vm1 fenced[2206]: fence vm2 failed
Mar  7 06:23:20 vm1 kernel: dlm: closing connection to node 2
Mar  7 06:23:20 vm1 kernel: dlm: closing connection to node 1
Mar  7 06:25:14 vm1 kernel: INFO: task clvmd:2376 blocked for more than 120 seconds.
Mar  7 06:25:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:25:14 vm1 kernel: clvmd         D ffff8800280de180     0  2376      1 0x00000080
Mar  7 06:25:14 vm1 kernel: ffff8805c8413ce0 0000000000000282 0000000000000000 ffff8800280841e8
Mar  7 06:25:14 vm1 kernel: 00001e756d9fc134 ffff8805c318d460 0000000000000001 0000000101fc1a63
Mar  7 06:25:14 vm1 kernel: ffff8805c8411a10 ffff8805c8413fd8 000000000000fc08 ffff8805c8411a10
Mar  7 06:25:14 vm1 kernel: Call Trace:
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149ead6>] ? _spin_unlock_irqrestore+0x16/0x20
Mar  7 06:25:14 vm1 kernel: [<ffffffff8105d639>] ? try_to_wake_up+0x2f9/0x480
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0422fdd>] dlm_clear_proc_locks+0x3d/0x260 [dlm]
Mar  7 06:25:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e9b0>] ? _spin_lock_irq+0x10/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffffa042e556>] device_close+0x66/0xc0 [dlm]
Mar  7 06:25:14 vm1 kernel: [<ffffffff811533a5>] __fput+0xf5/0x210
Mar  7 06:25:14 vm1 kernel: [<ffffffff811534e5>] fput+0x25/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff8114eabd>] filp_close+0x5d/0x90
Mar  7 06:25:14 vm1 kernel: [<ffffffff8114eb8a>] sys_close+0x9a/0xf0
Mar  7 06:25:14 vm1 kernel: [<ffffffff810140f2>] system_call_fastpath+0x16/0x1b
Mar  7 06:25:14 vm1 kernel: INFO: task kslowd001:2523 blocked for more than 120 seconds.
Mar  7 06:25:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:25:14 vm1 kernel: kslowd001     D ffff880028066180     0  2523      2 0x00000080
Mar  7 06:25:14 vm1 kernel: ffff8805c7ecf998 0000000000000246 0000210000002923 0001f5052c320000
Mar  7 06:25:14 vm1 kernel: 3200000000211000 211800000113052d 51052e3200000000 0000002120000000
Mar  7 06:25:14 vm1 kernel: ffff8805c7ec86b0 ffff8805c7ecffd8 000000000000fc08 ffff8805c7ec86b0
Mar  7 06:25:14 vm1 kernel: Call Trace:
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffffa04240d2>] dlm_lock+0x62/0x1e0 [dlm]
Mar  7 06:25:14 vm1 kernel: [<ffffffff81238c94>] ? vsnprintf+0x484/0x5f0
Mar  7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0471de2>] gdlm_lock+0xe2/0x120 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0471ee0>] ? gdlm_ast+0x0/0xe0 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0471e20>] ? gdlm_bast+0x0/0x50 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0456ca3>] do_xmote+0x163/0x250 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffffa045715a>] run_queue+0xfa/0x170 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffffa045735a>] gfs2_glock_nq+0x11a/0x330 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0457d79>] gfs2_glock_nq_num+0x69/0x90 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:25:14 vm1 kernel: [<ffffffffa046a753>] gfs2_recover_work+0x93/0x7b0 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffff81010b0f>] ? xen_restore_fl_direct_end+0x0/0x1
Mar  7 06:25:14 vm1 kernel: [<ffffffff8100c676>] ? xen_mc_flush+0x106/0x250
Mar  7 06:25:14 vm1 kernel: [<ffffffff8100c2dd>] ? xen_write_cr0+0x4d/0xa0
Mar  7 06:25:14 vm1 kernel: [<ffffffff8100b4be>] ? xen_end_context_switch+0x1e/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff81012776>] ? __switch_to+0x166/0x320
Mar  7 06:25:14 vm1 kernel: [<ffffffff810584b3>] ? finish_task_switch+0x53/0xe0
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0457d71>] ? gfs2_glock_nq_num+0x61/0x90 [gfs2]
Mar  7 06:25:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:25:14 vm1 kernel: [<ffffffff810f87d2>] slow_work_execute+0x232/0x310
Mar  7 06:25:14 vm1 kernel: [<ffffffff810f8a07>] slow_work_thread+0x157/0x370
Mar  7 06:25:14 vm1 kernel: [<ffffffff8108deb0>] ? autoremove_wake_function+0x0/0x40
Mar  7 06:25:14 vm1 kernel: [<ffffffff810f88b0>] ? slow_work_thread+0x0/0x370
Mar  7 06:25:14 vm1 kernel: [<ffffffff8108db56>] kthread+0x96/0xa0
Mar  7 06:25:14 vm1 kernel: [<ffffffff8101514a>] child_rip+0xa/0x20
Mar  7 06:25:14 vm1 kernel: [<ffffffff81014311>] ? int_ret_from_sys_call+0x7/0x1b
Mar  7 06:25:14 vm1 kernel: [<ffffffff81014a9d>] ? retint_restore_args+0x5/0x6
Mar  7 06:25:14 vm1 kernel: [<ffffffff81015140>] ? child_rip+0x0/0x20
Mar  7 06:25:14 vm1 kernel: INFO: task rgmanager:3108 blocked for more than 120 seconds.
Mar  7 06:25:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:25:14 vm1 kernel: rgmanager     D ffff8800280a2180     0  3108   3106 0x00000080
Mar  7 06:25:14 vm1 kernel: ffff8805c65d9ce0 0000000000000286 ffff8805c65d9ca8 ffff8805c65d9ca4
Mar  7 06:25:14 vm1 kernel: ffffffff81b52fa8 ffff8805e6c24500 ffff880028066180 0000000101fc1a6d
Mar  7 06:25:14 vm1 kernel: ffff8805c8f2c6b0 ffff8805c65d9fd8 000000000000fc08 ffff8805c8f2c6b0
Mar  7 06:25:14 vm1 kernel: Call Trace:
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffffa0422fdd>] dlm_clear_proc_locks+0x3d/0x260 [dlm]
Mar  7 06:25:14 vm1 kernel: [<ffffffff8149e9b0>] ? _spin_lock_irq+0x10/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffffa042e556>] device_close+0x66/0xc0 [dlm]
Mar  7 06:25:14 vm1 kernel: [<ffffffff811533a5>] __fput+0xf5/0x210
Mar  7 06:25:14 vm1 kernel: [<ffffffff811534e5>] fput+0x25/0x30
Mar  7 06:25:14 vm1 kernel: [<ffffffff8114eabd>] filp_close+0x5d/0x90
Mar  7 06:25:14 vm1 kernel: [<ffffffff8114eb8a>] sys_close+0x9a/0xf0
Mar  7 06:25:14 vm1 kernel: [<ffffffff810140f2>] system_call_fastpath+0x16/0x1b
Mar  7 06:27:14 vm1 kernel: INFO: task clvmd:2376 blocked for more than 120 seconds.
Mar  7 06:27:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:27:14 vm1 kernel: clvmd         D ffff8800280de180     0  2376      1 0x00000080
Mar  7 06:27:14 vm1 kernel: ffff8805c8413ce0 0000000000000282 0000000000000000 ffff8800280841e8
Mar  7 06:27:14 vm1 kernel: 00001e756d9fc134 ffff8805c318d460 0000000000000001 0000000101fc1a63
Mar  7 06:27:14 vm1 kernel: ffff8805c8411a10 ffff8805c8413fd8 000000000000fc08 ffff8805c8411a10
Mar  7 06:27:14 vm1 kernel: Call Trace:
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149ead6>] ? _spin_unlock_irqrestore+0x16/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffff8105d639>] ? try_to_wake_up+0x2f9/0x480
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0422fdd>] dlm_clear_proc_locks+0x3d/0x260 [dlm]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e9b0>] ? _spin_lock_irq+0x10/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffffa042e556>] device_close+0x66/0xc0 [dlm]
Mar  7 06:27:14 vm1 kernel: [<ffffffff811533a5>] __fput+0xf5/0x210
Mar  7 06:27:14 vm1 kernel: [<ffffffff811534e5>] fput+0x25/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8114eabd>] filp_close+0x5d/0x90
Mar  7 06:27:14 vm1 kernel: [<ffffffff8114eb8a>] sys_close+0x9a/0xf0
Mar  7 06:27:14 vm1 kernel: [<ffffffff810140f2>] system_call_fastpath+0x16/0x1b
Mar  7 06:27:14 vm1 kernel: INFO: task kslowd001:2523 blocked for more than 120 seconds.
Mar  7 06:27:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:27:14 vm1 kernel: kslowd001     D ffff880028066180     0  2523      2 0x00000080
Mar  7 06:27:14 vm1 kernel: ffff8805c7ecf998 0000000000000246 0000210000002923 0001f5052c320000
Mar  7 06:27:14 vm1 kernel: 3200000000211000 211800000113052d 51052e3200000000 0000002120000000
Mar  7 06:27:14 vm1 kernel: ffff8805c7ec86b0 ffff8805c7ecffd8 000000000000fc08 ffff8805c7ec86b0
Mar  7 06:27:14 vm1 kernel: Call Trace:
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffffa04240d2>] dlm_lock+0x62/0x1e0 [dlm]
Mar  7 06:27:14 vm1 kernel: [<ffffffff81238c94>] ? vsnprintf+0x484/0x5f0
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471de2>] gdlm_lock+0xe2/0x120 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471ee0>] ? gdlm_ast+0x0/0xe0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471e20>] ? gdlm_bast+0x0/0x50 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0456ca3>] do_xmote+0x163/0x250 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045715a>] run_queue+0xfa/0x170 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045735a>] gfs2_glock_nq+0x11a/0x330 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0457d79>] gfs2_glock_nq_num+0x69/0x90 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffffa046a753>] gfs2_recover_work+0x93/0x7b0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b0f>] ? xen_restore_fl_direct_end+0x0/0x1
Mar  7 06:27:14 vm1 kernel: [<ffffffff8100c676>] ? xen_mc_flush+0x106/0x250
Mar  7 06:27:14 vm1 kernel: [<ffffffff8100c2dd>] ? xen_write_cr0+0x4d/0xa0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8100b4be>] ? xen_end_context_switch+0x1e/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff81012776>] ? __switch_to+0x166/0x320
Mar  7 06:27:14 vm1 kernel: [<ffffffff810584b3>] ? finish_task_switch+0x53/0xe0
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0457d71>] ? gfs2_glock_nq_num+0x61/0x90 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffff810f87d2>] slow_work_execute+0x232/0x310
Mar  7 06:27:14 vm1 kernel: [<ffffffff810f8a07>] slow_work_thread+0x157/0x370
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108deb0>] ? autoremove_wake_function+0x0/0x40
Mar  7 06:27:14 vm1 kernel: [<ffffffff810f88b0>] ? slow_work_thread+0x0/0x370
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108db56>] kthread+0x96/0xa0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101514a>] child_rip+0xa/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffff81014311>] ? int_ret_from_sys_call+0x7/0x1b
Mar  7 06:27:14 vm1 kernel: [<ffffffff81014a9d>] ? retint_restore_args+0x5/0x6
Mar  7 06:27:14 vm1 kernel: [<ffffffff81015140>] ? child_rip+0x0/0x20
Mar  7 06:27:14 vm1 kernel: INFO: task gfs2_quotad:2531 blocked for more than 120 seconds.
Mar  7 06:27:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:27:14 vm1 kernel: gfs2_quotad   D ffff880028084180     0  2531      2 0x00000080
Mar  7 06:27:14 vm1 kernel: ffff8805c57e9a98 0000000000000246 0000000000000000 0000000000000008
Mar  7 06:27:14 vm1 kernel: 0000000000000001 0000000000016180 0000000000016180 0000000101fc6633
Mar  7 06:27:14 vm1 kernel: ffff8805c85fc6f0 ffff8805c57e9fd8 000000000000fc08 ffff8805c85fc6f0
Mar  7 06:27:14 vm1 kernel: Call Trace:
Mar  7 06:27:14 vm1 kernel: [<ffffffff81059356>] ? update_curr+0xe6/0x1e0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:27:14 vm1 kernel: [<ffffffffa04240d2>] dlm_lock+0x62/0x1e0 [dlm]
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b0f>] ? xen_restore_fl_direct_end+0x0/0x1
Mar  7 06:27:14 vm1 kernel: [<ffffffff8100c676>] ? xen_mc_flush+0x106/0x250
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471de2>] gdlm_lock+0xe2/0x120 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471ee0>] ? gdlm_ast+0x0/0xe0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471e20>] ? gdlm_bast+0x0/0x50 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0456ca3>] do_xmote+0x163/0x250 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045715a>] run_queue+0xfa/0x170 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045735a>] gfs2_glock_nq+0x11a/0x330 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa046f234>] gfs2_statfs_sync+0x54/0x1a0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa046f22c>] ? gfs2_statfs_sync+0x4c/0x1a0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0467006>] quotad_check_timeo+0x46/0xb0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0467134>] gfs2_quotad+0xc4/0x1f0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108deb0>] ? autoremove_wake_function+0x0/0x40
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0467070>] ? gfs2_quotad+0x0/0x1f0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108db56>] kthread+0x96/0xa0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101514a>] child_rip+0xa/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffff81014311>] ? int_ret_from_sys_call+0x7/0x1b
Mar  7 06:27:14 vm1 kernel: [<ffffffff81014a9d>] ? retint_restore_args+0x5/0x6
Mar  7 06:27:14 vm1 kernel: [<ffffffff81015140>] ? child_rip+0x0/0x20
Mar  7 06:27:14 vm1 kernel: INFO: task rgmanager:3108 blocked for more than 120 seconds.
Mar  7 06:27:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:27:14 vm1 kernel: rgmanager     D ffff8800280a2180     0  3108   3106 0x00000080
Mar  7 06:27:14 vm1 kernel: ffff8805c65d9ce0 0000000000000286 ffff8805c65d9ca8 ffff8805c65d9ca4
Mar  7 06:27:14 vm1 kernel: ffffffff81b52fa8 ffff8805e6c24500 ffff880028066180 0000000101fc1a6d
Mar  7 06:27:14 vm1 kernel: ffff8805c8f2c6b0 ffff8805c65d9fd8 000000000000fc08 ffff8805c8f2c6b0
Mar  7 06:27:14 vm1 kernel: Call Trace:
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0422fdd>] dlm_clear_proc_locks+0x3d/0x260 [dlm]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e9b0>] ? _spin_lock_irq+0x10/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffffa042e556>] device_close+0x66/0xc0 [dlm]
Mar  7 06:27:14 vm1 kernel: [<ffffffff811533a5>] __fput+0xf5/0x210
Mar  7 06:27:14 vm1 kernel: [<ffffffff811534e5>] fput+0x25/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8114eabd>] filp_close+0x5d/0x90
Mar  7 06:27:14 vm1 kernel: [<ffffffff8114eb8a>] sys_close+0x9a/0xf0
Mar  7 06:27:14 vm1 kernel: [<ffffffff810140f2>] system_call_fastpath+0x16/0x1b
Mar  7 06:27:14 vm1 kernel: INFO: task flush-253:5:4169 blocked for more than 120 seconds.
Mar  7 06:27:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:27:14 vm1 kernel: flush-253:5   D ffff880028084180     0  4169      2 0x00000080
Mar  7 06:27:14 vm1 kernel: ffff880533b31a80 0000000000000246 0000000000000000 000000000000002a
Mar  7 06:27:14 vm1 kernel: ffffffff8101027d ffff880533b31a50 ffffffff81010b22 0000000101fc64fb
Mar  7 06:27:14 vm1 kernel: ffff8805bee57080 ffff880533b31fd8 000000000000fc08 ffff8805bee57080
Mar  7 06:27:14 vm1 kernel: Call Trace:
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffff81010b0f>] ? xen_restore_fl_direct_end+0x0/0x1
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0455380>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045538e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149d37f>] __wait_on_bit+0x5f/0x90
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045e175>] ? gfs2_aspace_writepage+0x105/0x170 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0455380>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149d428>] out_of_line_wait_on_bit+0x78/0x90
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108def0>] ? wake_bit_function+0x0/0x50
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0456506>] gfs2_glock_wait+0x36/0x40 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa04574c0>] gfs2_glock_nq+0x280/0x330 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa046fe22>] gfs2_write_inode+0x82/0x190 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa046fe1a>] ? gfs2_write_inode+0x7a/0x190 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8117807c>] writeback_single_inode+0x22c/0x310
Mar  7 06:27:14 vm1 kernel: [<ffffffff81178434>] writeback_inodes_wb+0x194/0x540
Mar  7 06:27:14 vm1 kernel: [<ffffffff811788ea>] wb_writeback+0x10a/0x1d0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149d019>] ? schedule_timeout+0x199/0x2f0
Mar  7 06:27:14 vm1 kernel: [<ffffffff81178b44>] wb_do_writeback+0x194/0x1a0
Mar  7 06:27:14 vm1 kernel: [<ffffffff81178ba3>] bdi_writeback_task+0x53/0xf0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8111b660>] ? bdi_start_fn+0x0/0xe0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8111b6d1>] bdi_start_fn+0x71/0xe0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8111b660>] ? bdi_start_fn+0x0/0xe0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108db56>] kthread+0x96/0xa0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101514a>] child_rip+0xa/0x20
Mar  7 06:27:14 vm1 kernel: [<ffffffff81014311>] ? int_ret_from_sys_call+0x7/0x1b
Mar  7 06:27:14 vm1 kernel: [<ffffffff81014a9d>] ? retint_restore_args+0x5/0x6
Mar  7 06:27:14 vm1 kernel: [<ffffffff81015140>] ? child_rip+0x0/0x20
Mar  7 06:27:14 vm1 kernel: INFO: task dd:21764 blocked for more than 120 seconds.
Mar  7 06:27:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:27:14 vm1 kernel: dd            D ffff880028066180     0 21764   4165 0x00000080
Mar  7 06:27:14 vm1 kernel: ffff8804850c1820 0000000000000282 0000000000000000 000000000000000b
Mar  7 06:27:14 vm1 kernel: 00000040ffffffff 0000000000000000 ffff880000031908 0000000101fc5c6f
Mar  7 06:27:14 vm1 kernel: ffff8805bf8650c0 ffff8804850c1fd8 000000000000fc08 ffff8805bf8650c0
Mar  7 06:27:14 vm1 kernel: Call Trace:
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:27:14 vm1 kernel: [<ffffffffa04240d2>] dlm_lock+0x62/0x1e0 [dlm]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108dd77>] ? bit_waitqueue+0x17/0xd0
Mar  7 06:27:14 vm1 kernel: [<ffffffff8108de9f>] ? wake_up_bit+0x2f/0x40
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471de2>] gdlm_lock+0xe2/0x120 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471ee0>] ? gdlm_ast+0x0/0xe0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0471e20>] ? gdlm_bast+0x0/0x50 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0456ca3>] do_xmote+0x163/0x250 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045715a>] run_queue+0xfa/0x170 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa045735a>] gfs2_glock_nq+0x11a/0x330 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa046d895>] gfs2_inplace_reserve_i+0x385/0x820 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:27:14 vm1 kernel: [<ffffffff811404c5>] ? kmem_cache_alloc_notrace+0x115/0x130
Mar  7 06:27:14 vm1 kernel: [<ffffffffa0459a07>] gfs2_createi+0x1a7/0xbb0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff811e7261>] ? avc_has_perm+0x71/0x90
Mar  7 06:27:14 vm1 kernel: [<ffffffffa046559e>] gfs2_create+0x7e/0x1b0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffffa04598ff>] ? gfs2_createi+0x9f/0xbb0 [gfs2]
Mar  7 06:27:14 vm1 kernel: [<ffffffff8115ee64>] vfs_create+0xb4/0xe0
Mar  7 06:27:14 vm1 kernel: [<ffffffff811627b2>] do_filp_open+0x922/0xc70
Mar  7 06:27:14 vm1 kernel: [<ffffffff8116df95>] ? alloc_fd+0x95/0x160
Mar  7 06:27:14 vm1 kernel: [<ffffffff8114ec49>] do_sys_open+0x69/0x140
Mar  7 06:27:14 vm1 kernel: [<ffffffff8114ed60>] sys_open+0x20/0x30
Mar  7 06:27:14 vm1 kernel: [<ffffffff810140f2>] system_call_fastpath+0x16/0x1b
Mar  7 06:29:14 vm1 kernel: INFO: task clvmd:2376 blocked for more than 120 seconds.
Mar  7 06:29:14 vm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  7 06:29:14 vm1 kernel: clvmd         D ffff8800280de180     0  2376      1 0x00000080
Mar  7 06:29:14 vm1 kernel: ffff8805c8413ce0 0000000000000282 0000000000000000 ffff8800280841e8
Mar  7 06:29:14 vm1 kernel: 00001e756d9fc134 ffff8805c318d460 0000000000000001 0000000101fc1a63
Mar  7 06:29:14 vm1 kernel: ffff8805c8411a10 ffff8805c8413fd8 000000000000fc08 ffff8805c8411a10
Mar  7 06:29:14 vm1 kernel: Call Trace:
Mar  7 06:29:14 vm1 kernel: [<ffffffff8149e715>] rwsem_down_failed_common+0x85/0x1c0
Mar  7 06:29:14 vm1 kernel: [<ffffffff8149ead6>] ? _spin_unlock_irqrestore+0x16/0x20
Mar  7 06:29:14 vm1 kernel: [<ffffffff8105d639>] ? try_to_wake_up+0x2f9/0x480
Mar  7 06:29:14 vm1 kernel: [<ffffffff8149e8a6>] rwsem_down_read_failed+0x26/0x30
Mar  7 06:29:14 vm1 kernel: [<ffffffff81239a14>] call_rwsem_down_read_failed+0x14/0x30
Mar  7 06:29:14 vm1 kernel: [<ffffffff8149ddd4>] ? down_read+0x24/0x30
Mar  7 06:29:14 vm1 kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
Mar  7 06:29:14 vm1 kernel: [<ffffffffa0422fdd>] dlm_clear_proc_locks+0x3d/0x260 [dlm]
Mar  7 06:29:14 vm1 kernel: [<ffffffff8101027d>] ? xen_force_evtchn_callback+0xd/0x10
Mar  7 06:29:14 vm1 kernel: [<ffffffff81010b22>] ? check_events+0x12/0x20
Mar  7 06:29:14 vm1 kernel: [<ffffffff8149e9b0>] ? _spin_lock_irq+0x10/0x30
Mar  7 06:29:14 vm1 kernel: [<ffffffffa042e556>] device_close+0x66/0xc0 [dlm]
Mar  7 06:29:14 vm1 kernel: [<ffffffff811533a5>] __fput+0xf5/0x210
Mar  7 06:29:14 vm1 kernel: [<ffffffff811534e5>] fput+0x25/0x30
Mar  7 06:29:14 vm1 kernel: [<ffffffff8114eabd>] filp_close+0x5d/0x90
Mar  7 06:29:14 vm1 kernel: [<ffffffff8114eb8a>] sys_close+0x9a/0xf0
Mar  7 06:29:14 vm1 kernel: [<ffffffff810140f2>] system_call_fastpath+0x16/0x1b
 7 Mar 08:00:02 ntpdate[21981]: adjust time server 172.19.0.175 offset 0.090922 sec
Mar  7 08:38:35 vm1 ipwatchd[2801]: MAC address 0:1e:90:66:8e:f8 causes IP conflict with address 172.19.0.91 set on interface eth1 - passive mode - reply not sent
Mar  7 08:44:20 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:44:20 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:44:21 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:44:21 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:44:21 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:44:24 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:44:33 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:45:00 vm1 avahi-daemon[2364]: Invalid query packet.
Mar  7 08:47:40 vm1 ipwatchd[2801]: MAC address 0:1e:90:66:8e:f8 causes IP conflict with address 172.19.0.91 set on interface eth1 - passive mode - reply not sent
Mar  7 08:52:30 vm1 ipwatchd[2801]: MAC address 0:1e:90:66:8e:f8 causes IP conflict with address 172.19.0.91 set on interface eth1 - passive mode - reply not sent
Mar  7 09:06:13 vm1 ipwatchd[2801]: MAC address 0:1e:90:66:8e:f8 causes IP conflict with address 172.19.0.91 set on interface eth1 - passive mode - reply not sent
Mar  7 09:19:55 vm1 ipwatchd[2801]: MAC address 0:1e:90:66:8e:f8 causes IP conflict with address 172.19.0.91 set on interface eth1 - passive mode - reply not sent
Mar  7 09:33:16 vm1 ipwatchd[2801]: MAC address 0:1e:90:66:8e:f8 causes IP conflict with address 172.19.0.91 set on interface eth1 - passive mode - reply not sent
Mar  7 09:36:10 vm1 ipwatchd[2801]: MAC address 0:1e:90:66:8e:f8 causes IP conflict with address 172.19.0.91 set on interface eth1 - passive mode - reply not sent


clustat:
Cluster Status for gfscluster @ Wed Mar  7 10:04:48 2012
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 vm1                                         1 Offline
 vm2                                         2 Online, Local

cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Fri Dec 16 10:16:46 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=3e3d4927-c9ce-4543-96b7-940eaa429a49 /boot                   ext4    defaults        1 2
# UUID=cda393e2-cd8c-471a-9a42-11373d467e8e /vm                   ocfs2    _netdev,nointr        0 0
UUID="ae2f16af-f5c7-d867-75ad-c62f031a0b5f"     /vm gfs2    _netdev,noatime 0 0
/dev/mapper/VolGroup-lv_home /home                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0


Before running the OCFS2 problems, the replacement became the GFS2.
I replaced the Red Hat kernel, test a week no problem, the above problems using XEN kernel, how can I fix it? The cause of some kernel parameters?

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.