[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-API] [xcp 1.6] rrdd issues


  • To: xen-api@xxxxxxxxxxxxx
  • From: Sébastien RICCIO <sr@xxxxxxxxxxxxxxx>
  • Date: Fri, 30 Nov 2012 06:06:58 +0100
  • Delivery-date: Fri, 30 Nov 2012 05:07:16 +0000
  • List-id: User and development list for XCP and XAPI <xen-api.lists.xen.org>

Hi list!

First, I would like to send my congratulations to the team for the final release of XCP 1.6!

I've recently (manually) upgraded our production XCP 1.1 pool to XCP 1.6 and this worked flawlessly. After running the upgrade I can see that there seems to be a new feature for getting metrics informations such as iostat, power management, etc. and this is good news :) The less good news is that it seems it doesn't work (at least on our machines, some Dell M610 blades)

Here is what I can see in xensource.log, on the master:

Nov 30 01:16:30 xen-blade11 xcp-rrdd-xenpm: [ info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm Nov 30 01:16:31 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm Nov 30 01:16:31 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-iostat Nov 30 01:16:32 xen-blade11 xcp-rrdd-squeezed: [ info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to /dev/shm/metrics/xcp-rrdd-squeezed Nov 30 01:16:36 xen-blade11 xcp-rrdd-xenpm: [ info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm Nov 30 01:16:36 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm Nov 30 01:16:36 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-iostat Nov 30 01:16:37 xen-blade11 xcp-rrdd-squeezed: [ info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to /dev/shm/metrics/xcp-rrdd-squeezed Nov 30 01:16:41 xen-blade11 xcp-rrdd-xenpm: [ info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm Nov 30 01:16:41 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm Nov 30 01:16:41 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-iostat Nov 30 01:16:42 xen-blade11 xcp-rrdd-squeezed: [ info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to /dev/shm/metrics/xcp-rrdd-squeezed Nov 30 01:16:46 xen-blade11 xcp-rrdd-xenpm: [ info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm Nov 30 01:16:46 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm Nov 30 01:16:46 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-iostat Nov 30 01:16:47 xen-blade11 xcp-rrdd-squeezed: [ info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to /dev/shm/metrics/xcp-rrdd-squeezed Nov 30 01:16:51 xen-blade11 xcp-rrdd-xenpm: [ info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm Nov 30 01:16:51 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm Nov 30 01:16:51 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-iostat Nov 30 01:16:52 xen-blade11 xcp-rrdd-squeezed: [ info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to /dev/shm/metrics/xcp-rrdd-squeezed Nov 30 01:16:56 xen-blade11 xcp-rrdd-xenpm: [ info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm Nov 30 01:16:56 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm Nov 30 01:16:56 xen-blade11 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-iostat Nov 30 01:16:57 xen-blade11 xcp-rrdd-squeezed: [ info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to /dev/shm/metrics/xcp-rrdd-squeezed Nov 30 01:17:01 xen-blade11 xcp-rrdd-xenpm: [ info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm

xenstore.log on the slaves (all):
Nov 30 05:49:03 xen-blade15 /opt/xensource/libexec/xcp-rrdd: [error|xen-blade15|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm
Nov 30 05:49:38 xen-blade15 last message repeated 7 times
Nov 30 05:50:43 xen-blade15 last message repeated 13 times
Nov 30 05:51:48 xen-blade15 last message repeated 13 times
Nov 30 05:52:53 xen-blade15 last message repeated 13 times
Nov 30 05:53:58 xen-blade15 last message repeated 13 times
Nov 30 05:54:59 xen-blade15 last message repeated 12 times
Nov 30 05:56:04 xen-blade15 last message repeated 13 times
Nov 30 05:57:09 xen-blade15 last message repeated 13 times

Seems there is a problem with xcp-rrdd-xenpm, but if i run it manually:

[root@xen-blade15 ~]# /opt/xensource/libexec/xcp-rrdd-plugins/xcp-rrdd-xenpm
[20121130T05:02:26.139Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Entering main loop .. [20121130T05:02:26.139Z|debug|xen-blade15|0||mscgen] unset=>unset [label="Plugin.get_header"]; [20121130T05:02:26.139Z|debug|xen-blade15|0||mscgen] unset=>unset [label="Plugin.get_path"]; [20121130T05:02:26.140Z| info|xen-blade15|0||xcp-rrdd-xenpm] Obtained hdr=DATASOURCES
, path=/dev/shm/metrics/xcp-rrdd-xenpm

[20121130T05:02:26.140Z|debug|xen-blade15|0||mscgen] unset=>unset [label="Plugin.register"]; [20121130T05:02:26.141Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping for 3.8 seconds... [20121130T05:02:29.972Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking command /usr/sbin/xenpm get-cpufreq-states

[20121130T05:02:29.995Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking command /usr/sbin/xenpm get-cpuidle-states

[20121130T05:02:30.002Z| info|xen-blade15|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm [20121130T05:02:30.011Z|debug|xen-blade15|0||mscgen] unset=>unset [label="Plugin.register"]; [20121130T05:02:30.012Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping for 4.9 seconds... [20121130T05:02:34.971Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking command /usr/sbin/xenpm get-cpufreq-states

[20121130T05:02:34.993Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking command /usr/sbin/xenpm get-cpuidle-states

[20121130T05:02:34.999Z| info|xen-blade15|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm [20121130T05:02:35.010Z|debug|xen-blade15|0||mscgen] unset=>unset [label="Plugin.register"]; [20121130T05:02:35.011Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping for 5.0 seconds... [20121130T05:02:39.981Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking command /usr/sbin/xenpm get-cpufreq-states

[20121130T05:02:40.004Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking command /usr/sbin/xenpm get-cpuidle-states

[20121130T05:02:40.015Z| info|xen-blade15|0||xcp-rrdd-xenpm] Done outputing to /dev/shm/metrics/xcp-rrdd-xenpm [20121130T05:02:40.020Z|debug|xen-blade15|0||mscgen] unset=>unset [label="Plugin.register"]; [20121130T05:02:40.021Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping for 5.0 seconds...
(seems to work)


Also iostat plugin dies. I tryed to run it from command line:

[root@xen-blade11 plugins]# /opt/xensource/libexec/xcp-rrdd-plugins/xcp-rrdd-iostat [20121130T00:13:15.359Z|debug|xen-blade11|0||xcp-rrdd-iostat] Entering main loop .. [20121130T00:13:15.359Z|debug|xen-blade11|0||mscgen] unset=>unset [label="Plugin.get_header"]; [20121130T00:13:15.360Z|debug|xen-blade11|0||mscgen] unset=>unset [label="Plugin.get_path"]; [20121130T00:13:15.361Z| info|xen-blade11|0||xcp-rrdd-iostat] Obtained hdr=DATASOURCES
, path=/dev/shm/metrics/xcp-rrdd-iostat
[20121130T00:13:15.361Z|debug|xen-blade11|0||mscgen] unset=>unset [label="Plugin.register"]; [20121130T00:13:15.362Z|debug|xen-blade11|0||xcp-rrdd-iostat] Sleeping for 4.6 seconds... [20121130T00:13:20.026Z|debug|xen-blade11|0||xcp-rrdd-iostat] Forking command /usr/sbin/tap-ctl list

"/usr/bin/tap-ctl list" returns an unexpected output, which makes this plugin obsolete!! Offending line: pid=12339 minor=5 state=0 args=vhd:/var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/867e2f8d-9438-4101-bbd0-c0a349ab672b.vhd

(note there is also a typo in the error message /usr/bin/tap-ctl should be /usr/sbin/tap/ctl

If i run the tap-list command manually:

[root@xen-blade11 plugins]# /usr/sbin/tap-ctl list
12339 5 0 vhd /var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/867e2f8d-9438-4101-bbd0-c0a349ab672b.vhd 15130 3 0 vhd /var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/a42f99a5-848f-4cef-af43-3dc493ed2a57.vhd 29700 1 0 vhd /var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/915ae08f-9921-4fc7-bb12-5a2406990638.vhd 30679 6 0 vhd /var/run/sr-mount/97805239-b538-f13a-fe79-68f7f5d1ab22/23848d6b-3d4a-4b24-a6d2-782d12061458.vhd 3723 0 0 vhd /var/run/sr-mount/97805239-b538-f13a-fe79-68f7f5d1ab22/f619f6fc-09a0-4bc2-a20a-16bc2feeb0da.vhd 5289 4 0 vhd /var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/2e0b1484-4c73-46e9-944e-01716c64a620.vhd 8603 7 0 vhd /var/run/sr-mount/97805239-b538-f13a-fe79-68f7f5d1ab22/9bbef03c-5369-42fb-a5ab-1b65edb96202.vhd 9696 2 0 vhd /var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/c7fd2830-0e41-4bcf-adda-acc21674bf77.vhd

Seems some things are broken at the moment. I don't know if it's specific to our hardware or if it's a general problem. Please note that I've also installed a clean xcp 1.6 on a stand-alone server (no upgrade, no pool, standard setup) and the same errors occurs.

Any idea what could be wrong?

Do you need more informations to sort this out ?

Thanks a lot for your help.

Cheers,
Sébastien

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.