[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-API] Local Storage Missing 200+ GB


  • To: "xen-api@xxxxxxxxxxxxxxxxxxxx" <xen-api@xxxxxxxxxxxxxxxxxxxx>
  • From: Brian Menges <bmenges@xxxxxxxxxx>
  • Date: Fri, 22 Nov 2013 06:16:32 +0000
  • Accept-language: en-US
  • Delivery-date: Fri, 22 Nov 2013 06:16:59 +0000
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=january; d=gogrid.com; h=Received:Received:From:To:Subject:Thread-Topic:Thread-Index:Date:Message-ID:References:In-Reply-To:Accept-Language:Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:x-originating-ip:Content-Type:Content-Transfer-Encoding:MIME-Version; b=k4FlnSmDcPtaEN591MzLMDIY+NutpJZ7X61WT10S4X4Y8fKVElzb8UKlMlGyk9WTST4vP4qzxOzN7DMqT0MhIVe7mwTG0cllfoHG0DsP0RqbnWfqUoXeQP4iOIkjPPza;
  • List-id: User and development list for XCP and XAPI <xen-api.lists.xen.org>
  • Thread-index: AQHO4iWd03gixWTUu06ceypqcr51AJowz8QT
  • Thread-topic: [Xen-API] Local Storage Missing 200+ GB

snapshots track diffs, so if you keep them around, and the VM continues to run 
over a prolonged period of time, the size of a snapshot could be the same size 
of the original volume size, PER SNAPSHOT. Generally speaking, you wouldn't 
write the entire disk contents over, however it is possible, and that is why 
your space continues to grow.

trim your snapshot utilization... or monitor how many writes your VM farm 
produces. If you have a 1tb disk, and write 30g/day, then after 30 days you're 
going to consume close to that 1tb if you have 30 vms... of which you already 
have space issues at that volume.


---------------------------------------------------------
Brian S. Menges
Principal Engineer, DevOps
2 Harrison, Suite 200|San Francisco, CA|94105
D 415.869.7000|F 415.869.7001
________________________________
From: xen-api-bounces@xxxxxxxxxxxxx [xen-api-bounces@xxxxxxxxxxxxx] on behalf 
of Asai [asai@xxxxxxxxxxxxxxxxxxxxx]
Sent: Friday, November 15, 2013 09:10
To: xen-api@xxxxxxxxxxxxxxxxxxxx
Subject: [Xen-API] Local Storage Missing 200+ GB

Greetings,

Running XCP 1.6.  This seems to be a recurring problem with many people, 
running snapshots seems to be slowly eating up space. Where we have a 1 TB 
local storage capacity and 700 GB of VMs running, we're up to 99% capacity used 
on the local storage drive. Out of 9 VMs, there seem to be 6 that are hidden 
which I'm unable to view in XenCenter even by revealing hidden objects. 
Basically I need to know what the best technique is for cleaning up the space 
here.  I tried running the coalesce-leaf command but got an error:

[root@ultimata ~]# xe host-call-plugin 
host-uuid=7d56ad1f-34f2-4efb-9635-623aca23df39 plugin=coalesce-leaf 
fn=leaf-coalesce args:vm-uuid=fb7a4699-6e28-49ef-b4ca-6b6db122b458
There was a failure communicating with the plugin.
status: leaf-coalesce
stdout: KeyError
stderr: 'vm_uuid'

I hesitant to export and re-import the VMs because that can make for hours of 
downtime for critical servers.  Any insight is much appreciated.

 The output of vdi-list, vhd-util scan and lvscan is as follows:

[root@ultimata ~]# xe vdi-list
uuid ( RO)                : 20425ffb-102b-4a45-8022-c72ee2ae9a88
          name-label ( RW): xs-tools.iso
    name-description ( RW):
             sr-uuid ( RO): dcc6518f-adbb-9624-3663-3bd206a8b2e5
        virtual-size ( RO): 118558720
            sharable ( RO): false
           read-only ( RO): true


uuid ( RO)                : 148b97ef-09e6-4281-aac5-63fcc5cd41b5
          name-label ( RW): Triata Mail Storage
    name-description ( RW): Mail Server storage
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 214748364800
            sharable ( RO): false
           read-only ( RO): true


uuid ( RO)                : 50c348ca-1aff-401c-9d3a-8bfb96768892
          name-label ( RW): Nebula Storage
    name-description ( RW): Created by template provisioner
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 85899345920
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : 997b4d5e-743e-4aa1-a34c-a8a7c429eb7a
          name-label ( RW): Metamorphosis Storage
    name-description ( RW): Created by template provisioner
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 128849018880
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : 14bab14b-4b11-4ae0-8e53-1f5cbbeac60d
          name-label ( RW): Wisdom Storage
    name-description ( RW):
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 51539607552
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : 9c79e3f9-36bc-4981-becb-f0c66948d3ae
          name-label ( RW): Identity Storage
    name-description ( RW):
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 8589934592
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : f1833fcc-f04a-43fa-b688-bf1cb58a1ec8
          name-label ( RW): Triata Mail Storage
    name-description ( RW): Mail Server storage
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 214748364800
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : fb7a4699-6e28-49ef-b4ca-6b6db122b458
          name-label ( RW): Quadratic Storage
    name-description ( RW):
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 53687091200
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : af3b7dfa-9379-4029-b8da-ca8decfe8dc3
          name-label ( RW): Hexagon Storage
    name-description ( RW): Created by template provisioner
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 107374182400
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : 9f52582c-75e0-4907-afa7-f05db8e30ae7
          name-label ( RW): FreePBX Storage
    name-description ( RW):
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 85899345920
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : c756eda2-95a4-4b8e-bb45-c0de3591fc8b
          name-label ( RW): base copy
    name-description ( RW):
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 214748364800
            sharable ( RO): false
           read-only ( RO): true


uuid ( RO)                : 9c892a87-247f-45ac-8261-db3fb068a75a
          name-label ( RW): Pentagon Storage
    name-description ( RW): Created by template provisioner
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 8589934592
            sharable ( RO): false
           read-only ( RO): false


uuid ( RO)                : 302c421e-185e-4191-82f3-8ce54faed55e
          name-label ( RW): base copy
    name-description ( RW):
             sr-uuid ( RO): 2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
        virtual-size ( RO): 85899345920
            sharable ( RO): false
           read-only ( RO): true

[root@ultimata ~]# vhd-util scan -f -mm "VHD-*" -l 
VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a
vhd=VHD-c756eda2-95a4-4b8e-bb45-c0de3591fc8b capacity=214748364800 
size=174709538816 hidden=1 parent=none
vhd=VHD-e7240692-8e32-4bd6-9ff2-d175575fb06f capacity=8589934592 
size=7713325056 hidden=1 parent=none
vhd=VHD-af3b7dfa-9379-4029-b8da-ca8decfe8dc3 capacity=107374182400 
size=107592286208 hidden=0 parent=none
vhd=VHD-997b4d5e-743e-4aa1-a34c-a8a7c429eb7a capacity=128849018880 
size=129109065728 hidden=0 parent=none
vhd=VHD-2b4e9296-89a1-4c43-8892-96c991138884 capacity=8589934592 
size=7331643392 hidden=1 parent=none
vhd=VHD-302c421e-185e-4191-82f3-8ce54faed55e capacity=85899345920 
size=27837595648 hidden=1 parent=none
vhd=VHD-14bab14b-4b11-4ae0-8e53-1f5cbbeac60d capacity=51539607552 
size=51648659456 hidden=0 parent=none
vhd=VHD-fb7a4699-6e28-49ef-b4ca-6b6db122b458 capacity=53687091200 
size=53800337408 hidden=0 parent=none
vhd=VHD-9f52582c-75e0-4907-afa7-f05db8e30ae7 capacity=85899345920 
size=86075506688 hidden=0 parent=none
vhd=VHD-ccb2639c-f3dc-42d7-9f5b-c0a2ae3f29a9 capacity=214748364800 
size=13887340544 hidden=1 parent=VHD-c756eda2-95a4-4b8e-bb45-c0de3591fc8b
vhd=VHD-f1833fcc-f04a-43fa-b688-bf1cb58a1ec8 capacity=214748364800 
size=215176183808 hidden=0 parent=VHD-ccb2639c-f3dc-42d7-9f5b-c0a2ae3f29a9
vhd=VHD-9c79e3f9-36bc-4981-becb-f0c66948d3ae capacity=8589934592 
size=8615100416 hidden=0 parent=VHD-2b4e9296-89a1-4c43-8892-96c991138884
vhd=VHD-cacb557b-625f-4fd3-b62a-7f5d8d639f37 capacity=85899345920 
size=1182793728 hidden=1 parent=VHD-302c421e-185e-4191-82f3-8ce54faed55e
vhd=VHD-50c348ca-1aff-401c-9d3a-8bfb96768892 capacity=85899345920 
size=86075506688 hidden=0 parent=VHD-cacb557b-625f-4fd3-b62a-7f5d8d639f37
vhd=VHD-9c892a87-247f-45ac-8261-db3fb068a75a capacity=8589934592 
size=8615100416 hidden=0 parent=VHD-e7240692-8e32-4bd6-9ff2-d175575fb06f

[root@ultimata ~]# lvscan
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/MGT' [4.00 MB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-c756eda2-95a4-4b8e-bb45-c0de3591fc8b'
 [162.71 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-e7240692-8e32-4bd6-9ff2-d175575fb06f'
 [7.18 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-af3b7dfa-9379-4029-b8da-ca8decfe8dc3'
 [100.20 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-997b4d5e-743e-4aa1-a34c-a8a7c429eb7a'
 [120.24 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-2b4e9296-89a1-4c43-8892-96c991138884'
 [6.83 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-302c421e-185e-4191-82f3-8ce54faed55e'
 [25.93 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-14bab14b-4b11-4ae0-8e53-1f5cbbeac60d'
 [48.10 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-fb7a4699-6e28-49ef-b4ca-6b6db122b458'
 [50.11 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-9f52582c-75e0-4907-afa7-f05db8e30ae7'
 [80.16 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-f1833fcc-f04a-43fa-b688-bf1cb58a1ec8'
 [200.40 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-9c79e3f9-36bc-4981-becb-f0c66948d3ae'
 [8.02 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-cacb557b-625f-4fd3-b62a-7f5d8d639f37'
 [1.10 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-50c348ca-1aff-401c-9d3a-8bfb96768892'
 [80.16 GB] inherit
  ACTIVE            
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/VHD-9c892a87-247f-45ac-8261-db3fb068a75a'
 [8.02 GB] inherit
  inactive          
'/dev/VG_XenStorage-2b0ba7f5-aaa8-735e-93a1-dadd9c88762a/coalesce_cacb557b-625f-4fd3-b62a-7f5d8d639f37_1'
 [4.00 MB] inherit

--
--Asai

________________________________

The information contained in this message, and any attachments, may contain 
confidential and legally privileged material. It is solely for the use of the 
person or entity to which it is addressed. Any review, retransmission, 
dissemination, or action taken in reliance upon this information by persons or 
entities other than the intended recipient is prohibited. If you receive this 
in error, please contact the sender and delete the material from any computer.

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.