[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Revisiting Slow Apt (actually disk write)
All disk write activity. Read is within normal.
On 7/27/21 9:42 AM, Chris Myers wrote:
Out
of curiosity is it all disk activity or just apt-related? Eg.
how does
dd
if=/dev/zero of=blah bs=1M count=512
look?
I'm
running a similar-ish config as you (asrock "J" motherboards
and SSD-backed LVM storage instead of regular hard drives) and
thus far am not experiencing that specific issue.
I
have had weird stuff happen with other motherboards however,
in one case where activity on any domu (including dom0)
incurred a large "st"eal, unless the VM was pinned to one
specific dedicated CPU. I never did figure that one out; even
went as far as turning off all ACPI stuff just in case it was
interfering.
Sorry
for the long message, but wanted to give a lack-of-progress
report
Did a lot of troubleshooting but it hasn’t gotten me very far:
System: SuperMicro
Xeon E3-1230 V3 @ 3.3 Ghz
12 GB RAM
Physical Disk: 4 x 2 TB ST2000NM0033 -
Enterprise
Dom0 OS: Debian Buster 10.10, kernel 4.19.0-16-amd64
Xen 4.11
DomUs: All run debian buster as well. they are using Disk.Img
for their
swap and root drives. All have at least 1 GB of RAM
allocated. My test
VM has 2GB.
When I ran debian wheezy, I was not having this problem.
Dom0 is fast in disk read and writes.
Problem: DomU's Seem to be very slow disk writes. For
example, if I
just to an "apt update" this is my output:
Get:1 http://security.debian.org
buster/updates InRelease [65.4 kB]
Get:2 http://ftp.us.debian.org/debian
buster InRelease [122 kB]
Get:3 http://security.debian.org
buster/updates/main Sources [195 kB]
Get:4 http://security.debian.org
buster/updates/main amd64 Packages [297 kB]
Get:5 http://ftp.us.debian.org/debian
buster/non-free Sources [85.7 kB]
Get:6 http://ftp.us.debian.org/debian
buster/main Sources [7836 kB]
Get:7 http://ftp.us.debian.org/debian
buster/contrib Sources [42.5 kB]
Get:8 http://ftp.us.debian.org/debian
buster/main amd64 Packages [7907 kB]
Get:9 http://ftp.us.debian.org/debian
buster/main Translation-en [5968 kB]
Get:10 http://ftp.us.debian.org/debian
buster/contrib amd64 Packages
[50.1 kB]
Get:11 http://ftp.us.debian.org/debian
buster/contrib Translation-en
[44.2 kB]
Get:12 http://ftp.us.debian.org/debian
buster/non-free amd64 Packages
[87.7 kB]
Get:13 http://ftp.us.debian.org/debian
buster/non-free Translation-en
[88.9 kB]
Fetched 22.7 MB in 52min 3s (7277 B/s)
Reading package lists... Done
Notice that it says "52min 3s", this is something that Dom0
takes about
3 seconds to do to the same servers. Meaning, it is not a
network issue.
What I've tried:
- I first went through the I/O tweaks from the xen wiki
pages. Zero effect
- Moved all the DomUs off to another server except one
(tester).
- Formatted and reinstalled entire OS from scratch on Dom0.
- Tried using LVM disk instead of disk images. Slight
improvement,
but still extremely slow.
- Tried using physical disk, dedicating one drive entirely
to the test
DomU, mounting with "phys" in the config. This was also a
slight
improvement, but still very slow disk writes.
"top" on the DomU shows WA very high while trying to get the
update.
Drops to <1 when idle. All other "top" values are <1.
I thought I solved it by going back to an earlier kernel,
4.19.0-14-amd64, and I tried doing that again and it had zero
effect
this time.
Is it possible that there is a kernel flag that is turned off
that would
effect VM disk writes? Reading seems to be normal speed.
Writing to
a network drive from the DomU is very fast as well.
Flag Info from /proc/cpu on Dom0
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov
pat clflush
acpi mmx fxsr sse sse2 ss ht syscall nx
lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq
monitor
est ssse3 cx16 sse4_1 sse4_2
popcnt xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault
ssbd ibrs
ibpb stibp fsgsbase erms xsaveopt md_clear
--
|
