Hello,
We are experiencing a very
strange networking problem with the open source version of Xen. The host
OS was CentOS 5.1, but I believe it has been patched to the latest
version. We are using the Xen virtualization that ships with
CentOS. The Ubuntu guest VM is running 7.1.2008
When the Ubuntu guest VM boots
up, it exhibits the following behavior:
The VM can ping its default
gateway
The VM can ping remote hosts
by IP address
The VM can run nslookup,
connect to both local and remote DNS servers, and successfully resolve names
(so, UDP is working…)
The host OS can SSH into the
VM (both are on the same logical subnet)
The VM can SSH into the host
OS
The VM cannot do any kind of
TCP based connections to hosts other than the host OS. We have tried FTP,
HTTP, and SSH. It should also be noted that we attempted to perform an
installation of a CentOS 5.1 VM using the CD to boot and then selecting HTTP
and FTP downloads from known mirrors. These TCP transactions also failed,
so the problem doesn’t appear to be specific to the Ubuntu VM.
I ran a tcpdump of the virtual
bridge interface and I can see outbound TCP connections and the return
traffic. But for some reason the return traffic isn’t being passed
on to the VM. I tried disabling IPTables on the host OS and that didn’t
impact anything.
It should also be noted that
Xen worked perfectly on another CentOS 5.1 server. The problem is showing
up only on some new SuperMicro servers that my customer purchased.
Any help would be greatly
appreciated!
Here is the uname –an output
of the host OS:
Linux
sf-prod-host2.keystream.com 2.6.18-92.1.6.el5xen #1 SMP Wed Jun 25 14:13:10 EDT
2008 x86_64 x86_64 x86_64 GNU/Linux
Here is the Xen config file:
(logfile
/var/log/xen/xend.log)
(loglevel
DEBUG)
(xend-unix-server
yes)
(xend-unix-path
/var/lib/xend/xend-socket)
(xend-relocation-hosts-allow
'^localhost$ ^localhost\\.localdomain$')
(network-script
network-bridge)
(vif-script
vif-bridge)
(dom0-min-mem
256)
(dom0-cpus
0)
Here is the test.cfg file used
to launch the VM:
kernel
= '/boot/vmlinuz-2.6.22-14-xen'
ramdisk
= '/boot/initrd.img-2.6.22-14-xen'
memory
= '4096'
name = "test"
cpus =
"" # leave to Xen to
pick
vif = [
"mac=00:16:3e:77:b8:a4,bridge=xenbr0" ]
root
= '/dev/hda1 ro'
disk
= ['tap:aio:/xen/test/disk.img,hda1,w','tap:aio:/xen/test/swap.img,hda2,w']
extra = "4"
on_reboot =
'restart'
on_crash =
'restart'
extra = 'xencons=tty'
I don’t see any network
related error messages in the xend log file. Here is the output of the
console when starting the VM:
Started
domain test
[
0.000000] Linux version 2.6.22-14-xen (buildd@king) (gcc version 4.1.3 20070929
(prerelease) (Ubuntu 4.1.2-16ubuntu2)) #1 SMP Tue Feb 12 04:26:15 UTC 2008
(Unofficial)
[
0.000000] Command line: root=/dev/hda1 ro xencons=tty
[
0.000000] BIOS-provided physical RAM map:
[
0.000000] Xen: 0000000000000000 - 0000000100800000 (usable)
[
0.000000] end_pfn_map = 1050624
[
6852.977131] Zone PFN ranges:
[
6852.977137]
DMA 0
-> 1050624
[
6852.977139] DMA32 1050624 ->
1050624
[
6852.977140] Normal 1050624 -> 1050624
[
6852.977142] early_node_map[1] active PFN ranges
[
6852.977143]
0: 0 -> 1050624
[
6853.042698] No mptable found.
[
6853.056141] PERCPU: Allocating 25096 bytes of per cpu data
[
6853.056168] Built 1 zonelists. Total pages: 1036260
[
6853.056171] Kernel command line: root=/dev/hda1 ro xencons=tty
[
6853.056552] Initializing CPU#0
[
6853.056767] PID hash table entries: 4096 (order: 12, 32768 bytes)
[
6853.056812] Xen reported: 2500.086 MHz processor.
[
6853.056840] Console: colour dummy device 80x25
[
6853.058544] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[
6853.059961] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[
6853.060293] Software IO TLB disabled
[
6853.091659] Memory: 4085760k/4202496k available (2343k kernel code, 107940k
reserved, 1193k data, 208k init)
[
6853.091723] SLUB: Genslabs=22, HWalign=64, Order=0-1, MinObjects=4, CPUs=1,
Nodes=1
[
6853.156859] Calibrating delay using timer specific routine.. 5002.75 BogoMIPS
(lpj=10005511)
[
6853.156895] Security Framework v1.0.0 initialized
[
6853.156960] SELinux: Disabled at boot.
[
6853.156972] Mount-cache hash table entries: 256
[
6853.157166] CPU: L1 I cache: 32K, L1 D cache: 32K
[
6853.157172] CPU: L2 cache: 6144K
[
6853.157181] CPU: Physical Processor ID: 1
[
6853.157184] CPU: Processor Core ID: 0
[
6853.157190] SMP alternatives: switching to UP code
[
6853.157561] Freeing SMP alternatives: 23k freed
[
6853.157716] Early unpacking initramfs... done
[
6853.173134] Brought up 1 CPUs
[
6853.173728] NET: Registered protocol family 16
[
6853.174478] Brought up 1 CPUs
[
6853.174491] PCI: Fatal: No config space access function found
[
6853.174494] PCI: setting up Xen PCI frontend stub
[
6853.174847] ACPI: Interpreter disabled.
[
6853.174853] Linux Plug and Play Support v0.97 (c) Adam Belay
[
6853.174861] pnp: PnP ACPI: disabled
[
6853.175035] xen_mem: Initialising balloon driver.
[
6853.175849] Setting mem allocation to 4194304 kiB
[
6853.175934] PCI: System does not support PCI
[
6853.175938] PCI: System does not support PCI
[
6853.175964] NET: Registered protocol family 8
[
6853.175967] NET: Registered protocol family 20
[
6853.176255] NET: Registered protocol family 2
[
6853.177660] Time: xen clocksource has been installed.
[
6853.213677] IP route cache hash table entries: 131072 (order: 8, 1048576
bytes)
[
6853.213812] TCP established hash table entries: 131072 (order: 9, 3145728
bytes)
[
6853.214893] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[
6853.215280] TCP: Hash tables configured (established 131072 bind 65536)
[
6853.215284] TCP reno registered
[
6853.225740] checking if image is initramfs... it is
[
6853.241377] Freeing initrd memory: 22840k freed
[
6853.249233] audit: initializing netlink socket (disabled)
[
6853.249254] audit(1214977623.733:1): initialized
[
6853.250538] VFS: Disk quotas dquot_6.5.1
[
6853.250570] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[
6853.250635] io scheduler noop registered
[
6853.250638] io scheduler anticipatory registered
[
6853.250641] io scheduler deadline registered
[
6853.250701] io scheduler cfq registered (default)
[
6853.263242] rtc: IRQ 8 is not free.
[
6853.263297] Linux agpgart interface v0.102 (c) Dave Jones
[
6853.263651] RAMDISK driver initialized: 16 RAM disks of 65536K size 1024
blocksize
[
6853.263789] input: Macintosh mouse button emulation as /class/input/input0
[
6853.265750] Xen virtual console successfully installed as tty1
[
6853.265799] Event-channel device installed.
[
6853.272915] netfront: Initialising virtual ethernet driver.
[
6853.273941] PNP: No PS/2 controller found. Probing ports directly.
[
6853.274764] i8042.c: No controller found.
[
6853.274835] mice: PS/2 mouse device common for all mice
[
6853.274919] TCP cubic registered
[
6853.274961] NET: Registered protocol family 1
[
6853.275877] xen-vbd: registered block device major 3
[
6853.283185] netfront: device eth0 has flipping receive path.
[
6853.373709]
/build/buildd/linux-source-2.6.22-2.6.22/debian/build/custom-source-xen/drivers/rtc/hctosys.c:
unable to open rtc device (rtc0)
[
6853.373719] Freeing unused kernel memory: 208k freed
Loading,
please wait...
%GCouldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Begin:
Loading essential drivers... ...
[
6853.583996] AppArmor: AppArmor
initialized<5>audit(1214977624.069:2): type=1505
info="AppArmor initialized" pid=1030
[
6853.590474] fuse init (API version 7.8)
[
6853.595279] Failure registering capabilities with primary security module.
[
6853.633963] thermal: Unknown symbol acpi_processor_set_thermal_limit
Done.
Begin:
Running /scripts/init-premount ...
Done.
Begin:
Mounting root file system... ...
Begin:
Running /scripts/local-top ...
Done.
Begin:
Waiting for root file system... ...
Done.
Begin:
Running /scripts/local-premount ...
Done.
[
6853.931263] EXT3-fs: INFO: recovery required on readonly filesystem.
[
6853.931274] EXT3-fs: write access will be enabled during recovery.
[
6854.160033] kjournald starting. Commit interval 5 seconds
[
6854.160171] EXT3-fs: recovery complete.
[
6854.170302] EXT3-fs: mounted filesystem with ordered data mode.
Begin:
Running /scripts/local-bottom ...
Done.
Done.
Begin:
Running /scripts/init-bottom ...
Done.
*
Reading files needed to boot... _[80G
_[74G[
OK ]
*
Setting preliminary keymap... _[80G [
6856.898020] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing
enabled
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
Couldnt
get a file descriptor referring to the console
_[74G[
OK ]
*
Setting the system clock
Cannot
access the Hardware Clock via any known method.
Use
the --debug option to see the details of our search for an access method.
_[33m*_[39;49m
Unable to set System Clock to: Wed Jul 2 05:47:07 UTC 2008
*
Starting basic networking... _[80G
_[74G[
OK ]
*
Starting kernel event manager... _[80G
_[74G[
OK ]
*
Loading hardware drivers... _[80G [
6858.106545] NET: Registered protocol family 10
[
6858.106607] lo: Disabled Privacy Extensions
_[74G[
OK ]
*
Loading kernel modules... _[80G *
Loading manual drivers... _[80G
_[74G[
OK ]
*
Activating swap... _[80G
_[74G[
OK ]
*
Checking root file system... _[80G fsck
1.40.2 (12-Jul-2007)
/dev/hda1:
clean, 105135/1572864 files, 740677/3145728 blocks (check in 2 mounts)
_[74G[
OK ]
*
Checking file systems... _[80G fsck 1.40.2
(12-Jul-2007)
_[74G[
OK ]
*
Mounting local filesystems... _[80G
_[74G[
OK ]
*
Activating swapfile swap... _[80G
_[74G[
OK ]
*
Checking minimum space in /tmp... _[80G
_[74G[
OK ]
*
Configuring network interfaces... _[80G
_[74G[
OK ]
/dev/mem:
mmap: Bad address
/dev/mem:
mmap: Bad address
*
Starting system log daemon... _[80G
_[74G[
OK ]
*
Doing Wacom setup... _[80G cat: */id: No
such file or directory
_[74G[
OK ]
*
Starting kernel log daemon... _[80G
_[74G[
OK ]
*
Starting system message bus dbus _[80G
_[74G[
OK ]
*
Starting network connection manager
NetworkManager _[80G
_[74G[
OK ]
*
Starting network events dispatcher
NetworkManagerDispatcher _[80G
_[74G[
OK ]
*
Starting System Tools Backends
system-tools-backends _[80G
_[74G[
OK ]
*
Starting Hardware abstraction layer hald
_[80G
_[74G[
OK ]
*
Starting OpenBSD Secure Shell server sshd
_[80G
_[74G[
OK ]
*
Starting Avahi mDNS/DNS-SD Daemon
avahi-daemon _[80G
_[74G[
OK ]
*
Starting Common Unix Printing System: cupsd
_[80G
_[74G[
OK ]
/dev/mem:
mmap: Bad address
/dev/mem:
mmap: Bad address
/dev/mem:
mmap: Bad address
/dev/mem:
mmap: Bad address
*
Starting powernowd... _[80G
/etc/rc2.d/S20powernowd: 156: cannot create
/sys/devices/system/cpu/cpu0//cpufreq/scaling_governor: Directory nonexistent
*
CPU frequency scaling not supported
_[74G[
OK ]
Setting
up X font server socket directory /tmp/.font-unix...done.
Starting
X font server: xfs
Ubuntu
7.10 server30.vlnks.com tty1
server30.vlnks.com
login: .
*
Starting ConsoleKit daemon
console-kit-daemon _[80G
_[74G[
OK ]
*
Starting DHCP D-Bus daemon dhcdbd _[80G
_[74G[
OK ]
*
Starting Bluetooth services _[80G
_[74G[
OK ]
*
Starting GNOME Display Manager... _[80G
_[74G[
OK ]
*
Starting anac(h)ronistic cron anacron _[80G
_[74G[
OK ]
*
Checking battery state... _[80G /dev/mem:
mmap: Bad address
/dev/mem:
mmap: Bad address
/dev/mem:
mmap: Bad address
/dev/mem:
mmap: Bad address
_[74G[
OK ]
*
Running local boot scripts (/etc/rc.local)
_[80G
_[74G[
OK ]
Here is the tcpdump log of the
virtual bridge adapter on the host OS. The 10.199.202.30 is the Ubuntu guest
VM IP. The 10.199.202.125 was the IP assigned to the VM that we tried to
do a web based install of CentOS:
20:44:08.467039 IP 69.65.40.19.http >
10.199.202.125.44078: F 1662593966:1662593966(0) ack 1284042731 win 1448
<nop,nop,timestamp 1138161112 4294924267>
20:44:08.467079 IP 10.199.202.125.44078 >
69.65.40.19.http: F 123:123(0) ack 1 win 46 <nop,nop,timestamp 4294931787
1138161112>
20:44:08.467134 IP 10.199.202.125.filenet-tms >
vnsc-bak.sys.gtei.net.domain: 58985+ A? mirrors.gigenet.com. (37)
20:44:08.476795 IP vnsc-bak.sys.gtei.net.domain >
10.199.202.125.filenet-tms: 58985 1/0/0 A 69.65.40.19 (53)
20:44:08.476852 IP 10.199.202.125.filenet-tms >
vnsc-bak.sys.gtei.net.domain: 42117+ A? mirrors.gigenet.com. (37)
20:44:08.485459 IP vnsc-bak.sys.gtei.net.domain >
10.199.202.125.filenet-tms: 42117 1/0/0 A 69.65.40.19 (53)
20:44:08.485510 IP 10.199.202.125.44079 >
69.65.40.19.http: S 1313791500:1313791500(0) win 5840 <mss
1460,sackOK,timestamp 4294931792 0,nop,wscale 7>
20:44:08.544563 IP 69.65.40.19.http >
10.199.202.125.44078: . ack 1 win 1448 <nop,nop,timestamp 1138161191
4294924267,nop,nop,sack 1 {123:124}>
20:44:08.566229 IP 69.65.40.19.http >
10.199.202.125.44079: S 1307633567:1307633567(0) ack 1313791501 win 5792
<mss 1380,sackOK,timestamp 1138161210 4294931792,nop,wscale 2>
20:44:08.566261 IP 10.199.202.125.44079 >
69.65.40.19.http: . ack 1 win 46 <nop,nop,timestamp 4294931812
1138161210>
20:44:08.566275 IP 10.199.202.125.44079 >
69.65.40.19.http: P 1:117(116) ack 1 win 46 <nop,nop,timestamp 4294931812
1138161210>
20:44:08.566369 IP 69.65.40.19.http >
10.199.202.125.44079: P ack 1 win 5792
20:44:08.845092 IP 10.199.202.125.44079 >
69.65.40.19.http: P 1:117(116) ack 1 win 46 <nop,nop,timestamp 4294931882
1138161210>
20:44:08.845189 IP 69.65.40.19.http >
10.199.202.125.44079: P ack 1 win 5792
20:44:09.405088 IP 10.199.202.125.44079 >
69.65.40.19.http: P 1:117(116) ack 1 win 46 <nop,nop,timestamp 4294932022
1138161210>
20:44:09.405192 IP 69.65.40.19.http >
10.199.202.125.44079: P ack 1 win 5792
20:44:10.525083 IP 10.199.202.125.44079 >
69.65.40.19.http: P 1:117(116) ack 1 win 46 <nop,nop,timestamp 4294932302
1138161210>
20:44:10.525186 IP 69.65.40.19.http >
10.199.202.125.44079: P ack 1 win 5792
20:44:12.765074 IP 10.199.202.125.44079 >
69.65.40.19.http: P 1:117(116) ack 1 win 46 <nop,nop,timestamp 4294932862
1138161210>
20:44:12.765184 IP 69.65.40.19.http >
10.199.202.125.44079: P ack 1 win 5792
20:44:13.437071 IP 10.199.202.125.44078 >
69.65.40.19.http: P 1:123(122) ack 1 win 46 <nop,nop,timestamp 4294933030
1138161191>
20:44:13.437181 IP 69.65.40.19.http >
10.199.202.125.44078: P ack 1 win 1448
20:44:17.245053 IP 10.199.202.125.44079 >
69.65.40.19.http: P 1:117(116) ack 1 win 46 <nop,nop,timestamp 4294933982
1138161210>
20:44:17.245166 IP 69.65.40.19.http >
10.199.202.125.44079: P ack 1 win 5792
20:44:19.609046 IP 10.199.202.125.44077 >
69.65.40.19.http: P 1257735782:1257735898(116) ack 2415337884 win 46
<nop,nop,timestamp 4294934573 1138131014>
20:44:19.609149 IP 69.65.40.19.http >
10.199.202.125.44077: P ack 0 win 1448
20:44:54.259007 IP 10.199.202.30.filenet-nch >
h-64-105-172-26.snvacaid.covad.net.domain: 44172+ AAAA? www.ontai.net.
(31)
20:44:54.266961 IP
h-64-105-172-26.snvacaid.covad.net.domain > 10.199.202.30.filenet-nch:
44172 0/1/0 (99)
20:44:54.267052 IP 10.199.202.30.filenet-nch > h-64-105-172-26.snvacaid.covad.net.domain:
39103+ AAAA? www.ontai.net.vlnks.com. (41)
20:44:54.274047 IP
h-64-105-172-26.snvacaid.covad.net.domain > 10.199.202.30.filenet-nch:
39103 NXDomain 0/1/0 (105)
20:44:54.274134 IP 10.199.202.30.filenet-nch >
h-64-105-172-26.snvacaid.covad.net.domain: 65316+ A? www.ontai.net. (31)
20:44:54.281230 IP
h-64-105-172-26.snvacaid.covad.net.domain > 10.199.202.30.filenet-nch:
65316 1/2/1 A 209.234.240.110 (118)
20:44:54.281326 IP 10.199.202.30.54552 >
209.234.240.110.http: S 2789165360:2789165360(0) win 5840 <mss
1460,sackOK,timestamp 42945 0,nop,wscale 7>
20:44:54.293288 IP 209.234.240.110.http >
10.199.202.30.54552: S 640882385:640882385(0) ack 2789165361 win 5792 <mss
1380,sackOK,timestamp 691273371 42945,nop,wscale 4>
20:44:54.293328 IP 10.199.202.30.54552 >
209.234.240.110.http: . ack 1 win 46 <nop,nop,timestamp 42948 691273371>
20:44:54.293364 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 42948
691273371>
20:44:54.503495 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 43001
691273371>
20:44:54.927490 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 43107
691273371>
20:44:55.775506 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 43319
691273371>
20:44:57.471480 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 43743
691273371>
20:45:00.863480 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 44591
691273371>
20:45:07.647492 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 46287
691273371>
20:45:21.215432 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 49679
691273371>
20:45:48.351330 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 56463
691273371>
20:45:53.351293 arp who-has 10.199.202.1 tell
10.199.202.30
20:45:53.356047 arp reply 10.199.202.1 is-at
00:13:80:68:29:42 (oui Unknown)
20:46:42.623433 IP 10.199.202.30.54552 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 70031
691273371>
20:46:47.623417 arp who-has 10.199.202.1 tell
10.199.202.30
20:46:47.624018 arp reply 10.199.202.1 is-at
00:13:80:68:29:42 (oui Unknown)
20:46:54.285323 IP 209.234.240.110.http >
10.199.202.30.54552: F 1:1(0) ack 1 win 362 <nop,nop,timestamp 691303373
42948>
20:46:54.285446 IP 10.199.202.30.54552 >
209.234.240.110.http: F 112:112(0) ack 2 win 46 <nop,nop,timestamp 72946
691303373>
20:46:54.525563 IP 209.234.240.110.http >
10.199.202.30.54552: F 1:1(0) ack 1 win 362 <nop,nop,timestamp 691303430
42948>
20:46:54.525598 IP 10.199.202.30.54552 >
209.234.240.110.http: . ack 2 win 46 <nop,nop,timestamp 73006
691303430,nop,nop,sack 1 {1:2}>
20:46:55.287500 IP 10.199.202.30.54553 >
209.234.240.110.http: S 3509843207:3509843207(0) win 5840 <mss
1460,sackOK,timestamp 73197 0,nop,wscale 7>
20:46:55.298652 IP 209.234.240.110.http >
10.199.202.30.54553: S 1598225475:1598225475(0) ack 3509843208 win 5792 <mss
1380,sackOK,timestamp 691303622 73197,nop,wscale 4>
20:46:55.298687 IP 10.199.202.30.54553 >
209.234.240.110.http: . ack 1 win 46 <nop,nop,timestamp 73199 691303622>
20:46:55.298715 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 73199
691303622>
20:46:55.298805 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:46:55.503442 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 73251
691303622>
20:46:55.503540 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:46:55.919450 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 73355
691303622>
20:46:55.919549 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:46:56.751474 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 73563
691303622>
20:46:56.751569 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:46:58.415447 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 73979
691303622>
20:46:58.415556 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:47:01.743469 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 74811
691303622>
20:47:01.743565 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:47:08.399483 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 76475
691303622>
20:47:08.399601 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:47:21.711506 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 79803
691303622>
20:47:21.711619 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:47:48.335576 IP 10.199.202.30.54553 >
209.234.240.110.http: P 1:112(111) ack 1 win 46 <nop,nop,timestamp 86459
691303622>
20:47:48.335683 IP 209.234.240.110.http >
10.199.202.30.54553: P ack 1 win 5792
20:47:53.335556 arp who-has 10.199.202.1 tell 10.199.202.30
20:47:53.336126 arp reply 10.199.202.1 is-at
00:13:80:68:29:42 (oui Unknown)
