[Xen-users] Communication problem with virtual DMZ

["Diego Alvarez" <arcane.lord@xxxxxxxxx>]

Hi all,
I am running Xen 3.0.2-2 (taken from XenSource) with Linux kernel
2.6.16 (taken from Debian Sid), I compiled Xen and 2 kernels
(dom0 and domU).

Here is the ascii-art of my setup:

  ------------                        -------------
  |    LAN   |------------------------|   waste   | 192.168.0.94/24
  ------------                        -------------
       |
·····························
·      |                    ·
·      |        Dom0        ·
·      |                    ·
·  ---------                ·    ································
·  | peth0 |                ·    ·                              ·
·  ---------                ·    ·         DomU hades           ·
·      |                    ·    ·         (Firewall)           ·
·      |                    ·    ·                              ·
· -----------   ----------  ·    ·    --------                  ·
· | br-inet |---| hades0 |============| eth0 |  192.168.0.34/24 ·
· -----------   |  (vif) |  ·    ·    --------                  ·
·      |        ----------  ·    ·                              ·
·      |                    ·    ·    --------                  ·
· -----------               ·    ·    | eth1 |  192.168.0.34/32 ·
· | vif0.0  |               ·    ·    --------                  ·
· -----------               ·    ·       ||                     ·
·     ||                    ·    ········||······················
·     ||                    ·            ||
·     ||                    ·············||···········
·     ||                                 ||          ·
·  --------                              ||          ·
·  | eth0 | 192.168.0.22/24          ----------      ·
·  --------                          | hades1 |      ·
·                                    |  (vif) |      ·
·                                    ----------      ·
·                                        |           ·
·                -----------         ----------      ·
·                | pdummy0 |---------| br-dmz |      ·
·                -----------         ----------      ·
·                                        |           ·
·                                   -------------    ·
·                                   | agustina0 |    ·
·                                   |   (vif)   |    ·
·                                   -------------    ·
·                                        ||          ·
·                                        ||          ·
·········································||···········
                                         ||
                      ·-·-·-·-·-·-·-·-·-·||·-·-·-·-·-·-·-·-·
                      ·                  ||                ·
                      |        ··········||········        |
                      ·        ·         ||       ·        ·
                      |        ·     --------     ·        |
                      ·        ·     | eth0 |     ·        ·
                      |        ·     --------     ·        |
                      ·        · 192.168.0.39/32  ·        ·
                      |        ·                  ·        |
                      ·        ·   DomU agustina  ·        ·
                      |        ·    (DMZ Server)  ·        |
                      ·        ·                  ·        ·
                      |        ····················        |
                      ·                                    ·
                      |             Virtual DMZ            |
                      ·                                    ·
                      ·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-

--------

Network configuration for Dom0:

auto eth0
iface eth0 inet static
        address 192.168.0.22
        netmask 255.255.255.0
        gateway 192.168.0.2

auto dummy0
iface dummy0 inet static
        address 10.1.1.1
        netmask 255.255.255.255
        up ifconfig dummy0 0.0.0.0 up

--------

Network configuration for DomU agustina (DMZ Server):

auto eth0
iface eth0 inet static
        address 192.168.0.39
        netmask 255.255.255.255
        up route add -host 192.168.0.34 dev eth0
        up route add default gw 192.168.0.34 dev eth0

-------

Network configuration for DomU hades (Firewall):

auto eth0
iface eth0 inet static
  address 192.168.0.34
  netmask 255.255.255.0
  gateway 192.168.0.2
  up arp -Ds 192.168.0.39 eth0 pub

auto eth1
iface eth1 inet static
  address 192.168.0.34
  netmask 255.255.255.255
  up route add -host 192.168.0.39 dev eth1

It also have ip_forward activated by sysctl

------

In dom0, I do the following things:

In /etc/xen/xend-config.sxp I have:

(network-script 'network-bridge bridge=br-inet')
(vif-script 'vif-bridge bridge=br-inet')


I also have a script which brings up br-dmz bridge on dummy0

# brctl show:
bridge name     bridge id               STP enabled     interfaces

br-dmz          8000.feffffffffff       no              agustina0
                                                        hades1
                                                        pdummy0

br-inet         8000.feffffffffff       no              hades0
                                                        peth0
                                                        vif0.0

Here is the configuration for hades and agustina:

/etc/xen/auto/hades:
name="hades"
memory=128
kernel="/boot/vmlinuz-2.6.16-xenU"
vif = [ 'mac=00:16:3e:00:01:01,bridge=br-inet,vifname=hades0',
'mac=00:16:3e:00:00:02,bridge=br-dmz,vifname=hades1' ]
disk=['phy:/dev/xen/hades-OS,hda1,w','phy:/dev/xen/hades-SWAP,hda2,w']
root="/dev/hda1 ro"
on_crash="restart"

/etc/xen/auto/agustina:
name="agustina"
memory=64
kernel="/boot/vmlinuz-2.6.16-xenU"
vif = [ 'mac=00:16:3e:00:00:07,bridge=br-dmz,vifname=agustina0' ]
disk=['phy:/dev/xen/Agustina-OS,hda1,w','phy:/dev/xen/Agustina-SWAP,hda2,w']
root="/dev/hda1 ro"
on_crash="restart"

-------

So.... what is the problem?
well:
 - routing is Ok
 - ping works in all directions
 - ssh from waste (lan machine) to Dom0 works
 - ssh from Dom0 to waste works
 - ssh from waste to hades works
 - ssh from hades to waste works
 - ssh from Dom0 to agustina works
 - ssh from hades to agustina works
 - ssh from agustina to Dom0 works
 - ssh from agustina to hades works

but:
 - ssh from waste to agustina does not work
 - ssh from agustina to waste does not work

Here are is a tcpdump taken from agustina's eth0:

agustina:~# tcpdump -i eth0 -n host waste
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:45:15.242301 IP waste.4331 > agustina.22: S 30038281:30038281(0)
win 5840 <mss 1460,sackOK,timestamp 23876432 0,nop,wscale 2>
19:45:15.251956 IP agustina.22 > waste.4331: S
3550608405:3550608405(0) ack 30038282 win 5792 <mss
1460,sackOK,timestamp 867394 23876432,nop, wscale 1>
19:45:15.245850 IP waste.4331 > agustina.22: . ack 1 win 1460
<nop,nop,timestamp 23876783 867394>
19:45:15.255867 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896
<nop,nop,timestamp 867394 23876783>
19:45:15.468349 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896
<nop,nop,timestamp 867417 23876783>
19:45:15.888650 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896
<nop,nop,timestamp 867459 23876783>
19:45:16.728328 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896
<nop,nop,timestamp 867543 23876783>
19:45:18.408341 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896
<nop,nop,timestamp 867711 23876783>
19:45:21.768338 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896
<nop,nop,timestamp 868047 23876783>
19:45:28.491449 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896
<nop,nop,timestamp 868719 23876783>

And that goes and goes until timeout.

Those packets from 'agustina' _are_ received by 'waste' in the same way
(I verified that with tcpdump too), and then are dropped by 'waste'
(netfilter conntrack say they are INVALID), so TCP socket is established,
but there is no communication.

The problem I see there is the tcp window size of agustina's reply,
which is bigger than waste first ACK packet, or I am wrong?

The strange thing is that agustina does not have any strange
configuration, and if I connect it to 'br-inet' bridge and change his
netmask and gateway, it work as expected.

There is no firewall on Dom[0U].

Does any of you have any idea of what could be the problem?

Regards,
Diego.

PS1: sorry for the large mail.

PS2: I have also tried packages from
http://packages.debianbase.de/sid/i386/xen3,
     with xen-3.0.1 and kernel 2.6.12, and have the same results.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users