Re: [Xen-users] issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

[spi@xxxxxxxxx]

Hi Hans

Thanks for your reply.

>>
>> I filed this issue with the Debian user-list as well but as I think it is 
>> not Debian related I file it here as well.
> 
> What you can do best (for the Debian part of it) is file a bug against
> src:xen in Debian to start a discussion.

Will do.

> 
> The behaviour will likely not change any more for Xen 4.8 in Stretch,
> but this is one of the things that should be improved for Xen 4.1X in
> Buster. (I'm one of the people working on that.)
> 

No issue with that. I just think something is not working here and
wasn't really sure, it was my understanding or really an issue.

>> I have issues with the on domU startup automatically generated
>> antispoofing rules by
>>
>> /etc/xen/scripts/vif-bridge and
>> /etc/xen/scripts/vif-common.sh
> 
> I myself use openvswitch instead of bridges, because it's so much more
> convenient with vlans and in that situation all of the rules do not make
> sense anyway, since since openvswitch traffic to domUs does not hit
> iptables.
> 

I am not familiar with openvswitch but like the idea of bridges as I can
filter traffic with iptables.


>> Both are part of the Debian xen-utils-common package
>> (4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 installed on Debian 9.4).
>>
>> A domU test01 has two virtual interfaces - vif-test01-INT and
>> vif-test01-TEST, both are connected to separate bridges named brINT and
>> brTEST. The brINT is just an internal bridge without connectivity to an
>> outside network to just connect all domUs and the dom0. The IP
>> addressfor the vif-test01-INT interface is 192.168.240.68.
>>
>>
>> The automatically generated rules per domU are:
>>
>> 1    ACCEPT     all  --  anywhere             anywhere            
>> PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
>> [...]
> 
> Because of the above, I have actually never studied this behaviour in
> detail, I've just always patched it out, postponing looking for a proper
> fix. (My network at work only contains domUs which are managed by
> ourselves, noone else has shell/root access on them).
> 

That's the case with me as well. All domUs are managed by trusted people
- me ;-) The issue though is another one. Some of the domUs are
connected to the Internet and host services. The risk here is that if
they get hacked an intruder got access to other internal machines as
well by manipulating domU's IP address. Think for instance of NFS
authentication based on IP addresses. One could argue here that a proper
NFS authentication is needed here and that's completely true, but NFSv4
with Kerberos tickets for domU file services sounds like a nightmare.
However dom0 and domU bring a perfect way for antispoofing rules - we
know and trust all the interfaces as they are managed by the dom0 and we
can apply filter on these. In a physical environment it is not that easy
to obtain. Binding IP addresses to MAC addresses doesn't address this
issue as MAC addresses easily can be spoofed too. So the equivalent in
the physical world would be one-to-one physical connections between
hosts and a firewall to bind IP addresses to real physical interfaces.
That's a huge benefit of virtualization. Antispoofing much easier to get.

>> From what I see is that the rules 1/3 and 5/7 are doubled.
>>
>> The next issue is that antispoofing rules just don't work. If I change
>> the ip adress of the vif-test01-INT interface to something like
>> 192.168.240.168 IP packets between test01 and other domUs are still
>> forwarded.
>>
>> If I manually change the iptables rules to something like (in this
>> example just for the brINT connected interface):
>>
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
>> -p all ! -s 192.168.240.68 -j DROP
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
>> -p all ! -d 192.168.240.68 -j DROP
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
>> -p all -j ACCEPT
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
>> -p all -j ACCEPT
>> ...
>> -A FORWARD -j REJECT --reject-with icmp-port-unreachable
>>
>> then antispoofing works and IP packets with IP addresses different then
>> 192.168.240.68 get dropped.
>>
>> Can somebody confirm this is an issue? Or do I just not understand how
>> the antispoofing rules work on a virtual bridge?
> 
> The general feeling I have is that this area indeed needs some work to
> clean it up and make it better. TBH, I have not looked back yet in xen
> version control to see what the history of these scripts and the
> iptables changes is.
> 

Honestly I think that that approach is a really good one if one uses
bridges. The issue though is that these automatically generated filters
don't work (and yes, I set the sysctl options...).

If looking for the generated filters they look good to me, they just
don't work. So it might be an iptables/bridges issue which is not really
XEN related. I am happy to file another report some place else but would
ask somebody to confirm that's really an issue which is not XEN related.

> Taking one step back, what are the things we want to do with whatever
> kind of rules in the dom0?
> * filtering irrelevant L2 unicast flooding
> * whitelisting mac addresses
> * whitelisting ip addresses
> * ?
> 

These three are definitely important. For me antispoofing is important
for the reason I laid out earlier, but that's what you probably mean by
whitelisting. Although whitelisting mac addresses is important in the
end (most of the) services rely on IP that's why protecting IP addresses
is so crucial.

> For bridges, some of these things can be done with iptables (if you also
> set the right sysctl options to actually make bridged traffic hit
> iptables...), but for filtering specific mac address / ip address etc
> you need to know about them in the dom0. For openvswitch, it can
> probably also be done in a different way. Then, what about properly
> cleaning up things on domU shutdown, what about live migration, other
> things...
> 

As said for bridges this approach works to set up and delete filters on
domU startup and shutdown as the to be assigned IP address is configured
on dom0 level in the domU config file. That IP address is being used for
the iptables filters and is outside the manageable scope of the domU.

> So, there's not really a good out of the box solution for this now
> afaics. We should find out what's there now, and why it was done like
> that in the past and then see how it can be improved.
> 
> Your experience, opinion, feedback etc. matters. Thanks for starting
> this thread. Best thing is to do the improvements "upstream" (here)
> instead of in Debian.
> 
>> Is there a way to diable generation of antispoofing rules automatically
>> on domU startup? I could configure a different vif.default.script in
>> xl.conf and write a wrapper script, but it might be easier to just
>> disable it and load iptables rules manually.
> 
> Remove/comment-out the handle_iptable line.

That was exactly my approach to handle these filters and create them
directly/manually with iptables. I think now I'll change the scripts to
create the iptables rules automatically but change the rules to the ones
I suggested which seem to work in my environment.

Cheers,
Sebastian

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users