Secure LXC networking Marian HackMan Marinov CEO of 1H Ltd. - PowerPoint PPT Presentation

Secure LXC networking Marian HackMan Marinov CEO of 1H Ltd. <mm@1h.com> CTO of GetClouder.com

Who am I? ➢ System Administrator since 1998 ➢ CEO of 1H Ltd. ➢ CTO of GetClouder Ltd. ➢ Head of DevOps for Siteground.com ➢ Organizer of OpenFest, BG Perl workshops and others ➢ This year I helped with the organization of YAPC europe and EuroBSDcon in Sofia ➢ In my spare time I teach Linux System Administration and Network Security courses in Sofia University ➢ For the past year I'm playing mainly with containers!

We don't really need networking...

MAC addresses ➢ Keep a central DB with all MAC addresses to prevent collisions ➢ Use a reliable way to generate MAC addresses ➢ Keep in mind: ➢ local or global ➢ unicast or multicast

generate MAC address in bash function gen_mac() { mac_vars=(0 1 2 3 4 5 6 7 8 9 a b c d e f) mac_base='52:00:01:' ret='' for i in {1..6}; do n=$RANDOM let 'n %= 16' ret="${ret}${mac_vars[$n]}" if [ $i -eq 2 ] || [ $i -eq 4 ]; then ret="${ret}:" fi done echo "${mac_base}${ret}" }

generate mac address in PLPGSQL CREATE OR REPLACE FUNCTION generate_mac() RETURNS text LANGUAGE plpgsql AS $$DECLARE mac TEXT; a CHAR; count INTEGER; BEGIN mac='52:00:01:'; FOR count IN 1..6 LOOP SELECT substring('0123456789abcdef' FROM (random()*16)::int + 1 FOR 1) INTO a; -- This fixes an issue, where the above SELECT returns NULL or empty string -- If for some reason we concatenate with a NULL string, the result will be NULL string WHILE a IS NULL OR a = '' LOOP SELECT substring('0123456789abcdef' FROM (random()*16)::int + 1 FOR 1) INTO a; END LOOP; mac = mac || a; IF count = 2 OR count = 4 THEN mac = mac || ':'; END IF; END LOOP; RETURN mac; END;$$;

generate MAC address in Python #/usr/bin/python import random mac = [random.choice(range(256)) for i in range(6)] mac[0] |= 0x02 mac[0] &= 0xfe print ':'.join('%02x' % m for m in mac)

Types of LXC networking ➢ none ➢ empty ➢ macvlan ➢ macvtap (did not have time to test it) ➢ veth (linux or ovs bridge) ➢ vlan ➢ physical ➢ VPN device(haven't tried that either)

None lxc.network.type = none lxc.network.hwaddr = 00:16:3a:61:45:a6 lxc.network.flags = up

Empty lxc.network.type = empty lxc.network.hwaddr = 00:16:3a:61:45:a6 lxc.network.flags = up

VETH lxc.network.type = veth lxc.network.veth.pair = vethc3070 lxc.network.flags = up lxc.network.name = eth0 lxc.network.ipv4 = X.X.X.X/24 lxc.network.ipv4.gateway = X.X.X.1 lxc.network.hwaddr = 00:16:3e:28:73:b3

VETH lxc.network.veth.pair = vethc3070 11: vethD6YPJ1: <BROADCAST,MULTICAST,PROMISC,UP,LOWE R_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP qlen 1000 link/ether f2:0:32:02:55:2f brd ff:ff:ff:ff:ff:ff valid_lft forever preferred_lft forever

MACVLAN lxc.network.type = macvlan lxc.network.macvlan.mode = bridge lxc.network.flags = up lxc.network.link = lxcbr1 lxc.network.name = eth0 lxc.network.ipv4 = X.X.X.X/24 lxc.network.ipv4.gateway = X.X.X.1 lxc.network.hwaddr = 00:16:3e:28:73:b3

MACVLAN ➢ If you want to manually setup the networking ip link add link PARENT [NAME] type macvlan [address MAC] ➢ Auto generated MAC adresses # ip link add link eth0 lxc0 type macvlan ➢ Manually assigned # ip link add link eth0 lxc1 type macvlan address f0:de:f1:81:0a:2a ➢ Additional parameter: mode ➢ macvlan mode { private | vepa | bridge | passthru }

MACVLAN ➢ private (filter all incoming packets) ➢ bridge (all packets on the same iface can be seen from all vlans) ➢ pasthru (requires enabled STP) ➢ VEPA (Virtual Ethernet Port Aggregator)

MACVLAN ➢ Edge Virtual Bridging EVB ➢ Top-of-Rack (ToR) ➢ End-of-Row (EoR) ➢ Virtual Ethernet Bridge (VEB) ➢ Linux bridge ➢ OpenVswitch ➢ Virtual Ethernet Port Aggregator (VEPA) ➢ used for EVB ➢ VEPA 802.1Qab - HP, IBM, Brocade, Juniper ➢ Standard mode ➢ Multi-channel VEPA (Q-in-Q) ➢ VN-Tag 802.1Qbh - Cisco

VLAN lxc.network.type = vlan lxc.network.vlan.id = 10 lxc.network.flags = up lxc.network.link = eth0 lxc.network.name = eth0 lxc.network.ipv4 = X.X.X.X/24 lxc.network.ipv4.gateway = X.X.X.1 lxc.network.hwaddr = 00:16:3e:28:73:b3

VLAN # vconfig add eth0 10 # ip link add link eth0 vlan10 type vlan id 10 # ip link show vlan10 10: vlan10@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT link/ether f0:de:f1:81:0a:2a brd ff:ff:ff:ff:ff:ff

Physical lxc.network.type = phys lxc.network.flags = up lxc.network.link = eth2 lxc.network.name = eth0 lxc.network.ipv4 = X.X.X.X/24 lxc.network.ipv4.gateway = X.X.X.1 lxc.network.hwaddr = 00:16:3e:28:73:b3

Bridging :) ➢ Linux Bridge ➢ setup with brctl ➢ setup with ip route ➢ OpenVswitch (OVS) ➢ setup with its tools

Bridging :) ➢ What is OpenVswitch ➢ multilayer virtual switch ➢ Why OpenVswitch ➢ greater flexibility ➢ more control over the traffic ➢ native VXLAN support

Bridging :) # brctl show bridge name bridge id STP enabled interfaces # brctl addbr br0 # brctl show bridge name bridge id STP enabled interfaces br0 8000.000000000000 no

Bridging :) # brctl addif br0 eth0 # brctl show bridge name bridge id STP interfaces br0 8000.f0def1810a2a no eth0 adding a veth device # brctl addif br0 vethc3070 adding a vlan # brctl addif br0 eth0.4

Bridging :) # ip link add name lxcbr0 type bridge # ip link set dev lxcbr0 up # ip link show lxcbr0 7: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT link/ether fe:d8:b2:55:ce:5b brd ff:ff:ff:ff:ff:ff # ip link set dev eth0 promisc on # ip link set dev eth0 up # ip link set dev eth0 master bridge_name # ip link set dev eth0 nomaster

Securing all of these ➢ Do not allow traffic out of the container with MAC address that was not assigned to the container ➢ Do not allow traffic out of the container with IP address that was not assigned to the container ➢ Do not allow multicast traffic to go to container which is not part of the multicast group ➢ Actually if possible allow network traffic only to its gateway :)

Securing all of these ➢ Do not use NAT for connecting your containers ➢ NAT is susceptible to DoS. By spoofing many connections from one container can block the connectivity of the whole machine!

Broadcasts... ➢ It depends on your network design ➢ Generally limit the broadcast destinations that a container can reach ➢ If possible use source routing to route the traffic directly to where it is supposed to go

OpenVswitch security ➢ Implement OpenFlow rules to enforce the previous rules ➢ For each container hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=150 dl_type=0x0800 in_port=$input_port nw_dst=$container_gw actions=NORMAL hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=100 dl_type=0x0800 in_port=$input_port nw_dst=$container_network actions=drop hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac nw_src=$container_ip dl_type=0x0806 priority=50 actions=NORMAL hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac dl_type=0x0800 nw_src=$container_ip priority=25 actions=NORMAL hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=20 dl_type=0x0800 dl_src=$container_mac nw_dst=$container_ip actions=output:$input_port hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port priority=5 actions=drop ➢ For each additional IP on the container hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_type=0x0800 dl_src=$container_mac nw_src=$additional_ip priority=10 actions=NORMAL hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac nw_src=$additional_ip dl_type=0x0806 priority=50 actions=NORMAL

OpenVswitch security ➢ OpenVswitch networking DOES NOT go trough the normal linux networking so you CAN NOT use ipatables/ebtables to limit the traffic ➢ Even if you use net_cls it still DON'T WORK

Thank you! Thank you! Questions? Questions? Marian Marinov <mm@1h.com> http://getclouder.com Jabber: hackman@jabber.org IRC: irc.freenode.net HackMan ICQ: 7556201 GitHub: http://github.com/hackman