Cloud Infra Opensource…

Network outage simulation in libvirt using netfilter

2024-09-23T16:00:00-05:00

As I tend to work in cluster configuration and administration one of the most tricky tasks is how to test network outage in a virtualized environment.

One solution is to disable the network interface or set firewall rules but this is more of a hack as we are configuring this in the OS. A better approach is to use the libvirt netfilter which is configured at the hypervisor.

As an example I will block unicast traffic between two KVMs using the sender MAC address.

I will define my filter in an XML file and name it block_unicast_traffic.xml

<filter name='block-unicast-traffic' chain='ipv4'>
  <rule action='drop' direction='inout'>
      <ip protocol='udp' srcmacaddr='$SENDER_MAC' />
  </rule>
</filter>

The filter will use the ipv4 protocol-specific filtering chain and we will use a simple rule to drop packages that originate from a specific MAC address. We are defining the MAC address by using the custom $SENDER_MAC parameter.

The next step is to create the netfilter in libvirt using virsh and nwfilter-define.

virsh # nwfilter-define --file /home/ppetrou/Dev/libvirt_netfilter_samples/block_unicast_traffic.xml
Network filter block-unicast-traffic defined from /home/ppetrou/Dev/libvirt_netfilter_samples/block_unicast_traffic.xml

Next step is to create a netfilter binding as we need to bind the filter with a network interface of a KVM. We could do this just by adding a filterref element in the domain XML in the interface element but this would make things complex if we wanted to add/remove the filter on the fly without having to reboot the KVM.

<devices>
  <interface type='bridge'>
    <mac address='00:16:3e:5d:c7:9e'/>
    <filterref filter='clean-traffic'/>
  </interface>
</devices>

The filter binding can be added and removed from a KVM when the KVM is powered on so this can ease our testing on when connectivity is lost or resumed.

In the owner element we MUST define the KVM name and its UUID. You can get this from the domain XML of the KVM. The portdev element is mandatory and defines the virtual network port name of the interface we want to bind the filter. The mac element is the mac address of the interface we want to bind the filter.

The most important part is the filterref element were we define the filter and also initialize any parameters. In this case we need to initialize the $SENDER_MAC parameter with the MAC address of the sender.

<filterbinding>
  <owner>
    <name>vm1</name>
    <uuid>80d59483-aec0-4d9f-b5f0-55f39ad1871e</uuid>
  </owner>
  <portdev name='vnet1'/>
  <mac address='52:54:00:83:73:8a'/>
  <filterref filter='block-unicast-traffic'>
    <parameter name='SENDER_MAC' value='52:54:00:fc:ee:d4'/>
  </filterref>
</filterbinding>

In order to test this we will use two KVMs. The network interface which we will use is the enp7s0 and has the portdev name vnet1.

[root@node1 ~]# 
[root@node1 ~]# nmcli con show
NAME                UUID                                  TYPE      DEVICE 
enp1s0              8660f145-f5c6-46c1-995f-33460a027c47  ethernet  enp1s0 
Wired connection 1  51fdfc6c-df5e-35f3-a7bb-c11bf2c18676  ethernet  enp7s0 
Wired connection 2  abf63925-a804-391c-859a-498f6d1bb4fe  ethernet  enp8s0 
[root@node1 ~]# 
[root@node1 ~]# ip a show enp7s0
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:83:73:8a brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.182/24 brd 192.168.100.255 scope global dynamic noprefixroute enp7s0
       valid_lft 2560sec preferred_lft 2560sec
    inet6 fe80::88ca:707f:1bc3:d496/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@node1 ~]# 
[root@node1 ~]# 

[root@node2 ~]# nmcli con show
NAME                UUID                                  TYPE      DEVICE 
enp1s0              8660f145-f5c6-46c1-995f-33460a027c47  ethernet  enp1s0 
Wired connection 1  9500b4e5-f052-32b0-8cd4-0557250f55b0  ethernet  enp7s0 
Wired connection 2  50ed8da0-0cf7-34f1-a5d4-46b17d3f593c  ethernet  enp8s0 
[root@node2 ~]# 
[root@node2 ~]# ip a show enp7s0
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:fc:ee:d4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.201/24 brd 192.168.100.255 scope global dynamic noprefixroute enp7s0
       valid_lft 2479sec preferred_lft 2479sec
    inet6 fe80::daae:eb36:f3c3:d2e8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@node2 ~]# 
[root@node2 ~]# 

First we wil test that connectivity is OK.

[root@node2 ~]# nc -vu 192.168.100.182 2233
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.100.182:2233.
test


[root@node1 ~]# nc -lvu 192.168.100.182 2233
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Listening on 192.168.100.182:2233
Ncat: Connection from 192.168.100.201.
test

and then we will apply the netfilter binding and re-test to check that connectivity is lost

virsh # nwfilter-binding-create --file /home/ppetrou/Dev/libvirt_netfilter_samples/block_unicast_binding_vm1.xml 
Network filter binding on vnet1 created from /home/ppetrou/Dev/libvirt_netfilter_samples/block_unicast_binding_vm1.xml

[root@node2 ~]# nc -vu 192.168.100.182 2233
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.100.182:2233.
test
test2 <-- this message never arrives to node1
test3 <-- same this one
test4 <-- same this one

[root@node1 ~]# nc -lvu 192.168.100.182 2233
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Listening on 192.168.100.182:2233
Ncat: Connection from 192.168.100.201.
test
     <-- no new messages

Now we will remove the netfilter binding

virsh # nwfilter-binding-list 
 Port Dev   Filter
-----------------------------------
 vnet1      block-unicast-traffic

virsh # 
virsh # nwfilter-binding-delete --binding vnet1 
Network filter binding on vnet1 deleted

[root@node2 ~]# nc -vu 192.168.100.182 2233
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.100.182:2233.
test
test2
test3
test4
test5 <-- newer message is received in node1

[root@node1 ~]# nc -lvu 192.168.100.182 2233
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Listening on 192.168.100.182:2233
Ncat: Connection from 192.168.100.201.
test
test5 <-- newer message

You can find the XML artefacts in my GitHub.

That’s it. I hope you found this blog useful.

Thank you,

Petros

Who does the DNS Lookup? The browser or the operating system?

2022-10-10T14:00:00-05:00

A common interview question for infrastructure roles is “Who does the DNS Lookup? The browser or the Operating System”. A few years ago when I had this question from a UK FS Consultancy, my mind started spinning and was doing endless loops on what could be the correct answer. As a passionate programmer I tend to overthink and my mind generates lots of possible solutions so I was not sure. I ended up replying “The Browser” and since this day I always wanted to research this further and see if this was correct or not :)

In this blog I will present my analysis on this and walk you through the DNS lookup process.

First we need to analyse each term in the question:

Who does? -> Do we mean initiate the call or do the actual lookup on the nameserver?
DNS Lookup -> An API call that takes as an argument a URL (e.g. petrospetrou.co.uk) and returns its IP address (e.g. 173.236.127.52).
Browser -> An application which provides access to the World Wide Web
Operating System -> This can be a bit complex to define. Do we mean just the kernel? Do we include standard libraries? More on this towards the end.

Now lets see how a DNS Lookup works…

When an application such as a web browser needs to resolve a hostname to an IP address it requires a DNS Lookup. This will query a name server and return the IP address of the hostname. A typical example is the dig utility.

[ppetrou@fedora root]$
[ppetrou@fedora root]$ dig www.petrospetrou.co.uk

; <<>> DiG 9.16.33-RH <<>> www.petrospetrou.co.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58067
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.petrospetrou.co.uk.		IN	A

;; ANSWER SECTION:
www.petrospetrou.co.uk.	672	IN	CNAME	petrospetrou.co.uk.
petrospetrou.co.uk.	146	IN	A	173.236.127.52

;; Query time: 0 msec
;; SERVER: 192.168.2.1#53(192.168.2.1)
;; WHEN: Sat Oct 08 21:00:56 BST 2022
;; MSG SIZE  rcvd: 99

But how is the DNS Lookup implemented?

In order to explain this we will see Linux as the OS and C as the language that the application is implemented. Something similar exists in other languages and Operating Systems but we have no access to their source code so we cannot elaborate further.

The Standard C Library has a header file netdb.h with definitions for network database operations. This is part of the resolver library which includes the following two methods.

gethostbyname()
gethostbyaddr()

When an application requires a DNS Lookup it can use the gethostbyname() function in order to resolve the hostname. You can find a simple C program that does this here.

In order to see this in a known application we can search the source code of the text based Lynx browser. I have no time to analyse the code properly but we can see that there are multiple references to the gethostbyname() function.

ppetrou@fedora lynx2.9.0dev.10]$ grep -R gethostbyname *
.
.
WWW/Library/Implementation/HTTCP.c:	gbl_phost = gethostbyname(host);
WWW/Library/Implementation/HTTCP.c:    gbl_phost = gethostbyname(host);
WWW/Library/Implementation/HTTCP.c:     * fork-based gethostbyname() with checks for interrupts.
WWW/Library/Implementation/HTTCP.c:			HTInetStatus("CHILD gethostbyname");
WWW/Library/Implementation/HTTCP.c:	    CTRACE((tfp, "%s: INTERRUPTED gethostbyname.\n", this_func));
WWW/Library/Implementation/HTTCP.c:static void really_gethostbyname(const char *host,
WWW/Library/Implementation/HTTCP.c:    phost = gethostbyname(host);
WWW/Library/Implementation/HTTCP.c:    CTRACE((tfp, "really_gethostbyname() returned %d\n", phost));
WWW/Library/Implementation/HTTCP.c:    dump_hostent("CHILD gethostbyname", phost);
WWW/Library/Implementation/HTTCP.c:/*	Resolve an internet hostname, like gethostbyname
WWW/Library/Implementation/HTTCP.c: *  gethostbyname(), except for the following:
WWW/Library/Implementation/HTTCP.c: *  If NSL_FORK is not used, the result of gethostbyname is returned
WWW/Library/Implementation/HTTCP.c:#ifdef MVS			/* Outstanding problem with crash in MVS gethostbyname */
WWW/Library/Implementation/HTTCP.c:    CTRACE((tfp, "%s: Calling gethostbyname(%s)\n", this_func, host));
WWW/Library/Implementation/HTTCP.c:    if (!setup_nsl_fork(really_gethostbyname,
WWW/Library/Implementation/HTTCP.c:		gbl_phost = gethostbyname(host);
WWW/Library/Implementation/HTTCP.c:	phost = gethostbyname(host);	/* See netdb.h */
WWW/Library/Implementation/HTTCP.c:	CTRACE((tfp, "%s: gethostbyname() returned %d\n", this_func, phost));
WWW/Library/Implementation/HTTCP.c:#ifdef MVS			/* Outstanding problem with crash in MVS gethostbyname */
WWW/Library/Implementation/HTTCP.c:    phost = gethostbyname(name);	/* See netdb.h */
WWW/Library/Implementation/HTTCP.h:/*	Resolve an internet hostname, like gethostbyname
WWW/Library/Implementation/HTTCP.h: *  The interface is intended to be the same as for gethostbyname(),

Taking things a bit further we can also search for the same in the linux kernel source code and see if there are any references there.

[ppetrou@fedora linux]$ ls
arch   certs    CREDITS  Documentation  fs       init      ipc     Kconfig  lib       MAINTAINERS  mm   README   scripts   sound  usr
block  COPYING  crypto   drivers        include  io_uring  Kbuild  kernel   LICENSES  Makefile     net  samples  security  tools  virt
[ppetrou@fedora linux]$
[ppetrou@fedora linux]$ grep -R gethostbyname *
[ppetrou@fedora linux]$

As expected no references exist so the browser does the DNS Lookup… or NOT?? :)

If we go back to the beginning of the blog where we analysed the terms of the question we see that we raised of few ambiguities…

If we consider that “does” means initiates then yes the browser does the DNS Lookup as the gethostbyname() function is called within the browser code. On the contrary if “does” means the lookup of the host in the nameserver then this gets executed by the resolver which is part of the Standard C Library and is considered part of the operating system. It also cannot be removed as it includes lots of libraries such as the resolver or sudo, which could make the OS unusable. See below.

[root@lab1 ~]# sudo dnf remove glibc
Error:
 Problem: The operation would result in removing the following protected packages: sudo
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@lab1 ~]#

So the call sequence from the browser to the name server is the following:

Web browser -> glibc -> resolver -> gethostbyname() -> nameserver

Also what we have proved from this analysis is that the kernel DOES NOT do the DNS Lookup in any case!

I hope you enjoyed reading this blog as much I did when I was researching the topic. If you have any comments please leave them below.

Thank you,

Petros

How to write vmware pyvmomi mocked unit tests

2021-06-13T13:00:00-05:00

I have recently been doing some vmware api development work using pyvmomi and one of the issues I had is how we can unit test without a vSphere Installation. The solution is to mock the pyvmomi api but it turned out that it was a much more complex task than I initially thought, so here are my findings.

I will not go into depth of the pyvmomi api or the python mock object library as I assume anyone who will need to mock it would have the sufficient knowledge of both.

The pyvmomi wrapper class I have developed is VmwareUtil and I am using the context manager VmwareOpen to manage the lifecycle of the pyVim.connect.SmartConnectNoSSL which provides a vim.ServiceInstance. This is the root object of the vCenter/ESXi inventory.

from pyVim.connect import SmartConnectNoSSL, Disconnect
from pyVmomi import vim, vmodl

class VmwareOpen(object):

    def __init__(self, host, port, username, password):
        self._host = host
        self._port = port
        self._username = username
        self._password = password

    def __enter__(self):
        try:
            self._service_instance = SmartConnectNoSSL(host=self._host, user=self._username, pwd=self._password, port=self._port)
            return self._service_instance
        except IOError as e:
            raise e

    def __exit__(self, type, value, traceback):
        Disconnect(self._service_instance)

class VmwareUtil(object):
    
    def __init__(self, vmwarehost, vmwareport, username, password):
        self._host = vmwarehost
        self._port = vmwareport
        self._username = username
        self._password = password
    
    def _get_vm(self, identity, service_instance):
        content = service_instance.RetrieveContent()
        
        vmlist = []
        
        for child in content.rootFolder.childEntity:
            if hasattr(child, 'vmFolder'):
                datacenter = child
                vmfolder = datacenter.vmFolder
                vmlist = vmfolder.childEntity
                
        for vm in vmlist:
            if vm.summary.config.name == identity or vm.summary.config.uuid == identity:
                return vm
        
        raise Exception(identity + " not found.")
    
    def get_uuid(self, identity):
        with VmwareOpen(self._host, self._port, self._username, self._password) as service_instance:
            vm = self._get_vm(identity, service_instance)
            return vm.summary.config.uuid
    
def tryme():
    vmware_util = VmwareUtil("192.168.122.168", "443", "root", "password")
    vmuuid = vmware_util.get_uuid("test_vm_00")
    print("UUID : " + vmuuid)

if __name__ == "__main__":
    tryme()

As the simplest example I will just implement a method which will return the UUID of a Virtual Machine by passing its name.

In order to do this we first need to access the Service Instance Content vim.ServiceInstanceContent using the RetrieveContent method of the vim.ServiceInstance.

This will give us access to a set of Managed Objects in the inventory. The rootFolder is the root of the inventory and we need to look for the datacenter object vim.Datacenter. This would be the child which has the attribute vmFolder. There should be a more efficient way of doing this such as using the instanceof function but I kept the lookup code as it was in the pyvmomi community samples. The last step is to loop the children of the vmFolder and find the vm we look for based on name or uuid.

Now the hard stuff and the mocking bit.

The main issue I had is that all the pyvmomi objects are server managed so we cannot just instantiate and populate them at client side. This is why we need to mock the whole lot.

import unittest
from unittest import mock
from unittest.mock import patch

from vmware_util import VmwareUtil

from pyVim.connect import SmartConnectNoSSL, Disconnect
from pyVmomi import vim, vmodl

class TestVMwareUtil(unittest.TestCase):
    
    @mock.patch('vmware_util.VmwareOpen')
    def test_uuid(self, vmware_mock):
        
        # Initialize the Config Summary
        test_vm_00_config_summary = mock.MagicMock(spec_set=vim.vm.Summary.ConfigSummary())
        test_vm_00_config_summary.name = "test_vm_00"
        test_vm_00_config_summary.uuid = "c7a5fdbd-cdaf-9455-926a-d65c16db1809"
         
        # Initialize VM Summary 
        test_vm_00_summary = mock.MagicMock(spec_set=vim.vm.Summary())
        test_vm_00_summary.config = test_vm_00_config_summary
        
        # Initialize the Mock VM
        test_vm_00 = mock.MagicMock(spec_set=vim.VirtualMachine("vm-41"))
        test_vm_00.summary = test_vm_00_summary
        
        # Initialize the Data Center and add the Mock VM
        # Create the vmFolder       
        vm_folder_mock = mock.MagicMock(spec_set=vim.Folder("vm-folder"))
        vm_folder_mock.childType = { "vim.Folder", "vim.Virtualmachine", "vim.VirtualApp" }       
        # Add it to the Data Center and add the VM in the children
        ds_mock = mock.MagicMock(spec_set=vim.Datacenter("ds-00"))
        ds_mock.vmFolder = vm_folder_mock
        ds_mock.vmFolder.childEntity = [test_vm_00]
        
        # Initialize the Service Instance Mock and add the Data Center
        # Create the rootFolder
        root_folder_mock = mock.MagicMock(spec_set=vim.Folder("root-folder"))
        root_folder_mock.childEntity = [ds_mock]
        # Create the Service Instance and add the rootFolder
        si_mock = mock.MagicMock(spec_set=vim.ServiceInstance("si-00"))
        si_content_mock = mock.MagicMock(spec_set=vim.ServiceInstanceContent())
        si_content_mock.rootFolder = root_folder_mock
        # Mock the RetrieveContent Methos in the service instance
        si_mock.RetrieveContent = mock.MagicMock(return_value=si_content_mock)
        #si_mock.RetrieveContent.side_effect = Exception('Boom!')
        
        # Get the VmwareOpen reference
        vm_mock = vmware_mock.return_value
        # Mock the enter and exit methods
        vm_mock.__enter__ = mock.MagicMock(return_value=si_mock)
        vm_mock.__exit__ = mock.MagicMock(return_value=None)
        
        # Call the VMWare Util as normal. Any vmware_util.VmwareOpen reference within the function call will be replaced by the mocked object.
        vmware_util = VmwareUtil("", "", "", "")
        returned_uuid = vmware_util.get_uuid("test_vm_00")
         
        # Assert
        self.assertEqual('c7a5fdbd-cdaf-9455-926a-d65c16db1809', returned_uuid)


if __name__ == '__main__':
    unittest.main()

The hierarchy inside the ServiceInstanceContent is the following so we need to mock in reverve order.

.
+-- ServiceInstanceContent
    +-- rootFolder
        +-- dataCenter
            +-- vmFolder
                +-- childEntity
                    +-- VirtualMachine

The next step is to mock the RetrieveContent method and assign the mocked ServiceInstanceContent as its returned object and the last step is to mock the __enter__ method of the vmware_util.VmwareOpen class which is the context manager for the ServiceInstance. We use the mock.patch function decorator @mock.patch('vmware_util.VmwareOpen') in the test_uuid() function so every time we ran the unit test, any reference to the vmware_util.VmwareOpen will be replaced with the patched object so there will be no real call to the VMWare Hypervisor but the mocked object will be returned instead. The vmware_util.VmwareOpen reference is at with VmwareOpen(self._host, self._port, self._username, self._password) as service_instance:.

You can find the code listings in my Github

That’s it, I hope you find this useful and if you have a better way of doing it please leave a comment below.

Thank you,

Petros

Hello World!

2020-10-22T17:16:34-05:00

Hi, my name is Petros Petrou and this is my first blog post. If you want to know more about me have a look in About. I will try and keep this space active with stories and technical articles from my workplace and also my home lab.