{"type":"doc","content":[{"type":"heading","attrs":{"level":2},"content":[{"text":"Choosing a network card","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Depending on your needs, the network card will have to support several features","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PTP","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PCI-passthrough","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SR-IOV","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"None of these features are required by SEAPATH, however, PTP is required by the IEC 61850 standard.","type":"text"}]},{"type":"paragraph","content":[{"text":"If your machine doesn't have enough interfaces, you can use an usb-ethernet interface. For example, the TRENDnet TU3-ETG was tested and does work natively on SEAPATH. These added interfaces will not be compatible with any of the features described above.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"SEAPATH network on one hypervisor","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"The following description fits the case of a standalone hypervisor, but is also applicable to a machine in the cluster.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Physical interfaces","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"There are four different types of interfaces to consider on a SEAPATH hypervisor:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Administration: This interface is used to access the administration user with ssh and configure the hypervisor. It is also used by ansible.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"VM administration: This interface has the same role of the previous one, but for the virtual machines.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PTP interface: This interface is used to transmit PTP frames and synchronize the time on the machine.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"IEC61850 interface: This interface will receive the sample values and GOOSE messages.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"VM interfaces","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"There are three different ways to transmit data from a physical network interface to a virtual machine:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"virtio: A virtual NIC (Network Interface Card) is created inside the virtual machine. The data received on the hypervisor will be treated by the physical network card and then pass to the VM virtual card. This is the simplest method, but also the slowest.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PCI passthrough: The control of the entire physical network card is given to the VM. The received data is then directly treated by the virtual machine and does not go through the hypervisor. This method is much faster, but has two drawbacks:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The entire physical NIC is given to one VM. Other VMs or the hypervisor can’t use it anymore","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"It requires a compatible NIC","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SR-IOV: Single Root I/O Virtualization is a PCI Express Extended capability which makes one physical device appear as multiple virtual devices (more details ","type":"text"},{"text":"here","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"https://docs.kernel.org/PCI/pci-iov-howto.html"}}]},{"text":"). Multiple VMs can now use the same physical NIC. This is the fastest method, but it requires a compatible NIC.","type":"text"}]}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":749,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":822,"id":"595b6386-ed0b-4d52-911b-2c3b77e89da3","collection":"contentId-31821518","type":"file","height":359}}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Recommendations","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Many network configurations are possible using SEAPATH. Here are our recommendations:","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Administration","type":"text"}]},{"type":"paragraph","content":[{"text":"To avoid using too many interfaces, the administration of all VMs can use the same physical NIC. Hypervisor administration can also be done on this interface.","type":"text"},{"type":"hardBreak"},{"text":"This is possible by using a Linux or OVS bridge and connecting all VMs and the hypervisor to it.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"This behavior is achieved by the br0 bridge, preconfigured by Ansible (see Ansible inventories examples)","type":"text"}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"IEC 61-850 traffic","type":"text"}]},{"type":"paragraph","content":[{"text":"The Sample values and GOOSE messages should be received on an interface using PCI-passthrough or SR-IOV, the classic virtio interface is not fast enough.","type":"text"},{"type":"hardBreak"},{"text":"In this situation, all VMs receiving IEC 61-850 data must have one dedicated interface.","type":"text"}]},{"type":"paragraph","content":[{"text":"For testing purposes, to avoid using one interface per VM, these data can be received on classic interfaces using virtio. This is done by using a Linux or OVS bridge connected both to the physical NIC and to all the virtual machines.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"See the variables `bridges` and ovs_bridges` in Ansible inventories examples.","type":"text"}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"PTP","type":"text"}]},{"type":"paragraph","content":[{"text":"PTP synchronization does not generate much data. The synchronization of the hypervisor can thus be used:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Alone on a specific interface","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"On the administration interface","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Please remember that PTP requires specific NIC support.","type":"text"}]},{"type":"paragraph","content":[{"text":"In order to synchronize the virtual machines, the PTP clock of the host can be used (ptp_kvm). It is also possible to pass the PTP clock on the PCI-passthrough/SR-IOV interface given to the VM.","type":"text"}]},{"type":"paragraph","content":[{"text":"See the ","type":"text"},{"text":"Time synchronization","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"https://lf-energy.atlassian.net/wiki/display/SEAP/Time+synchronization"}}]},{"text":" page for more details.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Example of configuration","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Below is an example of a SEAPATH machine with two VMs. VM1 receives SV/GOOSE and must thus be synchronized with PTP and use PCI-passthrough. VM2 is just for monitoring, and does not require either PTP or PCI-passthrough.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"One interface is used for both VM and hypervisor administration","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"One interface to synchronize the host and VM1 in PTP","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"One interface for PCI-passthrough in VM1 handling IEC 61-850","type":"text"}]}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":330,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":261,"id":"2ce01ba9-98ed-47f8-ac84-866bd6a5831d","collection":"contentId-31821518","type":"file","height":342}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Connecting machines in a cluster","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"SEAPATH cluster is used to ensure redundancy of the machines (if one hypervisor fails) and of the network (if one link fails).","type":"text"}]},{"type":"paragraph","content":[{"text":"In order to connect all the machines together, two options are available:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Connecting all machines to two switches (the second is needed to ensure redundancy if the first switch fails)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Connecting all machines in a ring architecture","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The second method is recommended on SEAPATH because it doesn’t use an additional device (the switch). In that case, every machine is connected to it’s neighbors, forming a triangle. If one link breaks, the paquets still have another route to reach the targeted machine. See on ","type":"text"},{"text":"Github","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"https://github.com/seapath/seapath-architecture/blob/main/cluster-network.md"}}]},{"text":" for more information.","type":"text"}]},{"type":"paragraph","content":[{"text":"In that case, two more interfaces per machines are needed, which leads to a minimum of four interfaces per machine (two for the cluster, one for the administration, and one for IEC 61-850 traffic)","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Observer ","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"An observer machine doesn’t run any virtual machines. It is only used in the cluster to discriminate if a problem occurs between the two hypervisors.","type":"text"}]},{"type":"paragraph","content":[{"text":"On this machine, no IEC 61-850 paquets will be received.","type":"text"},{"type":"hardBreak"},{"text":"It can be synchronized in PTP but it is not needed; it can stick to the simple NTP synchronization.","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Even if PTP is not required, time synchronization must be ensured, at least in NTP. Otherwise, the machines will not be able to form the SEAPATH cluster.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"So, on an observer, only three interfaces are needed : two for the cluster and one for administration.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Administration machine","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"In order to configure all the machines in the cluster, we advise using a switch connected to the administration machine.","type":"text"}]},{"type":"paragraph","content":[{"text":"Below is an example with two hypervisors and one observer, all connected together in a triangle cluster. The administration machine is connected to them using a switch.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":96.0},"content":[{"type":"media","attrs":{"width":692,"id":"ddff4eda-caff-439f-8667-7b59e551865d","collection":"contentId-31821518","type":"file","height":421}}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Receiving Sample values and PTP frames","type":"text","marks":[{"type":"strong"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Machines","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"The standard way to generate sample values is to use a merging unit. For PTP, it is a Grand master clock.","type":"text"}]},{"type":"paragraph","content":[{"text":"However, in a test environment, these two machines can be simulated:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"See this page to ","type":"text"},{"text":"simulate IEC61850 traffic","type":"text","marks":[{"type":"link","attrs":{"href":"https://lf-energy.atlassian.net/wiki/spaces/SEAP/pages/31821125"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"See this page to ","type":"text"},{"text":"simulate a grand master clock","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"https://lf-energy.atlassian.net/wiki/display/SEAP/Time+synchronization"}}]}]}]}]},{"type":"paragraph","content":[{"text":"For this section example, we will use only one machine to simulate both PTP frames and IEC 61-850 traffic. This machine will be named “Publisher”.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Connections","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"In order to connect the publisher (or the merging unit/Grand master clock), a direct connection, with only one cable is the best. It avoids unnecessary latency, but necessitates having a machine with many interfaces.","type":"text"}]},{"type":"paragraph","content":[{"text":"A switch can then be used between the publisher and the cluster machine. In this situation, we advise you to use a separate switch from the one used for administration. It is possible to use only one switch for both, but in that case, the two networks must be separated using VLAN.","type":"text"}]},{"type":"paragraph","content":[{"text":"Note: To manage PTP, the best is to use a PTP-compliant switch. See more information on the ","type":"text"},{"text":"Time synchronization","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"https://lf-energy.atlassian.net/wiki/display/SEAP/Time+synchronization"}}]},{"text":" page","type":"text"}]},{"type":"paragraph","content":[{"text":"Below is a schema of a cluster with two hypervisors and an observer. The administration network is isolated on his switch. Another switch is used for PTP and IEC 61-850 traffic. Remember that if you use PCI-passthrough, you will need as many interfaces as you have Virtual machines.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":85.0},"content":[{"type":"media","attrs":{"width":931,"id":"e9fc8422-c47d-4b31-9d64-eec5876f3fc5","collection":"contentId-31821518","type":"file","height":641}}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"To avoid PTP frames interrupting with IEC 61-850 traffic, the best is to isolate PTP frames on a VLAN. This is done with the variable `ptp_vlanid` in the ansible inventories.","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reconfiguring the network entirely with netplan","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"It is possible to redefine the SEAPATH network entirely using ","type":"text"},{"text":"netplan.io","type":"text","marks":[{"type":"link","attrs":{"href":"https://netplan.io/"}}]},{"text":" yaml files. This behavior is achieved by the variable ","type":"text"},{"text":"netplan_configurations","type":"text","marks":[{"type":"code"}]},{"text":" in the inventory. Examples are provided in the Ansible repository.","type":"text"}]},{"type":"paragraph","content":[{"text":"Below are some useful examples for using netplan:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SR-IOV configuration","type":"text","marks":[{"type":"link","attrs":{"href":"https://netplan.readthedocs.io/en/stable/examples/#how-to-configure-sr-iov-virtual-functions"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Configuring a Linux bridge with vlans","type":"text","marks":[{"type":"link","attrs":{"href":"https://computingforgeeks.com/configuring-linux-bridge-vlan-using-netplan-ubuntu/"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Official documentation","type":"text","marks":[{"type":"link","attrs":{"href":"https://netplan.readthedocs.io/en/stable/"}}]}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"For now, using netplan disables all network configuration on SEAPATH. Everything, including the administration and cluster network, has to be handled by netplan yaml files.","type":"text"}]},{"type":"paragraph","content":[{"text":"The goal would be to be able to define the network entirely, either with systemd-networkd directly or with netplan, but this is not done now.","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"OpenVSwitch management on a standalone Seapath configuration","type":"text","marks":[{"type":"strong"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Creating an OVS bridge with Netplan","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Seapath supports the creation of an OVS bridge directly from a Netplan configuration file. To do so, you can use the Netplan configuration example file located in Seapath Ansible inventories directory.","type":"text"}]},{"type":"paragraph","content":[{"text":"In the following example, we create an OVS bridge connected to the physical interface pointed by ","type":"text"},{"text":"ovs_ext_interface","type":"text","marks":[{"type":"code"}]},{"text":" inventory variable, defined in the hypervisor inventory. All the network stream coming from this interface will be redirected to this bridge. This configuration can be useful in the case you want to share the network stream coming on one interface to multiple virtual port, and so virtual machines.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"plaintext"},"content":[{"text":"# Copyright (C) 2024 Savoir-faire Linux, Inc.\n# SPDX-License-Identifier: Apache-2.0\nnetwork:\nversion: 2\nrenderer: networkd\nopenvswitch:\nprotocols: [OpenFlow13, OpenFlow14, OpenFlow15]\nethernets:\n{{ ovs_ext_interface }}: {}\nbridges:\novs0:\ninterfaces: [{{ ovs_ext_interface }}]\nopenvswitch:\nprotocols: [OpenFlow10, OpenFlow11, OpenFlow12]","type":"text"}]},{"type":"paragraph","content":[{"text":"After applying the Netplan configuration using ","type":"text"},{"text":"netplan apply","type":"text","marks":[{"type":"code"}]},{"text":" command, you can identify the created OVS bridge (here ovs0) with command ","type":"text"},{"text":"ip a","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"root@minisforum:~# ip a\n1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000\nlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\ninet 127.0.0.1/8 scope host lo\nvalid_lft forever preferred_lft forever\n2: enp2s0: mtu 1500 qdisc mq state UP group default qlen 1000\nlink/ether 58:47:ca:72:49:50 brd ff:ff:ff:ff:ff:ff\n3: enp3s0: mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000\nlink/ether 58:47:ca:72:49:51 brd ff:ff:ff:ff:ff:ff\n4: enx782d7e14367c: mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000\nlink/ether 78:2d:7e:14:36:7c brd ff:ff:ff:ff:ff:ff\n7: br0: mtu 1500 qdisc noqueue state UP group default qlen 1000\nlink/ether fa:f7:2f:2c:1b:b8 brd ff:ff:ff:ff:ff:ff\ninet 192.168.216.74/24 brd 192.168.216.255 scope global br0\nvalid_lft forever preferred_lft forever\n8: wlp4s0: mtu 1500 qdisc noop state DOWN group default qlen 1000\nlink/ether 38:d5:7a:25:d4:2d brd ff:ff:ff:ff:ff:ff\n9: docker0: mtu 1500 qdisc noqueue state DOWN group default \nlink/ether 02:42:a2:71:7b:7e brd ff:ff:ff:ff:ff:ff\ninet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\nvalid_lft forever preferred_lft forever\n10: ovs-system: mtu 1500 qdisc noop state DOWN group default qlen 1000\nlink/ether 1e:86:fb:0e:ff:7c brd ff:ff:ff:ff:ff:ff\n11: ovs0: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000\nlink/ether 58:47:ca:72:49:51 brd ff:ff:ff:ff:ff:ff","type":"text"}]},{"type":"paragraph","content":[{"text":"You can also confirm the creation of the OVS bridge using ","type":"text"},{"text":"ovs-vsctl","type":"text","marks":[{"type":"code"}]},{"text":" show:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"root@minisforum:~# ovs-vsctl show\ndb96c66c-2cfb-40d0-bbcb-10038eb1e6fb\nBridge ovs0\nfail_mode: standalone\nPort enp3s0\nInterface enp3s0\nPort ovs0\nInterface ovs0\ntype: internal\novs_version: \"3.1.0\"","type":"text"}]},{"type":"paragraph"},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"enp3s0","type":"text","marks":[{"type":"code"}]},{"text":" is the interface pointed by ","type":"text"},{"text":"ovs_ext_interface","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Creating an OVS virtual port with Libvirt","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Seapath supports also the creation of an OVS virtual port directly from the VM Libvirt XML configuration file. To do so, you can reuse the templated VM file ","type":"text"},{"text":"guest.xml.j2","type":"text","marks":[{"type":"code"}]},{"text":" which supports this feature. ","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"{% if vm.bridges is defined %}\n{% for bridge in vm.bridges %}\n\n\n\n\n{% if bridge.type is defined %}\n\n{% endif %}\n{% if bridge.vlan is defined %}\n\n\n\n{% endif %}\n\n{% endfor %}\n{% endif %}","type":"text"}]},{"type":"paragraph","content":[{"text":"Then, in your VM inventory, you have to add the OVS bridge in a ","type":"text"},{"text":"bridge","type":"text","marks":[{"type":"code"}]},{"text":" field and specifies the name, it MAC address, and finally the type of bridge to ","type":"text"},{"text":"openvswitch","type":"text","marks":[{"type":"code"}]},{"text":". This type will tell to Libvirt that the selected bridge in an OpenVSwitch bridge, and so will add a virtual port connected to it. Based on the previous example, all network stream coming from the interface pointed by ","type":"text"},{"text":"ovs_ext_interface ","type":"text","marks":[{"type":"code"}]},{"text":"will be redirected to this new virtual port.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"plaintext"},"content":[{"text":"bridges:\n- name: \"ovs0\" \nmac_address: \"58:47:ca:72:49:51\"\ntype: openvswitch","type":"text"}]},{"type":"paragraph","content":[{"text":"After VM creation and starting this new virtual port will appeared on hypervisor side (here ","type":"text"},{"text":"vnet0","type":"text","marks":[{"type":"code"}]},{"text":") with command ","type":"text"},{"text":"ip a","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"root@minisforum:~# ip a\n1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000\nlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\ninet 127.0.0.1/8 scope host lo\nvalid_lft forever preferred_lft forever\n2: enp2s0: mtu 1500 qdisc mq state UP group default qlen 1000\nlink/ether 58:47:ca:72:49:50 brd ff:ff:ff:ff:ff:ff\n3: enp3s0: mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000\nlink/ether 58:47:ca:72:49:51 brd ff:ff:ff:ff:ff:ff\n4: enx782d7e14367c: mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000\nlink/ether 78:2d:7e:14:36:7c brd ff:ff:ff:ff:ff:ff\n7: br0: mtu 1500 qdisc noqueue state UP group default qlen 1000\nlink/ether fa:f7:2f:2c:1b:b8 brd ff:ff:ff:ff:ff:ff\ninet 192.168.216.74/24 brd 192.168.216.255 scope global br0\nvalid_lft forever preferred_lft forever\n8: wlp4s0: mtu 1500 qdisc noop state DOWN group default qlen 1000\nlink/ether 38:d5:7a:25:d4:2d brd ff:ff:ff:ff:ff:ff\n9: docker0: mtu 1500 qdisc noqueue state DOWN group default \nlink/ether 02:42:a2:71:7b:7e brd ff:ff:ff:ff:ff:ff\ninet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\nvalid_lft forever preferred_lft forever\n10: ovs-system: mtu 1500 qdisc noop state DOWN group default qlen 1000\nlink/ether 1e:86:fb:0e:ff:7c brd ff:ff:ff:ff:ff:ff\n11: ovs0: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000\nlink/ether 58:47:ca:72:49:51 brd ff:ff:ff:ff:ff:ff\n12: vnet0: mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000\nlink/ether fe:47:ca:72:49:51 brd ff:ff:ff:ff:ff:ff","type":"text"}]},{"type":"paragraph","content":[{"text":"You may need to isolate the network stream for a specific virtual port, and so create a specific VLAN for it. To do so, you can add in the VM inventory ","type":"text"},{"text":"bridge","type":"text","marks":[{"type":"code"}]},{"text":" field a VLAN field such as follow:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"plaintext"},"content":[{"text":"bridges:\n- name: \"ovs0\" # Change the bridge name\nmac_address: \"58:47:ca:72:49:51\" \ntype: openvswitch\nvlan: 100","type":"text"}]},{"type":"paragraph","content":[{"text":"In this example, the vnet0 OVS virtual port will be isolated on VLAN 100.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Seapath Cluster network","type":"text"}]},{"type":"paragraph","content":[{"text":"Seapath default recommended architecture is to use 3 hosts, directly interconnected to each other in a loop topology (\"cluster network\"). This network will be used for:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ceph clustering communication (public and private network)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"corosync clustering communication (used by pacemaker)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"vxlan trafic to extend the VM bridges topology","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"It would also be possible to use external switches for this cluster network, but the high availability requirements would demand to use 2 switches (in case one fails), interconnected together (2 times in case one link fails), to connect every host to both switches and to use a mecanism like LACP to deal with multiple paths.","type":"text"}]},{"type":"paragraph","content":[{"text":"We think that for a 3-node cluster, a no-switch architecture is simple to create, manage, debug, etc.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":471,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":401,"id":"692b27c7-44d4-47ab-80bb-92d49f7a5516","collection":"contentId-31821518","type":"file","height":211}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Bridge architecture","type":"text"}]},{"type":"paragraph","content":[{"text":"We propose a layer 2 networking solution (using OpenVSwitch since it's included in the seapath distribution), where each host runs a software bridge (we call it \"team0\"), with 2 ports connected to external network interfaces. Each of the 2 network interfaces is connected to one of the other host. The ip address for the host on this cluster network is directly set to the team0 bridge:","type":"text"}]},{"type":"paragraph"},{"type":"mediaSingle","attrs":{"layout":"align-start","width":591,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":541,"id":"f61f4ec0-f179-4842-a5aa-b3075220ad42","collection":"contentId-31821518","type":"file","height":271}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Loop prevention, RSTP","type":"text"}]},{"type":"paragraph","content":[{"text":"This will create an extended L2 network accross the 3 nodes, each node having a L3 ip address for all the mentionned communications, however this will create an obvious loop. L2 loops needs to be dealt with using some form of spanning tree protocol. We propose to use RSTP. In this example we set a smaller rstp priority on the host1, which will make rstp cut the loop between the","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":554,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":541,"id":"c6d89066-aa66-49be-b435-622bbeb421e5","collection":"contentId-31821518","type":"file","height":271}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Cabling","type":"text"}]},{"type":"paragraph","content":[{"text":"To simplify the cabling engineering, we recommend:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"thinking of the 3 hosts as \"1, 2, 3\", following each other in the loop \"1 --> 2 --> 3 --> 1\"","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"if N=1 then N+1=2","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"if N=2 then N+1=3","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"if N=3 then N+1=1","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"calling the 2 team0 bridge physical interfaces \"team0_0\" and \"team0_1\"","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"on host N, team0_0 should be connected to team0_1 of host N+1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"consequently on host N, team0_1 is connected to team0_0 of host N-1","type":"text"}]}]}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":562,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":612,"id":"b21c3af4-ab12-4331-91df-07e8bfe8b030","collection":"contentId-31821518","type":"file","height":342}}]}],"version":1}

Browser not supported