OVN L3 Gateway

by Vikrant

September 20, 2017

In one of my previous post, I talk about the L3 gateway. In this article, I am providing more information on this topic.

If you have worked with neutron DVR, you may already know that N/S traffic in case of instances without floating IP travel through controller node. Similarly, in case of OVN, if the instance is not having floating IP attached to it, then the SNAT happens on the node on which lrp port for the external network is present.

Example : In my case private network is 10.10.10.0/24 and floating IP network is 192.168.122.0/24, if I am spinning up an instance using private network i.e 10.10.10.0/24 and not attaching floating ip to it then all the instance N/S traffic will travel through the node on which your lrp port is present. It would be right to say that node running ovn-controller is candidate for running the L3 gateway. As I mentioned earlier, in packstack setup ovn-controller service is running on controller node also hence in my case controller node can also host the L3 gateway. After the packstack succesful installation, OVN l3 gateway was present on compute2 node, but to test some failover scenario, I shutdown the compute2 and L3 gateway got moved to controller node.

lrp port is currently present on controller node hence all the traffic for instances without floating IP will go through controller node.

[root@controller ~(keystone_admin)]# ovn-sbctl show
Chassis "07e20ac2-4f66-41d4-b8c6-a5b6ae80921b"
    hostname: controller
    Encap geneve
        ip: "192.168.122.39"
        options: {csum="true"}
    Port_Binding "cr-lrp-b95e9ae7-5c91-4037-8d2c-660d4af00974"
Chassis "4180095d-1528-4063-b135-5dc0abc4ecee"
    hostname: "compute2"
    Encap geneve
        ip: "192.168.122.207"
        options: {csum="true"}
Chassis "080677db-48d9-49ad-9ee4-e02eeb3da5c2"
    hostname: "compute1"
    Encap geneve
        ip: "192.168.122.15"
        options: {csum="true"}
    Port_Binding "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1"

Following setting decide the node on which L3 gateway will be hosted.

[root@controller ~(keystone_admin)]# grep 'ovn_l3_scheduler' /etc/neutron/plugins/networking-ovn/networking-ovn.ini
#ovn_l3_scheduler = leastloaded

Only two supported values are present for this setting as per the comment in ini file.

# The OVN L3 Scheduler type used to schedule router gateway ports on
# hypervisors/chassis.
# leastloaded - chassis with fewest gateway ports selected
# chance - chassis randomly selected (string value)
# Allowed values: leastloaded, chance

If you are having more than one VLAN based external network then it’s not necessary that all L3 gateways will be present on node, they can be distributed across multiple nodes to avoid the single point of bottleneck or failure. Even if the compute node hosting the L3 gateway is getting down, it will automatically move the gateway to another compute node according to the filter results.

In newer version ovs-ovn i.e 2.8 at the time of writing more useful commands are present to extract the information about L3 gateway chassis. I am planning to create devstack setup to try out those commands.

OVN Cheat Sheet

by Vikrant

September 20, 2017

OVN Cheat Sheet for ovn-nbctl and ovn-sbctl commands.

ovn-nbctl

1) Issue the following command to understand the logical network topology. As no port from the switch is used to spin up an instance hence only router port is seen in output.

[root@controller ~(keystone_admin)]# ovn-nbctl show
    switch 0d413d9c-7f23-4ace-9a8a-29817b3b33b5 (neutron-89113f8b-bc01-46b1-84fb-edd5d606879c)
        port 397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
            addresses: ["router"]
    switch 1ec08997-0899-40d1-9b74-0a25ef476c00 (neutron-e411bbe8-e169-4268-b2bf-d5959d9d7260)
        port provnet-e411bbe8-e169-4268-b2bf-d5959d9d7260
            addresses: ["unknown"]
        port b95e9ae7-5c91-4037-8d2c-660d4af00974
            addresses: ["router"]
    router 7418a4e7-abff-4af7-85f5-6eea2ede9bea (neutron-67dc2e78-e109-4dac-acce-b71b2c944dc1)
        port lrp-b95e9ae7-5c91-4037-8d2c-660d4af00974
            mac: "fa:16:3e:52:20:7c"
            networks: ["192.168.122.50/24"]
        port lrp-397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
            mac: "fa:16:3e:87:28:40"
            networks: ["10.10.10.1/24"]

2) Let’s spin up an instance and then check the output of command. We can see that from switch “0d413d9c-7f23-4ace-9a8a-29817b3b33b5” one port is used to spin up an instance, we can see the private IP and MAC address of the instance. I have also attached the floating IP to instance but we can’t see the floating IP in output.

[root@controller ~(keystone_admin)]# nova list
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------+
| ID                                   | Name          | Status | Task State | Power State | Networks                             |
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------+
| edef7dd9-0114-4463-a13c-c9a2edaf6ce4 | testinstance1 | ACTIVE | -          | Running     | internal1=10.10.10.9, 192.168.122.54 |
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------+

[root@controller ~(keystone_admin)]# ovn-nbctl show
    switch 0d413d9c-7f23-4ace-9a8a-29817b3b33b5 (neutron-89113f8b-bc01-46b1-84fb-edd5d606879c)
        port 6fe3cab5-5f84-44c8-90f2-64c21b489c62
            addresses: ["fa:16:3e:fa:d6:d3 10.10.10.9"]
        port 397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
            addresses: ["router"]
    switch 1ec08997-0899-40d1-9b74-0a25ef476c00 (neutron-e411bbe8-e169-4268-b2bf-d5959d9d7260)
        port provnet-e411bbe8-e169-4268-b2bf-d5959d9d7260
            addresses: ["unknown"]
        port b95e9ae7-5c91-4037-8d2c-660d4af00974
            addresses: ["router"]
    router 7418a4e7-abff-4af7-85f5-6eea2ede9bea (neutron-67dc2e78-e109-4dac-acce-b71b2c944dc1)
        port lrp-b95e9ae7-5c91-4037-8d2c-660d4af00974
            mac: "fa:16:3e:52:20:7c"
            networks: ["192.168.122.50/24"]
        port lrp-397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e
            mac: "fa:16:3e:87:28:40"
            networks: ["10.10.10.1/24"]

3) If you just want to list the logical switches present in setup.

[root@controller ~(keystone_admin)]# ovn-nbctl ls-list
0d413d9c-7f23-4ace-9a8a-29817b3b33b5 (neutron-89113f8b-bc01-46b1-84fb-edd5d606879c)
1ec08997-0899-40d1-9b74-0a25ef476c00 (neutron-e411bbe8-e169-4268-b2bf-d5959d9d7260)

4) Similarly to list only routers.

[root@controller ~(keystone_admin)]# ovn-nbctl lr-list
7418a4e7-abff-4af7-85f5-6eea2ede9bea (neutron-67dc2e78-e109-4dac-acce-b71b2c944dc1)

5) To list the ports associated with switch and router.

Logical switch ports

[root@controller ~(keystone_admin)]# ovn-nbctl lsp-list 0d413d9c-7f23-4ace-9a8a-29817b3b33b5
6ef6bbdb-fa68-4fff-a281-90387da961f2 (397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e)

Logical router ports

[root@controller ~(keystone_admin)]# ovn-nbctl lrp-list 7418a4e7-abff-4af7-85f5-6eea2ede9bea
f5bc5204-4c19-4cf8-b240-67fa268120b5 (lrp-397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e)
947dc0fb-cfe6-4737-aa27-9fb38b7c7ac9 (lrp-b95e9ae7-5c91-4037-8d2c-660d4af00974)

6) Two main factors which decide the communication with instance is ACL (Security groups) and NAT.

Security groups are applied at logical switch. I have allowed the ICMP and ssh traffic.

[root@controller ~(keystone_admin)]# ovn-nbctl acl-list 0d413d9c-7f23-4ace-9a8a-29817b3b33b5
from-lport  1002 (inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip4) allow-related
from-lport  1002 (inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip4 && ip4.dst == {255.255.255.255, 10.10.10.0/24} && udp && udp.src == 68 && udp.dst == 67) allow
from-lport  1002 (inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip6) allow-related
from-lport  1001 (inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip) drop
  to-lport  1002 (outport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip4 && ip4.src == $as_ip4_329529f7_d012_4ba1_a689_dd443b0ec795) allow-related
  to-lport  1002 (outport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
  to-lport  1002 (outport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip4 && ip4.src == 0.0.0.0/0 && tcp && tcp.dst == 22) allow-related
  to-lport  1002 (outport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip6 && ip6.src == $as_ip6_329529f7_d012_4ba1_a689_dd443b0ec795) allow-related
  to-lport  1001 (outport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && ip) drop

At the moment, I have deleted the previous instance hence only one default SNAT rule is present which is applicable for whole network if the instance is not having floating IP.

[root@controller ~(keystone_admin)]# ovn-nbctl lr-nat-list 7418a4e7-abff-4af7-85f5-6eea2ede9bea
TYPE             EXTERNAL_IP        LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
snat             192.168.122.50     10.10.10.0/24

Let’s spin the instance and attach floating ip to the instance and check the NAT rule again. New SNAT/DNAT rule is inserted for the instance.

[root@controller ~(keystone_admin)]# nova list
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------+
| ID                                   | Name          | Status | Task State | Power State | Networks                             |
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------+
| 69736780-e0cc-46d4-a1f7-f0fac7e1cf54 | testinstance1 | ACTIVE | -          | Running     | internal1=10.10.10.4, 192.168.122.54 |
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------+

[root@controller ~(keystone_admin)]# ovn-nbctl lr-nat-list 7418a4e7-abff-4af7-85f5-6eea2ede9bea
TYPE             EXTERNAL_IP        LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
dnat_and_snat    192.168.122.54     10.10.10.4
snat             192.168.122.50     10.10.10.0/24

ovn-sbctl

Checking all the nodes on which ovn-controller is running.

[root@controller ~(keystone_admin)]# ovn-sbctl show
Chassis "07e20ac2-4f66-41d4-b8c6-a5b6ae80921b"
    hostname: controller
    Encap geneve
        ip: "192.168.122.39"
        options: {csum="true"}
    Port_Binding "cr-lrp-b95e9ae7-5c91-4037-8d2c-660d4af00974"
Chassis "4180095d-1528-4063-b135-5dc0abc4ecee"
    hostname: "compute2"
    Encap geneve
        ip: "192.168.122.207"
        options: {csum="true"}
Chassis "080677db-48d9-49ad-9ee4-e02eeb3da5c2"
    hostname: "compute1"
    Encap geneve
        ip: "192.168.122.15"
        options: {csum="true"}
    Port_Binding "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1"

Command to check all the logical flows present for all DATAPATHs. In newer version “–ovs” option is available which will also show the openflow corresponding to logicalflow and eventually openflows are getting injected into compute nodes.

[root@controller ~(keystone_admin)]# ovn-sbctl lflow-list | head
Datapath: "neutron-89113f8b-bc01-46b1-84fb-edd5d606879c" (75c461bc-bf27-4402-ac7d-98630213a25e)  Pipeline: ingress
  table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
  table=0 (ls_in_port_sec_l2  ), priority=100  , match=(vlan.present), action=(drop;)
  table=0 (ls_in_port_sec_l2  ), priority=50   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == {fa:16:3e:55:52:80}), action=(next;)
  table=0 (ls_in_port_sec_l2  ), priority=50   , match=(inport == "397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e"), action=(next;)
  table=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67), action=(next;)
  table=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && ip4.src == {10.10.10.4}), action=(next;)
  table=1 (ls_in_port_sec_ip  ), priority=80   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && ip), action=(drop;)
  table=1 (ls_in_port_sec_ip  ), priority=0    , match=(1), action=(next;)
  table=2 (ls_in_port_sec_nd  ), priority=90   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && arp.sha == fa:16:3e:55:52:80 && arp.spa == {10.10.10.4}), action=(next;)

Checking all the DATAPATH present in ovn-sbctl output. For each logical switch and router datapath, two Pipelines are present for ingress and egress.

[root@controller ~(keystone_admin)]# ovn-sbctl lflow-list | grep Datapath
Datapath: "neutron-89113f8b-bc01-46b1-84fb-edd5d606879c" (75c461bc-bf27-4402-ac7d-98630213a25e)  Pipeline: ingress
Datapath: "neutron-89113f8b-bc01-46b1-84fb-edd5d606879c" (75c461bc-bf27-4402-ac7d-98630213a25e)  Pipeline: egress
Datapath: "neutron-67dc2e78-e109-4dac-acce-b71b2c944dc1" (98a36c00-e896-44b6-aafc-278a0bf21607)  Pipeline: ingress
Datapath: "neutron-67dc2e78-e109-4dac-acce-b71b2c944dc1" (98a36c00-e896-44b6-aafc-278a0bf21607)  Pipeline: egress
Datapath: "neutron-e411bbe8-e169-4268-b2bf-d5959d9d7260" (f527be20-ed2b-4cf6-a94e-1be7ef98bbb5)  Pipeline: ingress
Datapath: "neutron-e411bbe8-e169-4268-b2bf-d5959d9d7260" (f527be20-ed2b-4cf6-a94e-1be7ef98bbb5)  Pipeline: egress

To check the logical flows only for one Datapath for both ingress and egress pipeline.

[root@controller ~(keystone_admin)]# ovn-sbctl lflow-list neutron-89113f8b-bc01-46b1-84fb-edd5d606879c | head
Datapath: "neutron-89113f8b-bc01-46b1-84fb-edd5d606879c" (75c461bc-bf27-4402-ac7d-98630213a25e)  Pipeline: ingress
  table=0 (ls_in_port_sec_l2  ), priority=100  , match=(eth.src[40]), action=(drop;)
  table=0 (ls_in_port_sec_l2  ), priority=100  , match=(vlan.present), action=(drop;)
  table=0 (ls_in_port_sec_l2  ), priority=50   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == {fa:16:3e:55:52:80}), action=(next;)
  table=0 (ls_in_port_sec_l2  ), priority=50   , match=(inport == "397c019e-9bc3-49d3-ac4c-4aeeb1b3ba3e"), action=(next;)
  table=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67), action=(next;)
  table=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && ip4.src == {10.10.10.4}), action=(next;)
  table=1 (ls_in_port_sec_ip  ), priority=80   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && ip), action=(drop;)
  table=1 (ls_in_port_sec_ip  ), priority=0    , match=(1), action=(next;)
  table=2 (ls_in_port_sec_nd  ), priority=90   , match=(inport == "0bc5e22d-bd80-4cac-a9b3-51c0d0b284d1" && eth.src == fa:16:3e:55:52:80 && arp.sha == fa:16:3e:55:52:80 && arp.spa == {10.10.10.4}), action=(next;)

How to import existing DOCKERFILE into ansible-container

by Vikrant

September 15, 2017

In this post I am showing the procedure to import existing DOCKERFILE into the ansible-container.

Step 1 : Created directory /root/ANSIBLE-CONTAINER-PROJECTS/project1/DOCKERFILE and added the following files into that directory.

[root@devops DOCKERFILE]# cat Dockerfile
FROM centos:7
MAINTAINER The CentOS Project <cloud-ops@centos.org>
LABEL Vendor="CentOS" \
      License=GPLv2 \
      Version=2.4.6-40


RUN yum -y --setopt=tsflags=nodocs update && \
    yum -y --setopt=tsflags=nodocs install httpd && \
    yum clean all

EXPOSE 80

# Simple startup script to avoid some issues observed with container restart
ADD run-httpd.sh /run-httpd.sh
RUN chmod -v +x /run-httpd.sh

CMD ["/run-httpd.sh"]


[root@devops DOCKERFILE]# cat run-httpd.sh
#!/bin/bash

# Make sure we're not confused by old, incompletely-shutdown httpd
# context after restarting the container.  httpd won't start correctly
# if it thinks it is already running.
rm -rf /run/httpd/* /tmp/httpd*

exec /usr/sbin/apachectl -DFOREGROUND

I downloaded this from [1].

Step 2 : Start importing the project into the ansible-container. We just need to give the name of directory in which Dockerfile and script is present.

[root@devops ANSIBLE-CONTAINER-PROJECTS]# cd project1/
[root@devops project1]# ls
DOCKERFILE
[root@devops project1]# ansible-container import DOCKERFILE
Project successfully imported. You can find the results in:
/root/ANSIBLE-CONTAINER-PROJECTS/project1
A brief description of what you will find...


container.yml
-------------

The container.yml file is your orchestration file that expresses what services you have and how to build/run them.

settings:
  conductor_base: centos:7
services:
  DOCKERFILE:
    roles:
    - DOCKERFILE


I added a single service named DOCKERFILE for your imported Dockerfile.
As you can see, I made an Ansible role for your service, which you can find in:
`/root/ANSIBLE-CONTAINER-PROJECTS/project1/roles/DOCKERFILE`

project1/roles/DOCKERFILE/tasks/main.yml
----------------------------------------

The tasks/main.yml file has your RUN/ADD/COPY instructions.

- shell: yum -y --setopt=tsflags=nodocs update
- shell: yum -y --setopt=tsflags=nodocs install httpd
- shell: yum clean all
- name: Ensure / exists
  file:
    path: /
    state: directory
- name: Simple startup script to avoid some issues observed with container restart
    (run-httpd.sh)
  copy:
    src: run-httpd.sh
    dest: /run-httpd.sh
- shell: chmod -v +x /run-httpd.sh


I tried to preserve comments as task names, but you probably want to make
sure each task has a human readable name.

project1/roles/DOCKERFILE/meta/container.yml
--------------------------------------------

Metadata from your Dockerfile went into meta/container.yml in your role.
These will be used as build/run defaults for your role.

from: centos:7
maintainer: The CentOS Project <cloud-ops@centos.org>
labels:
  Vendor: CentOS
  License: GPLv2
  Version: 2.4.6-40
ports:
- '80'
command:
- /run-httpd.sh


I also stored ARG directives in the role's defaults/main.yml which will used as
variables by Ansible in your build and run operations.

Good luck!
Project imported.

Step 3 : Our project is imported, after importing the project here is the directory structure.

[root@devops project1]# ls
container.yml  DOCKERFILE  roles  run-httpd.sh

It contains container.yaml file, roles directory and run-httpd.sh script. 

container.yaml is using centos7 as base image and calling role DOCKERFILE. 

[root@devops project1]# cat container.yml
settings:
  conductor_base: centos:7
services:
  DOCKERFILE:
    roles:
    - DOCKERFILE

roles directory contains DOCKERFILE role. It shows all the tasks which it will run when the build process is triggered.

[root@devops project1]# cd roles/
[root@devops roles]# ls
DOCKERFILE
[root@devops roles]# cd DOCKERFILE/
[root@devops DOCKERFILE]# ls
defaults  meta  README.md  tasks  test
[root@devops DOCKERFILE]# cat tasks/main.yml
- shell: yum -y --setopt=tsflags=nodocs update
- shell: yum -y --setopt=tsflags=nodocs install httpd
- shell: yum clean all
- name: Ensure / exists
  file:
    path: /
    state: directory
- name: Simple startup script to avoid some issues observed with container restart
    (run-httpd.sh)
  copy:
    src: run-httpd.sh
    dest: /run-httpd.sh
- shell: chmod -v +x /run-httpd.sh
[root@devops DOCKERFILE]# cat meta/
container.yml  main.yml
[root@devops DOCKERFILE]# cat meta/
container.yml  main.yml
[root@devops DOCKERFILE]# cat meta/container.yml
from: centos:7
maintainer: The CentOS Project <cloud-ops@centos.org>
labels:
  Vendor: CentOS
  License: GPLv2
  Version: 2.4.6-40
ports:
- '80'
command:
- /run-httpd.sh

Step 4 : Initiating the ansible-container build will show all the tasks it’s running which is present in role.

[root@devops project1]# ansible-container build
Building Docker Engine context...
Starting Docker build of Ansible Container Conductor image (please be patient)...
Parsing conductor CLI args.
Docker™ daemon integration engine loaded. Build starting.       project=project1
Building service...     project=project1 service=DOCKERFILE
PLAY [DOCKERFILE] **************************************************************
TASK [Gathering Facts] *********************************************************
ok: [DOCKERFILE]
TASK [DOCKERFILE : command] ****************************************************
 [WARNING]: Consider using yum module rather than running yum
changed: [DOCKERFILE]
TASK [DOCKERFILE : command] ****************************************************
changed: [DOCKERFILE]
TASK [DOCKERFILE : command] ****************************************************
changed: [DOCKERFILE]
TASK [DOCKERFILE : Ensure / exists] ********************************************
ok: [DOCKERFILE]
TASK [DOCKERFILE : Simple startup script to avoid some issues observed with container restart (run-httpd.sh)] ***
changed: [DOCKERFILE]
TASK [DOCKERFILE : command] ****************************************************
 [WARNING]: Consider using file module with mode rather than running chmod
changed: [DOCKERFILE]
PLAY RECAP *********************************************************************
DOCKERFILE                 : ok=7    changed=5    unreachable=0    failed=0
Applied role to service role=DOCKERFILE service=DOCKERFILE
Committed layer as image        image=sha256:9200752cbe499023de2c755e7ee790a51e36cca3b8b11c48a3609b2e7436d429 service=DOCKERFILE
Build complete. service=DOCKERFILE
All images successfully built.
Conductor terminated. Cleaning up.      command_rc=0 conductor_id=841adaf51d887a1f2abc56a9f01996573c7074c8b217ae3113ae4fbc693a1c0e save_container=False

Build will create a new directory inside the directory.

[root@devops project1]# ls
ansible-deployment  container.yml  DOCKERFILE  roles  run-httpd.sh

Step 5 : Following are the images present after the build complete.

[root@devops project1]# docker images
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
project1-dockerfile   20170826195807      9200752cbe49        3 minutes ago       244.9 MB
project1-dockerfile   latest              9200752cbe49        3 minutes ago       244.9 MB
project1-conductor    latest              e499184e44d8        5 minutes ago       549.1 MB
docker.io/centos      7                   328edcd84f1b        3 weeks ago         192.5 MB

Checking the history of images created during build process. centos:7 is the base image which is used for build process.

All the tasks ran on conductor image.

[root@devops DOCKERFILE]# docker history project1-conductor:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
e499184e44d8        20 minutes ago      /bin/sh -c #(nop)  VOLUME [/usr]                0 B
012c53305347        20 minutes ago      /bin/sh -c ( test -f /_ansible/build/ansible-   0 B
8068be3c5409        20 minutes ago      /bin/sh -c #(nop) COPY dir:9695e7c3619885b00d   0 B
d6dca564d2ae        20 minutes ago      /bin/sh -c cd /_ansible &&     pip install -r   109.1 MB
382321fc6a20        23 minutes ago      /bin/sh -c #(nop) COPY dir:44abffe3306d203510   3.645 MB
d4031cbc2a95        23 minutes ago      /bin/sh -c python /get-pip.py &&     mkdir -p   93.07 MB
b9ad4b07a63b        23 minutes ago      /bin/sh -c #(nop) COPY file:7d575017b1192e86d   1.595 MB
fc8a7163d23f        24 minutes ago      /bin/sh -c #(nop) ADD tarsum.v1+sha256:ee0f28   26.51 MB
1b7d7763d224        24 minutes ago      /bin/sh -c yum update -y &&     yum install -   122.7 MB
392230f5245d        26 minutes ago      /bin/sh -c #(nop)  ENV ANSIBLE_CONTAINER=1      0 B
328edcd84f1b        3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0 B
<missing>           3 weeks ago         /bin/sh -c #(nop)  LABEL name=CentOS Base Ima   0 B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:63492ba809361c51e7   192.5 MB

[root@devops DOCKERFILE]# docker history project1-dockerfile:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
9200752cbe49        18 minutes ago      sh -c while true; do sleep 1; done              52.36 MB            Built with Ansible Container (https://github.com/ansible/ansible-container)
328edcd84f1b        3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0 B
<missing>           3 weeks ago         /bin/sh -c #(nop)  LABEL name=CentOS Base Ima   0 B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:63492ba809361c51e7   192.5 MB

[root@devops DOCKERFILE]# docker history project1-dockerfile:20170826195807
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
9200752cbe49        18 minutes ago      sh -c while true; do sleep 1; done              52.36 MB            Built with Ansible Container (https://github.com/ansible/ansible-container)
328edcd84f1b        3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0 B
<missing>           3 weeks ago         /bin/sh -c #(nop)  LABEL name=CentOS Base Ima   0 B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:63492ba809361c51e7   192.5 MB

Step 6 : Running ansible-container.

[root@devops project1]# ansible-container run
Parsing conductor CLI args.
Engine integration loaded. Preparing run.       engine=Docker™ daemon
Verifying service image service=DOCKERFILE
PLAY [localhost] ***************************************************************
TASK [docker_service] **********************************************************
changed: [localhost]
PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0
All services running.   playbook_rc=0
Conductor terminated. Cleaning up.      command_rc=0 conductor_id=947b2b03ead6fc6981b8ff5f2b8b0364c76d665dae041c1dedc61b0e6f159462 save_container=False

Step 7 : A new container we can see which is having port 80 mapped to the host port 32769

[root@devops project1]# docker ps -a --no-trunc
CONTAINER ID                                                       IMAGE                                COMMAND             CREATED             STATUS              PORTS                   NAMES
a897fc299fcdce5904dcbebf75bce565681c6155253c317613a96374edec2b32   project1-dockerfile:20170826195807   "/run-httpd.sh"     43 seconds ago      Up 42 seconds       0.0.0.0:32769->80/tcp   project1_DOCKERFILE_1

Using the host ip address and port number 32769, we can access the apache web page.

[1] https://github.com/CentOS/CentOS-Dockerfiles/blob/master/httpd/centos7/run-httpd.sh

How openstack DVR works

by Vikrant

May 7, 2017

In this article I am going to explain the DVR traffic flow. Distributed virtual routing (DVR) helps to reduce the number of hops through which packet need to traverse before reaching the final destination.

I have deployed DVR based setup using 1 controller and 2 compute nodes with the help of Red Hat openstack director. Templates used for deployment can be found at.

After deployment, created two internal and external networks. Connected each pair using neutron router.

neutron net-create internal1
neutron subnet-create --name internal1 internal1 10.10.20.0/24
neutron router-create router1
neutron router-interface-add router1 internal1
neutron router-gateway-set router1 external1
neutron net-create --provider:network_type vlan --provider:physical_network datacentre --provider:segmentation_id 10 --router:external True --name external1
neutron subnet-create --name external1 --gateway 10.11.48.254 --allocation-pool start=10.11.48.100,end=10.11.48.150 --disable-dhcp external1 10.11.48.0/24

In the default security group, rules are added to allow ssh and ICMP traffic.

neutron security-group-rule-create 08a7aa44-41c1-4701-8376-3a7b90f60736 --direction ingress --ethertype IPv4 --port-range-min 22 --port-range-max 22 --protocol tcp
neutron security-group-rule-create 08a7aa44-41c1-4701-8376-3a7b90f60736 --direction ingress --ethertype IPv4  --protocol icmp

Spawned two instances. Attached floating ip to one of the instance. Second instance contains only private IP address. i kept the scenario like this because I want to show two kind of traffic flows:

N/S for instance without floating ip.
N/S for instance with floating ip.

Common info for both scenarios.

As we want to do the packet capture at all the hops coming in the path hence we need to create OVS dummy ports to collect the capture from OVS bridges as explained in Red Hat solution.

Here is the sample for creating dummy port in br-tun ovs bridge. Same command by changing names can be used for br-ex and br-int.

ip link add name br-tun-snooper0 type dummy
ip link set dev br-tun-snooper0 up
ovs-vsctl add-port br-tun br-tun-snooper0
ovs-vsctl -- set Bridge br-tun mirrors=@m  \
-- --id=@br-tun-snooper0 get Port br-tun-snooper0 \
-- --id=@br-tun get Port br-tun \
-- --id=@m create Mirror name=mymirror1 \
select-dst-port=@br-tun \
select-src-port=@br-tun \
output-port=@br-tun-snooper0 \
select_all=1

First, I will be covering N/S for instance without floating ip.

In this case my instance is running on compute-0. Following are instance details:

Instance name: test02 Private ip : 10.10.10.8 Mac address : fa:16:3e:f1:f8:e1

Compute Node:

Let’s check the namespaces created on controller and compute node after spawning an instance without floating ip.

On controller node: SNAT is used to perform NAT operatioin for instances without floating ip.

[root@overcloud-controller-0 ~]# ip netns list
snat-d463a28e-e1f5-48f3-991a-6f90980490e1
qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1
qdhcp-d7554f84-c7f2-4a3b-9331-f9df22043d6a

On compute node: Default gateway of internal network is present in qrouter namespace.

[root@overcloud-controller-0 ~]# ip netns list
qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1

After creating dummy interfaces for OVS bridges br-int and br-tun. I used following commands to capture the tcpdump on compute node on which instance is running.

tcpdump -s0 -i tap006715b9-82 -w /tmp/$(hostname)_tap006715b9-82.pcap &
tcpdump  -s0 -i qvo006715b9-82 -w /tmp/$(hostname)_qvo006715b9-82.pcacp &
ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qr-b6129458-2e -w /tmp/$(hostname)_qr-b6129458-2e.pcap &
tcpdump -s0 -i br-int-snooper0 -w /tmp/$(hostname)_br-int-snooper.pcap &
tcpdump -s0 -i br-tun-snooper0 -w /tmp/$(hostname)_br-tun-snooper.pcap &
tcpdump -s0 -i vlan50 -w /tmp/$(hostname)_vlan50.pcap &

Similary on controller node, after creating dummy interfaces for br-int, br-tun and br-ex, used following commands to capture tcpdump.

ip netns exec snat-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i sg-ff19cae5-3a -w /tmp/$(hostname)_sg-ff19cae5-3a.pcap &
ip netns exec snat-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qg-ef46e09a-0b -w /tmp/$(hostname)_qg-ef46e09a-0b.pcap &
ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qr-b6129458-2e -w /tmp/$(hostname)_qr-b6129458-2e.pcap &
tcpdump -s0 -i br-int-snooper0 -w /tmp/$(hostname)_br-int-snooper.pcap &
tcpdump -s0 -i br-ex-snooper0 -w /tmp/$(hostname)_br-ex-snooper.pcap &
tcpdump -s0 -i br-tun-snooper0 -w /tmp/$(hostname)_br-tun-snooper.pcap &
tcpdump -s0 -i vlan50 -w /tmp/$(hostname)_vlan50.pcap &

Try to ping 8.8.8.8 from inside the instance.

As the instance doesn’t know the MAC address of 8.8.8.8 hence it’s taking default GW MAC address as destination MAC.

# tshark -tad -n -r overcloud-compute-0.localdomain_tap006715b9-82.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     22 fa:16:3e:c0:66:cd,fa:16:3e:f1:f8:e1	10.10.10.8	8.8.8.8
     22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	8.8.8.8		10.10.10.8

Traffic remains same on qvo interface of br-int bridge.

# tshark -tad -n -r overcloud-compute-0.localdomain_qvo006715b9-82.pcacp -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     22 fa:16:3e:c0:66:cd,fa:16:3e:f1:f8:e1	10.10.10.8	8.8.8.8
     22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	8.8.8.8		10.10.10.8

As packet is going from br-int to qr namespace and then coming back again into br-int namespace with destination MAC address hence we are seeing more information from br-int-snooper dummy interface packet capture.

# tshark -tad -n -r overcloud-compute-0.localdomain_br-int-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     22 fa:16:3e:42:5a:c5,fa:16:3e:c0:66:cd	10.10.10.8	8.8.8.8
     22 fa:16:3e:c0:66:cd,fa:16:3e:f1:f8:e1	10.10.10.8	8.8.8.8
     22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	8.8.8.8		10.10.10.8

This transformation of MAC address happens with the help of static MAC entry present in qrouter namespace present on compute node.

# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip neigh | grep '10.10.10.9'
10.10.10.9 dev qr-b6129458-2e lladdr fa:16:3e:42:5a:c5 PERMANENT

IP address 10.10.10.9 is present sg interface of SNAT namespace.

From br-int packet goes into br-tun through patch cable and notice the different of MAC address. Instance source MAC address “fa:16:3e:f1:f8:e1” is replaced with “fa:16:3e:42:5a:c5” and destination MAC address is changed from “fa:16:3e:c0:66:cd” to “fa:16:3f:ec:d7:ad”.

# tshark -tad -n -r overcloud-compute-0.localdomain_br-tun-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     22 fa:16:3e:42:5a:c5,fa:16:3f:ec:d7:ad	10.10.10.8	8.8.8.8
     22 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	8.8.8.8		10.10.10.8

On vlan interface, tunnel endpoint addresses are seen.

# tshark -tad -n -r overcloud-compute-0.localdomain_vlan50.pcap -p udp -T fields -e eth.addr -e ip.src -e ip.dst | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     20 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be	192.168.123.25	192.168.123.24
     20 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1	192.168.123.24	192.168.123.25

Let’s check the ip address of vlan50 interface from both controller and compute node.

[root@overcloud-compute-0 ~]# ip a show vlan50
10: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 0a:d1:5b:87:f8:e1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.24/24 brd 192.168.123.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::8d1:5bff:fe87:f8e1/64 scope link 
       valid_lft forever preferred_lft forever

[root@overcloud-controller-0 tmp]# ip a show vlan50
8: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether e6:36:e1:cc:b7:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.123.25/24 brd 192.168.123.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::e436:e1ff:fecc:b7be/64 scope link 
       valid_lft forever preferred_lft forever

Decoding the encapsulated vxlan traffic using vxlan decoder. Vxlan identifier it has taken is ‘93’.

# tshark -tad -n -r overcloud-compute-0.localdomain_vlan50.pcap -p -d udp.port==51929,vxlan -d udp.port==4789,vxlan -T fields -e eth.addr -e ip.src -e ip.dst -e vxlan.vni  | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
      1 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be			
     20 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be,fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	192.168.123.25,8.8.8.8		192.168.123.24,10.10.10.8	93
      1 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1			
     20 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1,fa:16:3e:42:5a:c5,fa:16:3f:ec:d7:ad	192.168.123.24,10.10.10.8	192.168.123.25,8.8.8.8		93

Controller node:

Same vxlan traffic reflected on vlan50 interface of controller node.

# tshark -tad -n -r overcloud-controller-0.localdomain_vlan50.pcap -p udp -T fields -e eth.addr -e ip.src -e ip.dst | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     10 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be	192.168.123.25	192.168.123.24
     10 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1	192.168.123.24	192.168.123.25

Again did the verification using vxlan decoder.

# tshark -tad -n -r overcloud-controller-0.localdomain_vlan50.pcap -p -d udp.port==51929,vxlan -d udp.port==4789,vxlan -T fields -e eth.addr -e ip.src -e ip.dst -e vxlan.vni  | sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     10 0a:d1:5b:87:f8:e1,e6:36:e1:cc:b7:be,fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	192.168.123.25,8.8.8.8		192.168.123.24,10.10.10.8	93
     10 e6:36:e1:cc:b7:be,0a:d1:5b:87:f8:e1,fa:16:3e:42:5a:c5,fa:16:3f:ec:d7:ad	192.168.123.24,10.10.10.8	192.168.123.25,8.8.8.8		93

capture on br-int interface is showing translation of vlan ids from “1” to “2”. Along with translation of source ip address from “10.10.10.8” to “10.11.49.108”.

# tshark -tad -n -r overcloud-controller-0.localdomain_br-int-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     12 a6:4b:b0:6e:aa:54,fa:16:3e:24:fe:56	10.11.49.108	8.8.8.8	2
     12 fa:16:3e:24:fe:56,a6:4b:b0:6e:aa:54	8.8.8.8	10.11.49.108	2
     12 fa:16:3e:42:5a:c5,fa:16:3e:c0:66:cd	10.10.10.8	8.8.8.8	1
     12 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	8.8.8.8	10.10.10.8	1

This NAT of source ip address happened in snat namespace.

# tshark -tad -n -r overcloud-controller-0.localdomain_sg-ff19cae5-3a.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     12 fa:16:3e:42:5a:c5,fa:16:3e:c0:66:cd	10.10.10.8	8.8.8.8	
     12 fa:16:3e:f1:f8:e1,fa:16:3e:42:5a:c5	8.8.8.8		10.10.10.8	

# tshark -tad -n -r overcloud-controller-0.localdomain_qg-ef46e09a-0b.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     12 a6:4b:b0:6e:aa:54,fa:16:3e:24:fe:56	10.11.49.108	8.8.8.8	
     12 fa:16:3e:24:fe:56,a6:4b:b0:6e:aa:54	8.8.8.8		10.11.49.108	

On br-ex, traffic reported to external world is reported. Noticed the changed vlan id to 11 from internal vlan id 2.

# tshark -tad -n -r overcloud-controller-0.localdomain_br-ex-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst -e vlan.id| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
     12 a6:4b:b0:6e:aa:54,fa:16:3e:24:fe:56	10.11.49.108	8.8.8.8	11
     12 fa:16:3e:24:fe:56,a6:4b:b0:6e:aa:54	8.8.8.8	10.11.49.108	11

All this magic of vxlan to vlan, and from internal vlan to external vlan happens with the help of flow rules.

# ovs-ofctl dump-flows br-int | grep mod_vlan_vid:2
 cookie=0x84d7e40d9e25542b, duration=11259.279s, table=0, n_packets=755, n_bytes=73782, idle_age=4568, priority=3,in_port=1,dl_vlan=11 actions=mod_vlan_vid:2,NORMAL

# ovs-ofctl dump-flows br-ex | grep mod_vlan_vid:11
 cookie=0x809f5e4c110018a5, duration=11288.680s, table=2, n_packets=747, n_bytes=70206, idle_age=4597, priority=4,in_port=3,dl_vlan=2 actions=mod_vlan_vid:11,NORMAL

# ovs-ofctl dump-flows br-tun | grep -i strip
 cookie=0xaf99cd17f7d44000, duration=11363.093s, table=20, n_packets=18, n_bytes=1826, idle_age=10215, priority=2,dl_vlan=1,dl_dst=fa:16:3e:c4:f4:96 actions=strip_vlan,load:0x5d->NXM_NX_TUN_ID[],output:2
 cookie=0xaf99cd17f7d44000, duration=9759.072s, table=20, n_packets=722, n_bytes=69895, idle_age=4703, priority=2,dl_vlan=1,dl_dst=fa:16:3e:f1:f8:e1 actions=strip_vlan,load:0x5d->NXM_NX_TUN_ID[],output:3
 cookie=0xaf99cd17f7d44000, duration=10774.209s, table=22, n_packets=3, n_bytes=262, idle_age=10761, priority=1,dl_vlan=3 actions=strip_vlan,load:0x12->NXM_NX_TUN_ID[],output:2
 cookie=0xaf99cd17f7d44000, duration=9759.382s, table=22, n_packets=1, n_bytes=42, idle_age=9617, priority=1,dl_vlan=1 actions=strip_vlan,load:0x5d->NXM_NX_TUN_ID[],output:2,output:3

It’s time for N/S for instance with floating ip.

Spawned a new instance and attached floating ip with the instance. Network namespaces remains same on controller node but one new fip namespace is appeared on compute node.

[root@overcloud-compute-1 ~]# ip netns list
fip-27989c01-09f9-457a-a750-febd553473ba
qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1

Let’s have a look what is available inside the namespaces.

[root@overcloud-compute-1 ~]# ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: fpr-d463a28e-e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether ca:6e:00:40:bc:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.106.115/31 scope global fpr-d463a28e-e
       valid_lft forever preferred_lft forever
    inet6 fe80::c86e:ff:fe40:bc0c/64 scope link 
       valid_lft forever preferred_lft forever
20: fg-cef4e483-b5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:e4:08:05 brd ff:ff:ff:ff:ff:ff
    inet 10.11.49.109/24 brd 10.11.49.255 scope global fg-cef4e483-b5
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fee4:805/64 scope link 
       valid_lft forever preferred_lft forever

[root@overcloud-compute-1 ~]# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: rfp-d463a28e-e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 6e:cc:5e:a5:05:36 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.106.114/31 scope global rfp-d463a28e-e
       valid_lft forever preferred_lft forever
    inet6 fe80::6ccc:5eff:fea5:536/64 scope link 
       valid_lft forever preferred_lft forever
17: qr-b6129458-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:c0:66:cd brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.1/24 brd 10.10.10.255 scope global qr-b6129458-2e
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec0:66cd/64 scope link 
       valid_lft forever preferred_lft forever

As in this case traffic will not traverse through controller nodes hence all the captures are done on compute node.

ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i qr-b6129458-2e -w /tmp/$(hostname)_qr-b6129458-2e.pcap &
ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 tcpdump -s0 -i rfp-d463a28e-e -w /tmp/$(hostname)_rfp-d463a28e-e.pcap &
ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba tcpdump -s0 -i fpr-d463a28e-e -w /tmp/$(hostname)_fpr-d463a28e-e.pcap &
ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba tcpdump -s0 -i fg-cef4e483-b5 -w /tmp/$(hostname)_fg-cef4e483-b5.pcap &
tcpdump -s0 -i tapda6c7254-26 -w /tmp/$(hostname)_tapda6c7254-26.pcap &
tcpdump  -s0 -i qvoda6c7254-26 -w /tmp/$(hostname)_qvoda6c7254-26.pcacp &
tcpdump -s0 -i br-int-snooper0 -w /tmp/$(hostname)_br-int-snooper.pcap &
tcpdump -s0 -i br-ex-snooper0 -w /tmp/$(hostname)_br-ex-snooper.pcap &
tcpdump -s0 -i eth2 -w /tmp/$(hostname)_eth2.pcap &

Traffic remains same on tap and qvo interface.

# tshark -tad -n -r overcloud-compute-1.localdomain_tapda6c7254-26.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
      4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96	10.10.10.11	8.8.8.8
      4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd	8.8.8.8		10.10.10.11

# tshark -tad -n -r overcloud-compute-1.localdomain_qvoda6c7254-26.pcacp -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
      4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96	10.10.10.11	8.8.8.8
      4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd	8.8.8.8		10.10.10.11

Magic happened at br-int interface. Traffic is going from br-int to qrouter namespace and then from qrouter to fip namespace using patch cable which is joing fpr (fip namespace) and rfp (router namespace).

# tshark -tad -n -r overcloud-compute-1.localdomain_br-int-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
      4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05	10.11.49.100	8.8.8.8
      4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96	10.10.10.11	8.8.8.8
      4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd	8.8.8.8		10.10.10.11
      4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54	8.8.8.8		10.11.49.100

Before packet go from qrouter to fip namespace. NAT happens on the packet to change the IP addresses.

# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 iptables -t nat -L | grep '10.11.49.100'
DNAT       all  --  anywhere             10.11.49.100         to:10.10.10.11
SNAT       all  --  10.10.10.11          anywhere             to:10.11.49.100

# tshark -tad -n -r overcloud-compute-1.localdomain_qr-b6129458-2e.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
Running as user "root" and group "root". This could be dangerous.
      4 fa:16:3e:c0:66:cd,fa:16:3e:c4:f4:96	10.10.10.11	8.8.8.8
      4 fa:16:3e:c4:f4:96,fa:16:3e:c0:66:cd	8.8.8.8		10.10.10.11

qrouter namespace is not using default route table to send the packet to fip namespace this magic happens using non-default route tables.

# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip route
10.10.10.0/24 dev qr-b6129458-2e  proto kernel  scope link  src 10.10.10.1 
169.254.106.114/31 dev rfp-d463a28e-e  proto kernel  scope link  src 169.254.106.114 

# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip rule
0:	from all lookup local 
32766:	from all lookup main 
32767:	from all lookup default 
57481:	from 10.10.10.11 lookup 16 
168430081:	from 10.10.10.1/24 lookup 168430081 

# ip netns exec qrouter-d463a28e-e1f5-48f3-991a-6f90980490e1 ip route list table 16
default via 169.254.106.115 dev rfp-d463a28e-e 

# ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba ip route list table 2852022899
default via 10.11.49.254 dev fg-cef4e483-b5 

# ip netns exec fip-27989c01-09f9-457a-a750-febd553473ba ip neigh | grep '10.11.49.254'
10.11.49.254 dev fg-cef4e483-b5 lladdr a6:4b:b0:6e:aa:54 STALE

External traffic is reported on other interfaces coming in path.

# tshark -tad -n -r overcloud-compute-1.localdomain_fg-cef4e483-b5.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst  | sort | uniq -c
      4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05	10.11.49.100	8.8.8.8
      4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54	8.8.8.8		10.11.49.100

# tshark -tad -n -r overcloud-compute-1.localdomain_br-ex-snooper.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
      4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05	10.11.49.100	8.8.8.8
      4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54	8.8.8.8		10.11.49.100

# tshark -tad -n -r overcloud-compute-1.localdomain_eth2.pcap -p icmp -T fields -e eth.addr -e ip.src -e ip.dst| sort | uniq -c
      4 a6:4b:b0:6e:aa:54,fa:16:3e:e4:08:05	10.11.49.100	8.8.8.8
      4 fa:16:3e:e4:08:05,a6:4b:b0:6e:aa:54	8.8.8.8		10.11.49.100