OpenStack、OpenDayLight、硬件SDN交换机集成

本文将讲述如何在OpenStack中使用SDN交换机,同时将OpenDayLight作为控制器,测试网络类型为vxlan。

本文适用目标为OpenStack Neutron组件,但理论上说包括任何可使用Neutron组件服务的云管理平台,比如oVirt。

在搭建Neutron时,笔者使用了现成的OpenStack单节点平台(可以参考https://github.com/lofyer/openstack-pxe-deployment,为防止出现3.1节末的错误,请使用Newton之前版本,我手贱部署了最新的),但为了减少搭建部署的麻烦,我会在后期尝试直接使用Neutron VM Appliance,方便集成(暂不确定是否影响性能)。

update 2017-05-27: 当前版本zstack的云路由功能vyos所实现的nfv软路由,笔者会考虑将Neutron移植到ZStack中,如此以来SDN即可将注意力主要放在Neutron。

update 2017-06-01: 加入盛科交换机,4.10内核的vtep可直接内核实现,参考Linux doc

update 2017-06-07: OpenStack版本从Ocata切换为Newton,实验3.3重新做了。

update 2017-06-12: 使用 Neutron appliance,并添加hooks,构建ZStack SDN解决方案,来源可自行搜索oVirt的相关资料

1. 环境准备

Pica8 P-5101 48+4*8 10Gb SDN交换机2台(吵地脑袋疼),OpenStack Ocata,OpenDayLight Lastest,管理机器。

mininet虚拟机地址:192.168.0.68

操作机地址:192.168.0.55

2. 架构方案与mininet环境模拟实验

拓扑使用miniedit制作,同时首选项中打开“Start CLI”,操作机“ssh -Y mininet@MININET_IP”后,运行“sudo ~/mininet/examples/mininet.py”以开启。

2.1. 基础模拟实验——交换机互联

文字描述可参考http://blog.scottlowe.org/2012/11/27/connecting-ovs-bridges-with-patch-ports/以及其中链接。

miniedit拓扑:

目的:连接两个交换机,使得h1与h3互通。

方法:将两个交换机上的一个端口组组成patch。

ovs-vsctl \
           -- add-port s1 patch1 \
           -- set interface patch1 type=patch options:peer=patch2 \
           -- add-port s2 patch2 \
           -- set interface patch2 type=patch options:peer=patch1

结果:

root@mininet-vm:/home/mininet# ovs-vsctl show
918037ec-b307-45d7-a75a-f1ac4337d135
    Bridge "s1"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "s1"
            Interface "s1"
                type: internal
        Port "s1-eth2"
            Interface "s1-eth2"
        Port "s1-eth1"
            Interface "s1-eth1"
        Port "patch1"
            Interface "patch1"
                type: patch
                options: {peer="patch2"}
    Bridge "s2"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "s2-eth1"
            Interface "s2-eth1"
        Port "s2"
            Interface "s2"
                type: internal
        Port "patch2"
            Interface "patch2"
                type: patch
                options: {peer="patch1"}
    ovs_version: "2.0.2"

2.2. 进阶实验——流表

miniedit拓扑:同上。

目的:阻断h1与h2、h2与h3,使得h1与h3互通。

方法:下发如下流表。

阻断h2与h1/h3的协议封包:
ovs-ofctl add-flow s1 in_port=2,arp,ip,nw_dst=10.0.0.1,actions=drop
ovs-ofctl add-flow s1 in_port=2,arp,ip,nw_dst=10.0.0.3,actions=drop

结果

mininet> h1 ping h2 -c 2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
^C
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1006ms

mininet> h2 ping h1 -c 2
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

mininet> h3 ping h2 -c 2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
^C
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms

mininet> h2 ping h3 -c 2
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
^C
--- 10.0.0.3 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1006ms

mininet> h1 ping h3 -c 2
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=2.37 ms
64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.561 ms

--- 10.0.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.561/1.466/2.372/0.906 ms
mininet> h3 ping h1 -c 2
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=2.49 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.548 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.548/1.521/2.494/0.973 ms
mininet> 

root@mininet-vm:~# ovs-ofctl dump-flows s1
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=485.308s, table=0, n_packets=0, n_bytes=0, idle_age=485, ip,in_port=2,nw_dst=10.0.0.1 actions=drop
 cookie=0x0, duration=483.569s, table=0, n_packets=3, n_bytes=294, idle_age=183, ip,in_port=2,nw_dst=10.0.0.3 actions=drop
 cookie=0x0, duration=1715.894s, table=0, n_packets=93, n_bytes=3906, idle_age=202, arp,in_port=1,arp_tpa=10.0.0.2 actions=drop
 cookie=0x0, duration=715.43s, table=0, n_packets=4, n_bytes=392, idle_age=400, ip,in_port=1,nw_dst=10.0.0.2 actions=drop

3. 综合实验环境配置

首先来看一下实验拓扑:

图。。。。。。

SDN交换机
管理地址:192.168.0.250/24,

OpenStack Controller/Neutron
IP: 192.168.0.80

OpenDayLight
IP: 192.168.0.90
Web: http://192.168.0.90:8181/index.html

3.1. 设置OpenDayLight

在主机192.168.0.90上下载启动Boron版本OpenDayLight(其他版本未测试),然后解压运行(需要Java环境):

unzip distribution-karaf-0.5.3-Boron-SR3.zip
cd distribution-karaf-0.5.3-Boron-SR3
# 启动ODL服务
./bin/start
# 本地安装并启动OpenvSwitch以作测试
yum localinstall -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
systemctl enable openvswitch
systemctl start openvswitch
ovs-vsctl set-manager tcp:192.168.0.90:6640
# 进入客户端
./bin/client
# 安装netvirt与dlux界面,其中yangman为包管理器UI
opendaylight-user@root> feature:install odl-netvirt-openstack odl-dlux-all odl-dlux-yangman odl-mdsal-apidocs odl-netvirt-ui

然后访问http://192.168.0.90:8181/index.html,用户密码为admin/admin,可以看到我们刚刚添加的本地OVS。

3.3. OpenStack与OpenDayLight集成

首先要清除原有的实例和neutron网络。

# 实例
openstack server list
+--------------------------------------+-----------+---------+------------+-------------+------------------------------------------------------+
| ID                                   | Name      | Status  | Task State | Power State | Networks                                             |
+--------------------------------------+-----------+---------+------------+-------------+------------------------------------------------------+
| d45b1646-b559-4d6d-963c-af3da205aa36 | instance3 | SHUTOFF | -          | Shutdown    | flat-ens39=192.168.0.108; sharednet1=192.168.118.203 |
+--------------------------------------+-----------+---------+------------+-------------+------------------------------------------------------+

openstack server delete instance3

# 网络

openstack subnet list
+--------------------------------------+-------------+--------------------------------------+------------------+
| ID                                   | Name        | Network                              | Subnet           |
+--------------------------------------+-------------+--------------------------------------+------------------+
| b0acb3ad-22d4-491c-94e7-488aca906398 | flat-subnet | b8f95552-9a45-49ae-b080-2e0041d4b2a0 | 192.168.0.0/24   |
| c066992d-294d-4cd6-a3fb-0386619922c7 | subnet1     | 5ac25e6f-bf49-4217-abbc-11bf171c0a3e | 192.168.118.0/24 |
+--------------------------------------+-------------+--------------------------------------+------------------+

openstack router list
+------------------------------+-------------------+--------+-------+-------------+-------+------------------------------+
| ID                           | Name              | Status | State | Distributed | HA    | Project                      |
+------------------------------+-------------------+--------+-------+-------------+-------+------------------------------+
| 78507b26-38f0-4b4a-          | router-flat-ens39 | ACTIVE | UP    | False       | False | de2aea51161642759a687b5768b2 |
| a3c6-13d15c4f9665            |                   |        |       |             |       | 3b7e                         |
+------------------------------+-------------------+--------+-------+-------------+-------+------------------------------+
........
# 删除所有port、router、subnet后,再用如下命令检查一遍,应该为空
openstack port list

接下来,我们将控制节点(neutron管理节点)与计算节点的OVS交于ODL管理。
首先停止相应服务。

# 计算节点,如果与控制节点相同则优先运行此部分内容
systemctl stop neutron-openvswitch-agent
systemctl disable neutron-openvswitch-agent
systemctl stop neutron-l3-agent
systemctl disable neutron-l3-agent

# Neutron控制节点
systemctl stop neutron-server
systemctl stop neutron-l3-agent

然后清空OVS数据库。

systemctl stop openvswitch
rm -rf /var/log/openvswitch/*
rm -rf /etc/openvswitch/conf.db
systemctl start openvswitch
# 清空后可看到如下
ovs-vsctl show
bdf45776-c7c2-4df4-9911-007e89b67bbe
    ovs_version: "2.6.1"

将其交予ODL管理。

ovs-vsctl set-manager tcp:192.168.0.90:6640

然后在控制节点与计算节点设置用来vxlan通信的本地端,这里只有一个节点192.168.0.80。

ovs-vsctl set Open_vSwitch . other_config:local_ip=192.168.0.80

设置好以后可以在节点上看到ODL自动创建了一个连接到ODL控制器的br-int。

ovs-vsctl show
bdf45776-c7c2-4df4-9911-007e89b67bbe
    Manager "tcp:192.168.0.90:6640"
        is_connected: true
    Bridge br-int
        Controller "tcp:192.168.0.90:6653"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.6.1"

ovs-vsctl get Open_vSwitch . other_config
{local_ip="192.168.0.80"}

此时reloadODL界面,可以看到多了个OVS。

接下来让neutron使用ODL,我们首先需要在neutron控制节点上安装一个对应包。

对于将ODL放置于OS控制节点的同学可以参考如下内容以修改swift端口:

First, ensure that port 8080 (which will be used by OpenDaylight to listen for REST calls) is available. By default, swift-proxy-service listens on the same port, and you may need to move it (to another port or another host), or disable that service. It can be moved to a different port (e.g. 8081) by editing /etc/swift/proxy-server.conf and /etc/cinder/cinder.conf, modifying iptables appropriately, and restarting swift-proxy-service. Alternatively, OpenDaylight can be configured to listen on a different port, by modifying the jetty.port property value in etc/jetty.conf.

yum install -y python-networking-odl

修改/etc/neutron/plugins/ml2/ml2_conf.ini并添加内容。

crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers opendaylight
crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan

# ml2_conf_odl.ini中的设置无效且报错,可能是我姿势不对,原因是我忘了修改neutron-server的启动命令。
cat > /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2_odl]
url = http://192.168.0.90:8080/controller/nb/v2/neutron
password = admin
username = admin
EOF

然后修改服务后端为ODL。

crudini --set /etc/neutron/neutron.conf DEFAULT service_plugins odl-router
crudini --set /etc/neutron/dhcp_agent.ini DEFAULT force_metadata True
crudini --set /etc/neutron/dhcp_agent.ini ovs ovsdb_interface vsctl

重置neutron数据库并重启服务。

mysql -e "DROP DATABASE IF EXISTS neutron_ml2;" -uroot -p
mysql -e "CREATE DATABASE neutron_ml2 CHARACTER SET utf8;" -uroot -p
neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head
systemctl start neutron-server

验证是否成功。

curl -u admin:admin http://192.168.0.90:8080/controller/nb/v2/neutron/networks
{
   "networks" : [ ]
}

最后在openstack中创建网络与实例。

neutron router-create router1
neutron net-create private
neutron subnet-create private --name=private_subnet 10.10.5.0/24
neutron router-interface-add router1 private_subnet
nova boot --flavor <flavor> --image <image id> --nic net-id=<network id> test1
nova boot --flavor <flavor> --image <image id> --nic net-id=<network id> test2

添加浮动IP。

ovs-vsctl set Open_vSwitch . other_config:provider_mappings=physnet1:eth1
neutron net-create public-net -- --router:external --is-default --provider:network_type=flat --provider:physical_network=physnet1
neutron subnet-create --allocation-pool start=10.10.10.2,end=10.10.10.254 --gateway 10.10.10.1 --name public-subnet public-net 10.10.0.0/16 -- --enable_dhcp=False
neutron router-gateway-set router1 public-net

neutron floatingip-create public-net
nova floating-ip-associate test1 <floating_ip>

在OpenStack O版中创建网络时ODL与OS分别出现如下错误。

ODL:
2017-06-03 11:27:10,768 | ERROR | pool-46-thread-1 | QosInterfaceStateChangeListener  | 350 - org.opendaylight.netvirt.neutronvpn-impl - 0.3.3.Boron-SR3 | Qos:Exception caught in Interface Operational State Up event
java.lang.IllegalArgumentException: Supplied value "tapd5e5b153-39" does not match required pattern "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$"
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)[65:com.google.guava:18.0.0]
	at org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.yang.types.rev130715.Uuid.<init>(Uuid.java:55)[80:org.opendaylight.mdsal.model.ietf-yang-types-20130715:2013.7.15.9_3-Boron-SR3]
	at org.opendaylight.netvirt.neutronvpn.QosInterfaceStateChangeListener.add(QosInterfaceStateChangeListener.java:61)[350:org.opendaylight.netvirt.neutronvpn-impl:0.3.3.Boron-SR3]
	at org.opendaylight.netvirt.neutronvpn.QosInterfaceStateChangeListener.add(QosInterfaceStateChangeListener.java:27)[350:org.opendaylight.netvirt.neutronvpn-impl:0.3.3.Boron-SR3]
	at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:136)[310:org.opendaylight.genius.mdsalutil-api:0.1.3.Boron-SR3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

OS:
ERROR neutron.plugins.ml2.managers [req-8174eb6e-6a01-428e-9c3d-d9f115dfa36b - - - - -] Failed to bind port d5e5b153-390a-4129-babd-7work': None, 'id': u'f01e69c5-096e-4d01-a239-5dbe37005e9b', 'network_type': u'vxlan'}]

原因可能是版本传参变了,ODL没及时更新,但我不会Java,会也不想去改ODL,所以我推荐读者用个老版本的Neutron,比如M版、N版啥的。

3.4. OVS创建vtep

IP: 192.168.0.101

3.5. OpenStack与SDN交换机集成

在ODL结束以后,我们再来试一下OpenStack与SDN交换机的集成,与上一节相互独立,即一个全新的OpenStack Newton环境。

由于商业法务问题,我不会将Pica8的OpenStack插件源码全部放出来,另外放出来意义也不大,现有SDN交换机的同学可以直接向厂商索要。

/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/

Pica8交换机工作在OVS模式下,需要将neutron server与OVS Manager相连(管理网口),或许有其他方法。

首先将插件文件拷贝至/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/,目录结构如下。

/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/pica8/
├── config.pyc
├── db.pyc
├── exceptions.pyc
├── __init__.pyc
├── mechanism_pica8.pyc
├── ml2_conf_pica8.ini
├── rpc.pyc
└── vtep.ovsschema

修改/usr/lib/python2.7/site-packages/neutron-9.3.1-py2.7.egg-info/entry_points.txt添加neutron驱动。

...
[neutron.ml2.mechanism_drivers]
...
pica8 = neutron.plugins.ml2.drivers.pica8.mechanism_pica8:Pica8Driver
...

创建/etc/neutron/plugins/ml2/ml2_conf_pica8.ini,并添加如下内容。

[ml2_pica8]
# (ListOpt) List of other VTEPs' IP address, either a software vtep or
# other vendor's hardware vtep.
#
# Example: vtep_list=10.0.0.100,10.0.0.101,10.0.0.102
vtep_list=192.168.0.101
# (IntOpt) Sync interval in seconds between Neutron plugin and PicaOS.
# This field defines how often the synchronization is performed.
# This is an optional field. If not set, a value of 180 seconds
# is assumed.
#
sync_interval = 60
# Example: sync_interval = 60
openstack_version = newton
# Example: openstack_version = kilo
# PicOS Switch configurations.
# Each switch to be managed by Openstack Neutron must be configured here.
# Format:
[ml2_mech_pica_switch:192.168.0.250]
=:,,...
=:
# =:project_name,,vni,
# ...
# ovsdb_port=
#
# Example:
# [ml2_mech_pica_switch:192.168.0.250]
# te-1/1/1=physnet1:dev-1,dev-2,dev-3
# te-1/1/2=physnet2:dev-4
# te-1/1/3=physnet1:project_name,admin,vni,3535
# ovsdb_port=6640
# source_ip=192.168.0.80

修改/usr/lib/systemd/system/neutron-server.service文件,在${DAEMON_ARGS}字段添加pica8的引导配置,需要停止neutron-server服务并systemctl daemon-reload。

--config-file=/etc/neutron/plugins/ml2/ml2_conf_pica8.ini

修改/etc/neutron/plugins/ml2/ml2_conf.ini,如下。

type_drivers = vlan,vxlan
project_network_types = vxlan
mechanism_drivers = pica8,openvswitch
# Make sure the <physical_network> is consistent in the following configuration
bridge_mappings=<physical_network>:br-ex
network_vlan_ranges = <physical_network>:1:4094

安装python-ovs库。

yum install -y python-pip
pip install ovs

运行如下命令创建pica8插件数据库动作。

neutron-db-manage --config-file /etc/neutron/neutron.conf \
--config-file /etc/neutron/plugins/ml2/ml2_conf.ini \
revision -m "add pica8 mechanism driver" --expand

在生成的/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/versions/newton/expand/94ca4fc9191b_add_pica8_mechanism_driver.py文件中添加如下内容。

def upgrade():
    op.create_table(
        'pica8_interfaces_v2',
        sa.Column('id', sa.String(length=36), nullable=False,
            primary_key=True),
        sa.Column('switch', sa.String(length=36), nullable=False),
        sa.Column('interface', sa.String(10), nullable=False)
        )
    op.create_table(
        'pica8_vlan_allocations_v2',
        sa.Column('id', sa.String(length=36), nullable=False,
        primary_key=True),
        sa.Column('project_id', sa.String(length=255), nullable=False),
        sa.Column('network_id', sa.String(length=36), nullable=False),
        sa.Column('segmentation_id', sa.Integer, nullable=False),
        sa.Column('vlan_id', sa.Integer, nullable=False),
        sa.Column('vm_reference', sa.Integer, nullable=False, default=0),
        sa.Column('interface_id', sa.String(length=36), nullable=True),
        sa.ForeignKeyConstraint(['interface_id'], ['pica8_interfaces_v2.id'],
        ondelete='CASCADE')
    )
def downgrade():
    op.drop_table('pica8_interfaces')
    op.drop_table('pica8_vlan_allocations_v2')

执行如下命令升级数据库(如果失败请将最后参数替换为head)。

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade 94ca4fc9191b

重启服务。

systemctl restart openstack-nova-api
systemctl restart neutron-server

物理主机接入配置。

Edit the file /etc/neutron/plugins/ml2/ml2_conf_pica8.ini to contain the following content:
<interface_name>=<physical_network>:project_name,<project_name>,vni,<vxlan_id>
<interface_name> is the interface of Pica8 switch connected to the physical host
< project_name > is the name of the project owning the physical host
<vxlan_id> is the vxlan id of the project network the physical host connects to

miniedit使用
Pica8手册
ODL手册
OpenDaylight with Openstack Guide
OpenStack with NetVirt
Installing OpenStack and OpenDaylight using DevStack
https://wiki.opendaylight.org/images/5/59/CloudIntegrationwithOpenStackOVSDBNetVirt.pdf

One thought on “OpenStack、OpenDayLight、硬件SDN交换机集成

  1. 您好,接私活吗?指导我们用openstack+ODL+PICA8测试
    电话:18963090522

发表评论

您的电子邮箱地址不会被公开。