Solaris 11 IP Multipathing VLANs in Zones

I just started with Solaris 11 and recognized first that the networking works totally different than before and also different from other known OSs.

While the Oracle documentation on Solaris 11 is quite good, it only takes on one topic at a time.

So you have one documentation for IP configuration, one for VLAN, one for IP Multipathing, one for zones, but what if you want VLAN with IP Multipathing in a zone?

I´ve thought about the ways, how I did this in Solaris 10. There are basically two methods:

1) You create two VLAN interfaces, give them exclusive to the zone and then configure IPMP in the zone.

2) You create two VLAN interfaces on top of two other interfaces, but do the IPMP configuration already in the global zone and give it as shared interface to your zone.

With the first method you have the problem, that no other zone can use the same VLAN on that interface, because it's used exclusive by the other zone.

And with the second method you are not so free in doing your network configuration more independent in your zone.

Now I tried to apply these two methods to Solaris 11:

Method 1

Create 2 VLAN interfaces, put them as physical to the zone and create an ipmp in the zone.

In Global-Zone

In my virtual machine to test this, I had two network interfaces.

root@ahutest:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --

So I create two VLAN interfaces for VLAN ID 13.

root@ahutest:~# dladm create-vlan -v 13 -l net0 v13app0
root@ahutest:~# dladm create-vlan -v 13 -l net1 v13app1
root@ahutest:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --
v13app0             vlan      1500   up       net0
v13app1             vlan      1500   up       net1

Then I create a zone and gave it to the two interfaces exclusive.

root@ahutest:~# zonecfg -z app
app: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:app> create
create: Using system default template 'SYSdefault'
zonecfg:app> set zonepath=/zones/app
zonecfg:app> set autoboot=true
zonecfg:app> remove anet
zonecfg:app> add net
zonecfg:app:net> set physical=v13app0
zonecfg:app:net> end
zonecfg:app> add net
zonecfg:app:net> set physical=v13app1
zonecfg:app:net> end
zonecfg:app> verify
zonecfg:app> commit
zonecfg:app> exit

And the usual install and boot console configure stuff.

root@ahutest:~# zoneadm -z app install
root@ahutest:~# zoneadm -z app boot
root@ahutest:~# zlogin -C -e \# app
(I just like #. the most to exit the console)

F2 - enter host name - F2 enter other stuff - F2 F2 F2.

In Non-Global-Zone

Create the interfaces for VLAN and put them together in an ipmp0 interface.

ipadm create-ip v13app0
ipadm create-ip v13app1
ipadm create-ipmp -i v13app0 -i v13app1 ipmp0

Then just do the normal static IP configuration for that interface.

ipadm create-addr -T static -a 192.168.211.47/25 ipmp0/v4
route -p add default 192.168.211.1

Before proceeding with Method 2, we need to halt the zone, remove the interfaces and start again, because the interfaces are given exclusive to the zone and are already in use.

root@ahutest:~# zoneadm -z app halt
root@ahutest:~# dladm delete-vlan v13app0
root@ahutest:~# dladm delete-vlan v13app1

Method 2

Create VLAN ipmp already in global zone, give it as virtual interface to non-global zone.

In Global-Zone

We create the VLAN devices again with id 13 and put them already in an ipmp together in the global zone.

root@ahutest:~# dladm create-vlan -v 13 -l net0 v13db0
root@ahutest:~# dladm create-vlan -v 13 -l net1 v13db1
root@ahutest:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --
v13db0              vlan      1500   up       net0
v13db1              vlan      1500   up       net1

root@ahutest:~# ipadm create-ip v13db0
root@ahutest:~# ipadm create-ip v13db1
root@ahutest:~# ipadm create-ipmp -i v13db0 -i v13db1 v13ipmp0
root@ahutest:~# ipadm show-if
IFNAME     CLASS    STATE    ACTIVE OVER
lo0        loopback ok       yes    --
ipmp0      ipmp     ok       yes    net0 net1
(yes, I already have an ipmp in my globl zone, wanna be safe there, too! ;)
net0       ip       ok       yes    --
net1       ip       ok       yes    --
v13db0     ip       ok       yes    --
v13db1     ip       ok       yes    --
v13ipmp0   ipmp     down     no     v13db0 v13db1

Now we create the zone, and give it shared to the zone.

zonecfg -z db
db: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:db> create
create: Using system default template 'SYSdefault'
zonecfg:db> set zonepath=/zones/db
zonecfg:db> set autoboot=true
zonecfg:db> remove anet
zonecfg:db> set ip-type=shared
zonecfg:db> add net
zonecfg:db:net> set address=192.168.211.48
zonecfg:db:net> set configure-allowed-address=false
zonecfg:db:net> set defrouter=192.168.211.1
zonecfg:db:net> set physical=v13ipmp0
zonecfg:db:net> end
zonecfg:db> verify
zonecfg:db> commit
zonecfg:db> exit

Then again, install the zone, boot it, login to it and F2 through the setup.

root@ahutest:~# zoneadm -z db install
root@ahutest:~# zoneadm -z db boot
root@ahutest:~# zlogin -C -e \# db

F2 and F2 till it's done, this time it won't ask you for IP configuration, because it's already set in the zone config. And there you go:

root@db:~# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/?             from-gz  ok           127.0.0.1/8
v13ipmp0/?        from-gz  ok           192.168.211.48/24
lo0/?             from-gz  ok           ::1/128

And thats it! If you now change the zone config for the app zone to use shared and another IP then this zone you can connect them to each other.

Now image you have not just one VLAN but dozens of VLANs and would have to create ipmp VLAN interfaces in the global zone for each one of them!

When you want to export your VLAN zones and import it somewhere else, you need to make VLAN ipmp configuration there, too.

But in Solaris 11 there is a third way to do this or more specific. There is another way to do it like my Method 1, but other zones can use the interfaces for same VLAN as well! So here is my

Method 3

Again I uninstall my zones and delete all the interfaces I created before to start from scratch! So there are no VLAN interfaces anymore:

root@ahutest:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --

Lets go and configure the zone right away.

root@ahutest:~# zonecfg -z app
app: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:app> create
create: Using system default template 'SYSdefault'
zonecfg:app> set zonepath=/zones/app
zonecfg:app> set autoboot=true
zonecfg:app> select anet linkname=net0
zonecfg:app:anet> set linkname=app0
zonecfg:app:anet> set vlan-id=13
zonecfg:app:anet> set lower-link=net0
zonecfg:app:anet> end
zonecfg:app> add anet
zonecfg:app:anet> set linkname=app1
zonecfg:app:anet> set vlan-id=13
zonecfg:app:anet> set lower-link=net1
zonecfg:app:anet> end
zonecfg:app> verify
zonecfg:app> commit
zonecfg:app> exit

So that's something really new! You can set the VLAN ID directly in the zone config, no matter which interface you gave to it.

With "add anet" Solaris 11 will create a vnic each time the zone is starting. So again, we can install and boot our new zone.

root@ahutest:~# zoneadm -z app install
root@ahutest:~# zoneadm -z app boot
zlogin -C -e \# app

F2 F2 F2 and login. Now we have two VLAN virtual interfaces.

root@app:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
app0                vnic      1500   up       ?
app1                vnic      1500   up       ?

Setup the IP, like we did in Method 1.

root@app:~# ipadm create-ip app1
root@app:~# ipadm create-ipmp -i app0 -i app1 ipmp0
root@app:~# ipadm show-if
IFNAME     CLASS    STATE    ACTIVE OVER
lo0        loopback ok       yes    --
app0       ip       ok       yes    --
app1       ip       ok       yes    --
ipmp0      ipmp     down     no     app0 app1
root@app:~# ipadm create-addr -T static -a 192.168.211.47/25 ipmp0/v4
root@app:~# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
ipmp0/v4          static   ok           192.168.211.47/25
lo0/v6            static   ok           ::1/128
app0/_a           static   ok           fe80::8:20ff:fe12:86fb/10
app1/_a           static   ok           fe80::8:20ff:fe66:4c9c/10

In the global Zone we can now see these vnics with VLAN.

root@ahutest:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --
app/app0            vnic      1500   up       net0
app/app1            vnic      1500   up       net1

So in the same way I can create a second zone.

root@ahutest:~# zonecfg -z db
db: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:db> create
create: Using system default template 'SYSdefault'
zonecfg:db> set zonepath=/zones/db
zonecfg:db> set autoboot=true
zonecfg:db> select anet linkname=net0
zonecfg:db:anet> set linkname=db0
zonecfg:db:anet> set vlan-id=13
zonecfg:db:anet> set lower-link=net0
zonecfg:db:anet> end
zonecfg:db> add anet
zonecfg:db:anet> set linkname=db1
zonecfg:db:anet> set vlan-id=13
zonecfg:db:anet> set lower-link=net1
zonecfg:db:anet> end
zonecfg:db> verify
zonecfg:db> commit
zonecfg:db> exit

Install, Boot, config as usual.

root@ahutest:~# zoneadm -z db install
root@ahutest:~# zoneadm -z db boot
zlogin -C -e \# db

F2 F2 F2 and login. After login, do the IP/IPMP config.

root@db:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
db0                 vnic      1500   up       ?
db1                 vnic      1500   up       ?
root@db:~# ipadm create-ip db0
root@db:~# ipadm create-ip db1
root@db:~# ipadm create-ipmp -i db0 -i db1 ipmp0
root@db:~# ipadm show-if
IFNAME     CLASS    STATE    ACTIVE OVER
lo0        loopback ok       yes    --
db0        ip       ok       yes    --
db1        ip       ok       yes    --
ipmp0      ipmp     down     no     db0 db1
root@db:~# ipadm create-addr -T static -a 192.168.211.48/25 ipmp0/v4
root@db:~# route -p add default 192.168.211.1
root@db:~# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
ipmp0/v4          static   ok           192.168.211.48/25
lo0/v6            static   ok           ::1/128
db0/_a            static   ok           fe80::8:20ff:fe1a:3ab0/10
db1/_a            static   ok           fe80::8:20ff:fe69:a4c1/10

In the global zone it now looks like this:

root@ahutest:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --
app/app0            vnic      1500   up       net0
app/app1            vnic      1500   up       net1
db/db0              vnic      1500   up       net0
db/db1              vnic      1500   up       net1

Generally this third method should work this way, but sadly in my tryout I was just able to ping the other zone seconds after I had set it up and then not again.

I opened a case at Oracle Support for this and hope that I will learn what I did wrong here or if this is maybe a bug. I will update this blog entry if I know more about the matter.

Maybe there are still other and better ways to do this. This was just my first tryout with Solaris 11 networking. If you have any suggestions or hints how to do it better, do not hesitate to leave me a comment.

Andreas Hubert

Andreas Hubert ist stellvertretender Leiter der IT Services bei censhare. Außerdem kümmert er sich als Ausbildungsleiter um den fachlichen Nachwuchs. Als gelernter Fachinformatiker sind seine Schwerpunkte Netzwerk Administration und Linux/Solaris. Bei Twitter kann man mit ihm zu den Themen Apple, iPhone, Oracle Database, Sun Solaris, Sicherheitslücken, ´nem Caffè Latte vor seinem Mac und natürlich censhare zwitschern.

Neuer Kommentar: