Aktuelles

06.03.2012 | Kommentare: 8

Solaris 11 IP Multipathing VLANs in Zones

I just started with Solaris 11 and recognized first that the networking works totally different than before and also different from other known OSs.

While the Oracle documentation on Solaris 11 is quite good, it only takes on one topic at a time.

So you have one documentation for IP configuration, one for VLAN, one for IP Multipathing, one for zones, but what if you want VLAN with IP Multipathing in a zone?

I´ve thought about the ways, how I did this in Solaris 10. There are basically two methods:

1) You create two VLAN interfaces, give them exclusive to the zone and then configure IPMP in the zone.

2) You create two VLAN interfaces on top of two other interfaces, but do the IPMP configuration already in the global zone and give it as shared interface to your zone.

With the first method you have the problem, that no other zone can use the same VLAN on that interface, because it's used exclusive by the other zone.

And with the second method you are not so free in doing your network configuration more independent in your zone.

Now I tried to apply these two methods to Solaris 11:

Method 1

Create 2 VLAN interfaces, put them as physical to the zone and create an ipmp in the zone.

In Global-Zone

In my virtual machine to test this, I had two network interfaces.

root@ahutest:~# dladm show-link
LINK CLASS MTU STATE OVER
net1 phys 1500 up --
net0 phys 1500 up --

So I create two VLAN interfaces for VLAN ID 13.

root@ahutest:~# dladm create-vlan -v 13 -l net0 v13app0
root@ahutest:~# dladm create-vlan -v 13 -l net1 v13app1
root@ahutest:~# dladm show-link
LINK CLASS MTU STATE OVER
net1 phys 1500 up --
net0 phys 1500 up --
v13app0 vlan 1500 up net0
v13app1 vlan 1500 up net1

Then I create a zone and gave it to the two interfaces exclusive.

root@ahutest:~# zonecfg -z app
app: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:app> create
create: Using system default template 'SYSdefault'
zonecfg:app> set zonepath=/zones/app
zonecfg:app> set autoboot=true
zonecfg:app> remove anet
zonecfg:app> add net
zonecfg:app:net> set physical=v13app0
zonecfg:app:net> end
zonecfg:app> add net
zonecfg:app:net> set physical=v13app1
zonecfg:app:net> end
zonecfg:app> verify
zonecfg:app> commit
zonecfg:app> exit

And the usual install and boot console configure stuff.

root@ahutest:~# zoneadm -z app install
root@ahutest:~# zoneadm -z app boot
root@ahutest:~# zlogin -C -e \# app
(I just like #. the most to exit the console)

F2 - enter host name - F2 enter other stuff - F2 F2 F2.

In Non-Global-Zone

Create the interfaces for VLAN and put them together in an ipmp0 interface.

ipadm create-ip v13app0
ipadm create-ip v13app1
ipadm create-ipmp -i v13app0 -i v13app1 ipmp0

Then just do the normal static IP configuration for that interface.

ipadm create-addr -T static -a 192.168.211.47/25 ipmp0/v4
route -p add default 192.168.211.1

Before proceeding with Method 2, we need to halt the zone, remove the interfaces and start again, because the interfaces are given exclusive to the zone and are already in use.

root@ahutest:~# zoneadm -z app halt
root@ahutest:~# dladm delete-vlan v13app0
root@ahutest:~# dladm delete-vlan v13app1

Method 2

Create VLAN ipmp already in global zone, give it as virtual interface to non-global zone.

In Global-Zone

We create the VLAN devices again with id 13 and put them already in an ipmp together in the global zone.

root@ahutest:~# dladm create-vlan -v 13 -l net0 v13db0
root@ahutest:~# dladm create-vlan -v 13 -l net1 v13db1
root@ahutest:~# dladm show-link
LINK CLASS MTU STATE OVER
net1 phys 1500 up --
net0 phys 1500 up --
v13db0 vlan 1500 up net0
v13db1 vlan 1500 up net1
root@ahutest:~# ipadm create-ip v13db0
root@ahutest:~# ipadm create-ip v13db1
root@ahutest:~# ipadm create-ipmp -i v13db0 -i v13db1 v13ipmp0
root@ahutest:~# ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
ipmp0 ipmp ok yes net0 net1
(yes, I already have an ipmp in my globl zone, wanna be safe there, too! ;)
net0 ip ok yes --
net1 ip ok yes --
v13db0 ip ok yes --
v13db1 ip ok yes --
v13ipmp0 ipmp down no v13db0 v13db1

Now we create the zone, and give it shared to the zone.

zonecfg -z db
db: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:db> create
create: Using system default template 'SYSdefault'
zonecfg:db> set zonepath=/zones/db
zonecfg:db> set autoboot=true
zonecfg:db> remove anet
zonecfg:db> set ip-type=shared
zonecfg:db> add net
zonecfg:db:net> set address=192.168.211.48
zonecfg:db:net> set configure-allowed-address=false
zonecfg:db:net> set defrouter=192.168.211.1
zonecfg:db:net> set physical=v13ipmp0
zonecfg:db:net> end
zonecfg:db> verify
zonecfg:db> commit
zonecfg:db> exit

Then again, install the zone, boot it, login to it and F2 through the setup.

root@ahutest:~# zoneadm -z db install
root@ahutest:~# zoneadm -z db boot
root@ahutest:~# zlogin -C -e \# db

F2 and F2 till it's done, this time it won't ask you for IP configuration, because it's already set in the zone config. And there you go:

root@db:~# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/? from-gz ok 127.0.0.1/8
v13ipmp0/? from-gz ok 192.168.211.48/24
lo0/? from-gz ok ::1/128

And thats it! If you now change the zone config for the app zone to use shared and another IP then this zone you can connect them to each other.

Now image you have not just one VLAN but dozens of VLANs and would have to create ipmp VLAN interfaces in the global zone for each one of them!

When you want to export your VLAN zones and import it somewhere else, you need to make VLAN ipmp configuration there, too.

But in Solaris 11 there is a third way to do this or more specific. There is another way to do it like my Method 1, but other zones can use the interfaces for same VLAN as well! So here is my

Method 3

Again I uninstall my zones and delete all the interfaces I created before to start from scratch! So there are no VLAN interfaces anymore:

root@ahutest:~# dladm show-link
LINK CLASS MTU STATE OVER
net1 phys 1500 up --
net0 phys 1500 up --

Lets go and configure the zone right away.

root@ahutest:~# zonecfg -z app
app: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:app> create
create: Using system default template 'SYSdefault'
zonecfg:app> set zonepath=/zones/app
zonecfg:app> set autoboot=true
zonecfg:app> select anet linkname=net0
zonecfg:app:anet> set linkname=app0
zonecfg:app:anet> set vlan-id=13
zonecfg:app:anet> set lower-link=net0
zonecfg:app:anet> end
zonecfg:app> add anet
zonecfg:app:anet> set linkname=app1
zonecfg:app:anet> set vlan-id=13
zonecfg:app:anet> set lower-link=net1
zonecfg:app:anet> end
zonecfg:app> verify
zonecfg:app> commit
zonecfg:app> exit

So that's something really new! You can set the VLAN ID directly in the zone config, no matter which interface you gave to it.

With "add anet" Solaris 11 will create a vnic each time the zone is starting. So again, we can install and boot our new zone.

root@ahutest:~# zoneadm -z app install
root@ahutest:~# zoneadm -z app boot
zlogin -C -e \# app

F2 F2 F2 and login. Now we have two VLAN virtual interfaces.

root@app:~# dladm show-link
LINK CLASS MTU STATE OVER
app0 vnic 1500 up ?
app1 vnic 1500 up ?

Setup the IP, like we did in Method 1.

root@app:~# ipadm create-ip app1
root@app:~# ipadm create-ipmp -i app0 -i app1 ipmp0
root@app:~# ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
app0 ip ok yes --
app1 ip ok yes --
ipmp0 ipmp down no app0 app1
root@app:~# ipadm create-addr -T static -a 192.168.211.47/25 ipmp0/v4
root@app:~# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
ipmp0/v4 static ok 192.168.211.47/25
lo0/v6 static ok ::1/128
app0/_a static ok fe80::8:20ff:fe12:86fb/10
app1/_a static ok fe80::8:20ff:fe66:4c9c/10

In the global Zone we can now see these vnics with VLAN.

root@ahutest:~# dladm show-link
LINK CLASS MTU STATE OVER
net1 phys 1500 up --
net0 phys 1500 up --
app/app0 vnic 1500 up net0
app/app1 vnic 1500 up net1

So in the same way I can create a second zone.

root@ahutest:~# zonecfg -z db
db: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:db> create
create: Using system default template 'SYSdefault'
zonecfg:db> set zonepath=/zones/db
zonecfg:db> set autoboot=true
zonecfg:db> select anet linkname=net0
zonecfg:db:anet> set linkname=db0
zonecfg:db:anet> set vlan-id=13
zonecfg:db:anet> set lower-link=net0
zonecfg:db:anet> end
zonecfg:db> add anet
zonecfg:db:anet> set linkname=db1
zonecfg:db:anet> set vlan-id=13
zonecfg:db:anet> set lower-link=net1
zonecfg:db:anet> end
zonecfg:db> verify
zonecfg:db> commit
zonecfg:db> exit

Install, Boot, config as usual.

root@ahutest:~# zoneadm -z db install
root@ahutest:~# zoneadm -z db boot
zlogin -C -e \# db

F2 F2 F2 and login. After login, do the IP/IPMP config.

root@db:~# dladm show-link
LINK CLASS MTU STATE OVER
db0 vnic 1500 up ?
db1 vnic 1500 up ?
root@db:~# ipadm create-ip db0
root@db:~# ipadm create-ip db1
root@db:~# ipadm create-ipmp -i db0 -i db1 ipmp0
root@db:~# ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
db0 ip ok yes --
db1 ip ok yes --
ipmp0 ipmp down no db0 db1
root@db:~# ipadm create-addr -T static -a 192.168.211.48/25 ipmp0/v4
root@db:~# route -p add default 192.168.211.1
root@db:~# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
ipmp0/v4 static ok 192.168.211.48/25
lo0/v6 static ok ::1/128
db0/_a static ok fe80::8:20ff:fe1a:3ab0/10
db1/_a static ok fe80::8:20ff:fe69:a4c1/10

In the global zone it now looks like this:

root@ahutest:~# dladm show-link
LINK CLASS MTU STATE OVER
net1 phys 1500 up --
net0 phys 1500 up --
app/app0 vnic 1500 up net0
app/app1 vnic 1500 up net1
db/db0 vnic 1500 up net0
db/db1 vnic 1500 up net1

Generally this third method should work this way, but sadly in my tryout I was just able to ping the other zone seconds after I had set it up and then not again.

I opened a case at Oracle Support for this and hope that I will learn what I did wrong here or if this is maybe a bug. I will update this blog entry if I know more about the matter.

Maybe there are still other and better ways to do this. This was just my first tryout with Solaris 11 networking. If you have any suggestions or hints how to do it better, do not hesitate to leave me a comment.

Andreas Hubert

Andreas Hubert

Andreas Hubert ist stellvertretender Leiter der IT Services bei censhare. Außerdem kümmert er sich als Ausbildungsleiter um den fachlichen Nachwuchs. Als gelernter Fachinformatiker sind seine Schwerpunkte Netzwerk Administration und Linux/Solaris. Bei Twitter kann man mit ihm zu den Themen Apple, iPhone, Oracle Database, Sun Solaris, Sicherheitslücken, ´nem Caffè Latte vor seinem Mac und natürlich censhare zwitschern.

Neuer Kommentar:

Wir behalten uns vor, Kommentare, die gegen unsere Nutzungsbedingungen oder Kommentarrichtlinien verstoβen, teilweise oder vollständig zu löschen.
von Tedy 06.02.2013 19:22

what is the case number

HI Andreas,
may I know the case number you created on Oracle Support? I want to know what Oracle propose for the fix.

Antwort

Neuer Kommentar:

Wir behalten uns vor, Kommentare, die gegen unsere Nutzungsbedingungen oder Kommentarrichtlinien verstoβen, teilweise oder vollständig zu löschen.
von Tedy 06.02.2013 19:22

what is the case number

HI Andreas,
may I know the case number you created on Oracle Support? I want to know what Oracle propose for the fix.

Antwort

Neuer Kommentar:

Wir behalten uns vor, Kommentare, die gegen unsere Nutzungsbedingungen oder Kommentarrichtlinien verstoβen, teilweise oder vollständig zu löschen.
von Lucien Hercaud 24.08.2012 11:02

what abour device aggregattion

Hello

Is it possible to do the same but using AGGREGATION (a.k.a. dladm create-aggr) instead of IPMP ?

I must agree I tried pretty hard with no success.

TIA

Lucien

Antwort

Neuer Kommentar:

Wir behalten uns vor, Kommentare, die gegen unsere Nutzungsbedingungen oder Kommentarrichtlinien verstoβen, teilweise oder vollständig zu löschen.
von Jan Holzhüter 07.03.2012 17:01

vnic don't work in virtual maschines

vnics don't work in vm. (mac spoofing problem)
What might work is to add an virtual switch between the zones. This way they could talk to each other (I think have not done it) but still not to the rest of the world :)

Antwort

Neuer Kommentar:

Wir behalten uns vor, Kommentare, die gegen unsere Nutzungsbedingungen oder Kommentarrichtlinien verstoβen, teilweise oder vollständig zu löschen.
Antwort von Klaus 05.06.2012 21:52

RE: vnic don't work in virtual maschines

Have you checked /var/adm/messages for errors?

If the gateway is not reachable OR is blocked from ICMP traffic then the interfaces will fail and the host is not reachable.

Antwort von Andreas Hubert 08.03.2012 11:44

RE: vnic don't work in virtual maschines

Thanks for the Feedback!
Okay, thats maybe a hint. But if I change in the anet interface
link-protection: ip-nospoof
and set the allowed-address
it doesn't work either.

And with etherstubs.... I guess the configuration would then look totaly different. If I think through it, I would create an etherstub for the zones then I only need one vnic each, because they are vnics on a vswitch, so no need for a kind of hardware failover in this area. With the etherstub I have them connected, but to get a link to the outside world, I would need a router zone, which then has the IPMP configuration as "net"-configuration and a "anet" to connect to the etherstub ... but this makes things just more complicated then I want them to be and there would be no big difference to method 1 or 2 apart from that it's just more complex. Or am I wrong here?

Antwort von Jan Holzhüter 08.03.2012 14:52

RE: vnic don't work in virtual maschines

I think is does not matter what you set link-protection to.
This is not a solaris problem that can be fixed it's a virtuallizer problem in general. A vnic on a vnic which you basicly do here since the nic in the virtual maschine is a vic too just does not work.
If you try this on a sparc ldom this fails with an error and tells you that you can't do it. You need to add more nics via the virtual host and then assign those nics exclusive to the zone. Your methods do work on real hardware though

von Mirko von Schlachta 07.03.2012 12:24

Hard stuff!

Hi Ahu,
that is really hard stuff!
The only unix command in this article that I understand is:
zonecfg:db> exit
Our daily work is so different! ;-)
I am so happy that I don´t have to care about VLAN in Solaris zones...
cheers

Mirko

Antwort

Neuer Kommentar:

Wir behalten uns vor, Kommentare, die gegen unsere Nutzungsbedingungen oder Kommentarrichtlinien verstoβen, teilweise oder vollständig zu löschen.

Unser Newsletter informiert Sie über aktuelle censhare Neuigkeiten.

Newsletter:
GlobalSchweiz
Vorname:
Nachname:
Firma:
E-Mail:

Ihre Daten werden von uns nicht an Dritte weitergegeben.

Apps mit censhare

censhare Apps

censhare unterstützt gleich zwei Arten von Apps für iOS und Android: Content auf Basis von Layout-Dokumenten oder von Webseiten. Damit steht für jede Anwendung die richtige Technologie bereit.

Download censhare News App

Get censhare!

etatumcenblogcenshare News appcenshare @ LinkedIncenshare @ XINGcenshare on google+censhare @ YouTubecenshare @ SlideShare