VPN: Cisco ASA with AnyConnect Essentials

Certificate provider: SecureAuth

I skipping the setup of these two systems, assuming you currently have a working two-factor AnyConnect VPN.  SecureAuth and Cisco Anyconnect both provide a web based (ActiveX, FF Plugin, or Java) installation process for your clients.  Depending on your setup, you will first install a certificate, and then install the VPN client.  Once completed, launching the VPN client will locate your certificate, and prompt you for your password to complete authentication.

However, on Ubuntu 64-bit you will receive a certificate error.  The truth is, the certificate is fine, we just need to install some additional packages.  The are located in the 32bit version of Firefox though.  It is a few technical hoops to jump through, but you are already using Linux as your desktop, so it’s easy:

- Go to firefox.com.
- Download Firefox.  The version you want should be a tar.bz file
- Extract it to /usr/local/firefox

Time for the CLI, of course:

sudo apt-get install ia32-libs lib32nss-mdns
cd /usr/local/firefox

sudo ln -s libnss3.so /opt/cisco/vpn/lib/nss3.so
sudo ln -s libplc4.so /opt/cisco/vpn/lib/libplc4.so
sudo ln -s libnspr4.so /opt/cisco/vpn/lib/libnspr4.so
sudo ln -s libsmime3.so /opt/cisco/vpn/lib/libsmime3.so
sudo ln -s libsoftokn3.so /opt/cisco/vpn/lib/libsoftokn3.so
sudo ln -s libnssdbm3.so /opt/cisco/vpn/lib/libnssdbm3.so
sudo ln -s libfreebl3.so /opt/cisco/vpn/lib/libfreebl3.so
sudo ln -s libnssutil3.so /opt/cisco/vpn/lib/libnssutil3.so
sudo ln -s libplds4.so /opt/cisco/vpn/lib/libplds4.so
sudo ln -s libsqlite3.so /opt/cisco/vpn/lib/libsqlite3.so

Go back to Firefox, and browse to your VPN’s URL.  The install should connect right away.  You will also now have a working client in the Gnome GUI now too.

My new Dell R710 has 4 on-board Broadcom NICs, and an add-on Intel Quad Port NIC.  After installing ESXi 4 Update 1 (customized ISO for Dell) I saw the Broadcoms, but not Intels.  The solution is a driver CD that is available from VMWare, but the installation process is a couple of hoops.

- Download the CLI (command line interface) and the drivers you are looking to install:

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4#drivers_tools

I download the VMWare vSphere CLI 4.0 U1   (I ran it from Windows 7 64bit).  It is under the ‘Drivers and Tools” tab,  ”Automation Tools and SDK” branch.  I also needed the “VMware ESX/ESXi 4.0 Driver CD for Intel 82575 and 82576 Gigabit Ethernet Controller”, which was also under the “Drivers and Tools” tab, within the “Driver CDs” branch.

2 – Install the CLI tool (on Windows)

3 – You will need to burn the driver ISO to CD and put it in the drive of the same machine where the CLI tool is installed (or mount the ISO on that machine).

4 – In Windows, Start – All Programs – VMWare – VMWare vSphere CLI.  This will open a familiar looking Windows command prompt window.  You should default into the installation directory of the CLI tool

5 – type “cd bin” then press Enter in that command prompt window

C:\Program Files (x86)\VMware\VMware vSphere CLI>cd bin

6 – type “d:” then press Enter (assuming d: is the drive where your driver CD in mounted)

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>d:

Occasionally the VSS service will cause a problem.  I have a new VSS integrated backup system (R1Soft – best backup solution I have ever used) that works flawlessly on almost 40 systems, but there are 3 servers in particular that occasionally have problems.  These systems are being deprecated as their are multiple issues on these platforms (and they’ve been in service for 7 years).  There are several articles floating around online on how to address issues with VSS.  I have distilled the steps I perform into a quick process that works in getting the backups going again.

One common issue appears to be with removing and then installing competing VSS based backup agents.  We recently removed Backup Exec to switch to R1Soft, and some of the behaviors we experienced seemed to indicate a less than graceful uninstall process in Backup Exec caused issues with the VSS service.  From this our procedure going forward is to remove Backup Exec and reboot, then run through this process, and then install our new VSS backup agent, R1Soft.  I’d recommend exploring this process as an option nytime you are switching between VSS backup systems.

On to the steps… on the server where the VSS service is throwing errors to our backup agent, we perform the following (BTW – we have only seen this on 2003 Operating System).

- Click Start, click Run, type Regedit, and then click OK.
- Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EventSystem\{26c409cc-ae86-11d1-b616-00805fc79216}\Subscriptions
- On the Edit menu, click Delete, and then click Yes to confirm that you want to delete the subkey (you are deleting the “Subscriptions” subkey)
- Close Registry Editor.

- Click Start, click Run, type cmd, and then click OK. (this opens a command prompt).  Type each of the following commands (easiest to cut and paste, and put into a batch file):

net stop “System Event Notification”
net stop “COM+ System Application”
net stop “Background Intelligent Transfer Service”
net stop “COM+ Event System”
net start “COM+ Event System”
net start “Background Intelligent Transfer Service”
net start “COM+ System Application”
net start “System Event Notification”
net stop “Microsoft Software Shadow Copy Provider”
net start “Microsoft Software Shadow Copy Provider”
net stop “Volume Shadow Copy”

- the Volume Shadow Copy service will likely NOT stop.  The command will return with “The Volume Shadow Copy service could not be stopped”.  You can verify this by opening the Services Control Panel applet.  It will be listed as “Stopping”.

- Press Ctrl-Alt-Del, and open Task Manager.  Select the “Processes” tab, check the “Show processes from all users” box (lower left corner).  Locate the vssvc.exe process.  Right click it, and select “End Process Tree”.

- You will now need to restart the Volume Shadow Copy service.  Go back to the command prompt window are type:

net start “Volume Shadow Copy”
You are done.

This has fixed our VSS issues 100% of the time.  Test it first for yourself to make sure it gives you the result you need.

On the Cisco switch, we need to configure a trunk, and then add ports to that trunk.  In my example, there are no existing Port-channels, so the trunk will be Port-channel1.

interface Port-channel1
description *** Proxmox Host PRX01 bitbud.2009.10.10 ***
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
duplex full
spanning-tree portfast trunk

I will be using ports 1 and 2 on Gigabit slot 2 for the network ports:

interface GigabitEthernet2/1
description *** Proxmox Host PRX01 bitbud.2009.10.10 ***
switchport trunk encapsulation dot1q
switchport mode trunk
speed 1000
duplex full
spanning-tree portfast trunk
channel-group 1 mode desirable

interface GigabitEthernet2/2
description *** Proxmox Host PRX01 bitbud.2009.10.10 ***
switchport trunk encapsulation dot1q
switchport mode trunk
speed 1000
duplex full
spanning-tree portfast trunk
channel-group 1 mode desirable

That takes care of the port configuration on the switch.

Now, in Proxmox, I need to edit the trunk information.  In Linux, this is called a network bond.  We will also configure vLANs and bridges as well.  In Proxmox 1.4, you can have an unlimited number of bridges (bridges are what your virtual NICs connect to), so I like to use a bridge per vLAN (makes sense).  In my example, my primary vLAN that most systems use is vlan10, and I’ll also configure vlan20, and vlan30.  I’ve included some extra information as an example as well.

All of this configuration is stored in /etc/network/interfaces.  This is my complete interfaces file:

# standard net config DW 2009.11.12
#
# update ethx cards available
# update bond0 config (which ethx cards to use)
# update each bond0.x per vlan
#   and its associated auto vmbrx per vlan
#   exception ___ that vmbr0 should use the bond.x vlan
#   that you want to use as the admin interface

# network interface settings
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto bond0
iface bond0 inet manual
slaves eth0 eth1
bond_miimon 100
#    bond_mode active-backup
bond_mode 4

auto bond0.10
iface bond0.4 inet manual
vlan-raw-device bond0

auto bond0.20
iface bond0.5 inet manual
vlan-raw-device bond0

auto bond0.30
iface bond0.11 inet manual
vlan-raw-device bond0

auto vmbr0
iface vmbr0 inet static
address 10.5.56.71
netmask 255.255.254.0
gateway 10.5.57.4
bridge_ports bond0.10
bridge_stp off
bridge_fd 0

auto vmbr20
iface vmbr20 inet manual
bridge_ports bond0.20
bridge_stp off
bridge_fd 0

auto vmbr30
iface vmbr30 inet static
address 10.101.7.1
netmask 255.255.0.0
bridge_ports bond0.30
bridge_stp off
bridge_fd 0

Let me point out a few things in that example:

• If you have additional interfaces, such as eth2, eth3, etc that you want to add to the bond, simply copy the example with the new interface id.
  • Add:
    auto eth2
    iface eth2 inet manual
  • Change:
    slaves eth0 eth1
    To:
    slaves eth0 eth1 eth2
  • You’ll also need to configure your switch port as well
• I have a separate vLAN and IP scheme for my SAN.  Using Proxmox’s storage management layer, I connect to these from the Hypervisor.  You’ll see this on the bridge configuration vmbr30.  Note the additional IP and netmask, but no gateway.
• Proxmox likes to see a vmbr0 as the ‘default’ bridge.  Since vLAN10 is my ‘main’ vLAN, I set this to the bridge vmbr0, rather than vmbr10.  All other bridges are named after the vLAN they represent (vLAN20 is on bridge vmbr20).

On my production systems, I have over 20 vLANs configured, and they work great.  Once you have setup the vLANs in the interfaces file, selecting them from the web interface is easy.

Questions?

One of the simpler application for this purpose is Munin .  Munin is easy to install and get going on Proxmox, and in 5 minutes you can have detailed, historical reports for your Virtual Host.  There are other solutions available, but this is very lightweight, and keeps Proxmox as an “all-in-one” solution.  Hopefully, we may see the maintainers include Munin in the future as a part of the base install.  Until then, here’s how to do it yourself:

Login to the Proxmox console, either locally or via SSH, then run this command:

apt-get -y install munin munin-node

Right away, your system performance is being tracked, but it isn’t yet available via the web, and it doesn’t have all the information we want to see.  So lets customize it, by editing this file:

nano /etc/apache2/sites-enabled/pve.conf

search for this in the file editor:

Alias /images/  /usr/share/pve-manager/images/
Alias /css/ /usr/share/pve-manager/css/
Alias /javascript/ /usr/share/javascript/
Alias /vncterm/ /usr/share/vncterm/

And add this line below the last “Alias” line:

Alias /munin /var/www/munin

Now we’ll edit another file:

nano /etc/munin/munin.conf

search for this in the file editor:

[localhost.localdomain]
address 127.0.0.1
use_node_name yes

and change it to:

[yourserver.yourdomain.com]
address 127.0.0.1
use_node_name yes

Substitute yourserver.yourdomain.com with the fully qualified hostname of your server.

Now we will need to add any additional network interfaces that need to be monitored.  On my servers I have a single trunk into the switch using all of my available physical network adapters (ethX).  The trunk interface is a bond called bond0.  I then have the bond0 broken down into  several vLANs, and those interfaces are titled by vLAN ID.  For example bond0.56, bond0.28 for vLANs 56 and 28.  Each virtual guest connects to a bridge interface, which is titled by the bond interface it uses, one for each vLAN as well: vmbr56, vmbr28.  When installing munin, the ethX interfaces are added automatically, so I will need to add these additional interfaces with these commands (since the bond.vLAN and vmbrvLAN interfaces are 1-to-1, I’ll just add the bridge interface).  On some systems, I’ll have more than 30 interfaces:

(just copy and past, changing the interface name at the end to whatever you are using)
ln -s /usr/share/munin/plugins/if_ /etc/munin/plugins/if_bond0
ln -s /usr/share/munin/plugins/if_ /etc/munin/plugins/if_vmbr0
ln -s /usr/share/munin/plugins/if_ /etc/munin/plugins/if_vmbr56
ln -s /usr/share/munin/plugins/if_ /etc/munin/plugins/if_vmbr28

To make it pretty, add a link to your web interface:

(will add link soon)

Now restart your services:

/etc/init.d/apache2 reload
/etc/init.d/munin-node stop
/etc/init.d/munin-node start

Now you can browse to your Proxmox server at https://servername/munin and view detailed information on Disk, Network, Process, and other system information.  This gives good, basic  historical information on your systems.  It will take some time for the graphs to populate, but I am sure you will find this a useful troubleshooting tool.

Later, I’ll show how you can aggregate all of this information onto one page, which is especially useful if you are using Proxmox with Clustered nodes, and have many systems to manage.

Some samples:

bond0.56 – shows the in/out traffic on this particular vLAN

CPU Usage – shows that the system isn’t very busy (user), has a lot of capacity to spare (idle), and lets me know the disks are fast enough for the current work load (io wait).

CPU Usage

Virtualization is a key management tool for IT, and especially useful for Church’s and other non-profit organizations.  It can can simply day-to-day administration, enable easy business continuity planning, but it allows you to get more value for your limited financial resources.  Unfortunately, for many years the entry point has been at too high a cost.  Even for larger organizations that can justify the expense, it is money that could usually be better spent elsewhere.

Virtualization in the sense we understand it today has been made practical and useful, not by the software companies, but by the hardware manufacturers, specifically Intel and AMD.  Looking back, when Connectix released the first desktop virtualization product, Virtual PC, in 1997 the available hardware at the time made for an interesting experiment, but nothing more.  Even when 2 years later VMWare release its first product, VMWare Workstation, the state of computing power at the time was nothing like today.  In 1999, if you had a powerful x86 system it was a PIII or K7, running at 500mhz, with 100Mhz memory bus.  Not exactly an ideal machine for virtualization.  Fast forward to today where multi-core, hyper-threaded processors run at 3Ghz w/ on-die memory controllers and more memory and storage than you could imagine a few years ago.  Virtualization is now not only feasible, but warranted, as most bare-metal systems are being utilized at less than 10% of their full capacity.

Over the last few years, I have been a strong advocate of Linux w/ KVM, seeing the potential I knew it would eventually realize.  Over that time I have heard many valid concerns from those in the IT field about virtualization, and using Linux as a solution.  I think the events over the past 50 days have addressed those concerns, and make it a viable solution for any IT organization, but especially useful in the Church, and other non-profits.

Sept 2 – Red Hat releases RHEL 5.4, the first Red Hat Linux distribution with full support for the KVM Hypervisor. It also includes SAN Storage capabilities, via the iSCSI Target Framework, and an easy to use Virtual Systems management console, Virt-Manager.

September 24 – Red Hat releases Virtio drivers for Windows, providing fast and stable block and network drivers for Windows.

October 7 – Microsoft Certifies Red Hat’s KVM Hypervisor as a supported hardware platform.  This is important to IT shops that require a fully supported systems stack.

October 19 – Proxmox releases PVE 1.4.  I’ve just started playing with the new release, and it’s new storage model is nothing short of sweet.  This is a full featured, easy to install, easy to manage solution.  Included in this release is a new, flexible storage model allowing for virtual and storage Live Migration, and clustering without requiring a SAN.  Expect more information from me on this soon, as I dive into upgrading my systems (most of my Virtual Host systems are running Proxmox 1.3)

October 21 – CEntOS 5.4 is released. CEntOS is the binary compatible community support version of Red Hat 5.4.  You basically get Red Hat (including updates) for free, but without support from Red Hat.

For anyone who works in the IT field, you really owe it to yourself to at least look at Proxmox now. It is free, stable, and takes only a few minutes to install.  Even if you are already in deep with another virtualization vendor, you never know when you’ll need one more virtual host that you didn’t budget for, and you may be surprised by the powerful features packed into that open source solution.

If you want a sneak peak at what the new RH Enterprise Hypervisor management system might look like, take a look over at ovirt.org.

To get a taste of what is possible with the Linux/KVM based hypervisor, there is an easy to install distribution from Proxmox (which works exceedingly well). I am currently using Proxmox in many production workloads and am excited about what RedHat is going to bring to the table.

Red Hat’s move to KVM will bring Linux based virtualization into the mainstream, and further the march towards a commodity hypervisor.  The management tools will be the difference maker, and Red Hat’s new platform will include all the capabilities you would expect from an Enterprise grade virtualization infrastructure, including, among other things,  live migration, snapshots, system scheduler, power and image managers, comprehensive monitoring, reporting.  But the difference here, it’s all open source.

An article about it and Red Hat’s official press release.

Last week it was annonced that Intel would be aquiring Wind River for nearly $900 million.  In case you didn’t know Wind River is an embedded application/ OS provider for a long list of companies, including Sony, BMW, and NASA.  They provide BSD and Linux based distributions/ applications for various embedded systems. What I find paticularly interesting is how it may affect companies in the IT field, such as is Dell/ Equalogic.  The Equalogic SANs are a mid-market leader in iSCSI storage, and use WindRiver Technology (w/ NetBSD) in their SANs (along with LSI controllers, which leverage Wind River).  As Dell is one of the largest buyers of Intel chips, I am sure that this will only help to advance the technologies Equallogic is able to integrate with it’s SANs down the road.  With the full backing of Intel behind Wind River now, it will be interesting to see what new technologies it can help bring to the table.  Intel is looking to improve it’s access to the embedded systems space,and Wind River provides a wide customer base for that purpose.

Intel has been a good steward of creating new technologies, and has a good history with supporting Linux and Open technologies, so I am sure this will be a win for all involved.

Installation and use is fairly straight forward.  I would recommend using a RedHat based distribution at this time.  That could be Fedora, RedHat, CentOS, or one of the other derivatives.  The tgt that is available on Debian based distributions is old (ancient) and not stable enough (IMO).  I would stick with the IET if you are required to use a Debian based system (like Ubuntu).  However, if you are serious about virtualizing your storage, use the tgt on a RH system.

I am using CentOS 5.3 here.  Fedora 10 will work just as well.  After you have installed your OS, installation is straightforward:

yum install scsi-target-utils
chkconfig tgtd on
service tgtd start

That was easy.  Installs the app, enables service on restarts, and starts the service now.  All configuration is handled from the ‘tgtadm’ command.  An excellent resource for information is to simply type ‘man tgtadm‘ from the CLI.  I’ll highlight some basics here:
First note that all commands for our purposes will start with ‘tgtadm –lld iscsi’.  –lld iscsi specifies that we are working with an iscsi target (recall that this framework is for managing many types of SCSI devices).

Lets assume you have a partition on your SAN that you would like to share as an iSCSI target.  For example, I have created an LV (Logical Volume) on my server named /dev/vg1san06/lvSERVER02dbs.  By my naming convention, you can see my SAN is named is SAN06, and the volume is lvSERVER02dbs – in this case I am setting up a target for my SQL server to store it’s databases.

- tgtadm –lld iscsi –op show –mode target
This command will display any existing targets
If you have already created targets, note the next largest target number

To create a new Target, select the next available Target #:

- tgtadm –lld iscsi –op new –mode target –tid 4 -T iqn.2009-03.com.bitbud:san06.tgt.server02
I like to name the target by the systems(s) that may use it, in this case the server is ts02

- tgtadm –lld iscsi –op bind –mode target –tid 4 -I 10.5.56.52
After creating the target (above), you may now want to restrict it to just connections from one system.  In this case IP 10.5.56.52

That is nice and all, but the target doesn’t point to anything on disk yet.  Let’s create a LU(logical unit) for the LV you created, and assign a LUN to it, starting with 1:
- tgtadm –lld iscsi –op new –mode logicalunit –tid 4 –lun 1 -b /dev/vg1san06/lvSERVER02dbs

add additional LU(s) to this target as needed, incrementing the LUN by 1 each time.  My SQL server needs a target for it’s logs, so I’ll add that now too:
tgtadm –lld iscsi –op new –mode logicalunit –tid 4 –lun 2 -b /dev/vg1san06/lvSERVER02logs

(note in the example above, I’ve would have already created the Logical Volume lvSERVER02logs)

On reboot, the commands are lost, so lets make it permanent by adding it to the config file:
nano /etc/rc.local

In the editor, simply paste each tgtadm command you typed above.  Put the commands in the same logical order that we did here, create the target, assign properties, then create each LU.

Those are the basics with the tgt framework.  There are many other features to consider, such as clustering, multipathing, and volume management.  Most of those functions occur in other components of your Free and Open Source SAN software stack, which I’ll cover in upcoming posts.

NOTE:
If you are considering building a standalone iSCSI SAN using Linux, I would highly recommend using either the 64-bit CentOS 5.3, or opt for RedHat if support is your requirement.  If/ when Ubuntu pulls an updated tgt into it’s kernel and moves it to it’s main repository (ie – supported) then I will recommend it again.  The IET in Ubuntu works well, and I have it running in production on several large SANs.  However, any future SAN installations for me will be built as detailed here.

Some extra tips:
- Don’t forget to open port 3260 on the firewall, to allow traffic for iSCSI connections:

iptables -I INPUT -p tcp –dport 3260 -j ACCEPT
iptables -I INPUT -p udp –dport 3260 -j ACCEPT
iptables -I INPUT -p tcp –dport 3260 -j ACCEPT

- Don’t use a capital letter for the iqn