When you think of your SAN (or any IT system, for that matter) there are two areas to address: the hardware and the software.  There are many ways to go about putting together a SAN for your environment.  Some possible options might be:

• Buy a SAN from a SAN provider: Equalogic/ Dell, Compellent, or Xiotech.
  This is your hardware and software in one package.  This will provide you with a quick, high quality solution with an easy to use interface.  You won’t even need an in-depth understanding of the technologies involved, as it is all nicely packaged for you.  The biggest barrier is usually the cost.  If that’s not an issue for you, if you are comfortable with someone else being responsible for your critical software stack, and you’d rather not get into the nitty-gritty of SANs (i.e., you want the super easy way to do it :-)), then I can point you to a brother who is a reseller.
  
• Buy a commercial application for PC/ Server hardware to turn it into a SAN.
  A good example would be the application San Melody, which I have been told works quite well.  Of course, you need the appropriate hardware, and the license is a little expensive.  Like option 1, it is nicely packaged and you don’t have to get your hands dirty with the technical details.
• Use a completely free and open source system like OpenFiler.
  This system, like San Melody, will provide you will a packaged software solution that you install on your own hardware.  It has many easy to use features.  It probably requires just a little bit more technical understanding than options 1 and 2, but it is still easy to use and best of all, its free.
• Use Linux
  … or more specifically, use Linux with a few specific packages: LVM2, iSCSI Enterprise Target, Ethernet Bonding, and (if you want to cluster, like I do) NFS.  This is what I will focus on the rest of this series of SAN articles And yes, this is what OpenFiler offers in a completely packaged system.  I’ll go into detail soon why I am not using it later.

Don’t assume that must give up some feature or functionality by using Linux.  On the contrary, for a much reduced cost you can have the equivalent system (I would argue a better system).  For me, Linux has become my Swiss Army Knife of SANs.  The same stable system can be used on low-end or high-end hardware, which keeps administrative costs down (keep it simple).  On the low-end: in Part 1 I mentioned that I built a full featured system (minus stellar performance and some hardware redundancy) for less than $500 with spare PCs and a few new hard drives. On the high-end: if you do have some money to spend on hardware, and need the best in performance and redundancy, you can purchase the same redundant hardware as a commercial SAN, same hardware warranty, for only 10%-25% of the cost!
I didn’t arrive at option 4 lightly, but after several months of investigation, testing, and consideration of these options.  There were several aspects to consider:

• Cost. The cost factor can’t be ignored.  It wasn’t the biggest reason for me, but it will be for most.  It is important to be a good steward your resources, regardless of how limited or unlimited they may be, the money spent has to be considered
• Ease of use. I’m sure some will disagree, but I don’t think having a GUI automatically makes something easier to use.  In the case of a critical component like a SAN, a GUI may simply make it easier to break.  If you think typing an e-mail is easy, and you can handle typing a few sentences from instructions, then using Linux from the command line for managing a SAN is easy too.  In fact, you could argue that typing in a command is much more precise than clicking around a GUI, and therefore could be easier to use, especially considering the importance of getting it right.
• Rock Solid IT. A SAN is about as critical a component as you can have.  All your data is congregated in one place.  It is the disk space for all your servers.  It has to run all the time with no down time - it has to work.  Period.
• Good Stewardship. Someone else will eventually need to come behind you and work on this.  You WANT others to be able to help.  I toyed with this thought for a while, as as a friend from the RT pointed out to me to make sure it was covered.  See ‘Ease of Use’ above, I think so long as you document what you have done, and how to do it, you cover the ability for someone else to pick it up.  The interface doesn’t really matter.
• Ownership. I’m an engineer.  I realize everyone in IT may not be an engineer, but for me there is a huge value add when not using a proprietary software stack.  The ability to fully own what you are using, and not be open to a company pulling the rug out from under you when you least expect it (as can happen with proprietary/ closed source offerings).  Some see the Linux/ Open Source aspect as a disadvantage, but I disagree.  True, you do need some technical knowledge, but you hopefully do, working in IT.  For me, I am the last line of support, not some vendor.  If I am responsible for a system, I should understand in 100% and be able to address any issues with it.  Others would disagree, and prefer to outsource that responsibility, and I understand how they arrive at that conclusion.  On that note, I will tell you that this IS easy to do, and if you follow good production IT principles (like testing and stabilizing production systems before implementation) you will have a system that will be Rock Solid reliable.

Here’s whats coming up in the SAN For All series:

• Specs for building a high-end, fully redundant 16 TB SAN for less than $9000
• Implementing a SAN for cheap
• Installing an iSCSI SAN - Step by step process for using Ubuntu LTS Linux as a Rock Solid SAN
  • Working with physical disks and LVM2
  • Configuring iSCSI Targets
  • Managing LVs - snapshots, resizing
• Configuring Network bonding - How to get 10Gb speed with redundancy for less
• Virtualization methodologies
• NFS for clustering, just like an HPC
• RAID configuration suggestionsUpdate: I have 3 systems running Ubuntu with the iSCSI ET now, and the performance and ease of use is great.  More to follow soon.

Note: There are other options for getting a SAN for your use.  For example, there are iSCSI Target applications that you can add to Windows to allow it to function as an iSCSI server.  My focus here is on serious, production use of SAN technology.  It has to be rock solid stable and running for years with minimal maintenance.  Moving on…

I found a useful script that handled the importing of all my Joomla ‘articles’.  Sadly, none of the comments made it, but I may do that manually later.

There is still more work to do, most importantly handling redirects from the old Joomla links to the new Wordpress permalinks.  Also need to import my graphics, find a new template, and get my modules configured.

I’ll update this post with some more details on the script I used for importing later.

Enjoy!

The new version of KVM (KVM-79) was just released, and it includes a feature that is unique to KVM (compared to all other x86 virtualization platforms - in other words, VMWare and Microsoft can’t do this, but I’m sure they’ll add it soon),

PCI device assignment

With this new release, you can now pass an entire PCI device directly to a Guest virtual machine.  No emulation or para-virtualization required.  If you understand what that means, then you understand just how cool it is.  This is HUGE, both in the virtualization world, and for PCs in general.  In time, the paradigm is going to shift to having your complete PC virtualized - this is the first step.

What in the world would I use it for, you ask?  Examples?  You could have a PC with an additional video card that is directly passed to the Guest virtual machine, so that it can have its own dedicated video.  A network card that can now operate at full native speed, as it is completely controller by the Guest system.  Or a hard disk controller card managed by your Guest computer.  This opens the door to now have a completely virtual desktop experience, and native speed servers.  It will be possible, over time, to have a completely virtual desktop without the current issues with heavy graphics/ 3D systems.

The code for this has been in the cooker for several months.  This is the first inclusion in a “release” of KVM, so expect continued improvements.  I believe at this stage it is supported only with Intel’s Virtualization extensions (which means the AMD PC I have wont do it, but the Intel one will).  I’ll double check on that and update that here with the requirements.

I am looking forward to testing this out with my MythTV (Tivo type software under Linux) system in the next few weeks.

Its true! Codeweavers is offering Crossover for Linux and Mac for free (including support) - but for today only.  Check it out at Click Here to Register.  Crossover is an application that allows certain Windows applications (including games) to run natively on Linux and Mac.  I’ve already got my copy.

Now, if you know me, you know I am partial to FOSS (Free and Open Source Software - not necessarily free as in price), and Microsoft is not FOSS.  I tell you that as a disclaimer, so you know I am not a Microsoft fan trying to sell you on HyperV.  In IT, it’s a Microsoft world, so I have to study it, understand it, and use it.  I have been working with Virtualization in production environments for about 5 years now, most recently testing the viability of ESX and HyperV for our production use.

To the uninitiated, it is easy to pick HyperV.  I have already seen engineers, who would not otherwise consider virtualization, move to it ever since HyperV was made available as a part of the 2008 OS.  It makes sense.  With just a few simple clicks, suddenly you can run multiple OSes on your PC or server.  Dead simple.  It is also easy to use, and has some useful features already included.  The management console, snapshots, and import/export tools are of great value, and I would argue that it’s Quick Migration (as opposed to VMWare’s VMMotion or KVM’s Live Migration) is practical and meets almost all requirements. 

To the better informed, a closer look is needed, and that is what I have been doing for the past few weeks.  I can tell you straight away that HyperV suspiciously looks like Xen… a lot (With Windows 2008 as Dom0).  An interesting footnote: Microsoft’s longtime buddy Citrix purchased Xen Source last year.  Theoretically, that purchase would allow MS to obtain a private license with Citrix to use Xen code outside the open source license under which it was developed, and under a different license.  I can also tell you that side by side HyperV doesn’t match up as well with ESXi.  Even with similarly configured guests with identical host hardware, the tests I performed showed that ESXi was plain faster (on a side note, KVM was faster than both).  And the ESXi management features are impressive with the detail that you can manage your Guest VMs.  ESXi does binary translation, whereas HyperV requires virtualization extensions.  This means that ESXi will run on older hardware as well.

So if ESXi is technically better, than whats the deal with HyperV?  For evidence, see the above referenced article.  Microsoft will continue to work on HyperV, and will very quickly include all the features that the Enterprise requires (read that as all the features that VMWare has).  On top of that, it is included in their OS (akin to how Internet Explorer took over), and costs almost nothing to install it.  To the average IT person, this is a no-brain-er, and is a VERY easy to path to virtualization.  I can tell you that when I first started Virtualization 5 years ago, if HyperV existed and was as easy to install (and as functional) as it is today, we would be all HyperV by now.  That is saying a lot, coming from me.  And the performance difference, while noticeable, wasn’t so great as to exclude HyperV from consideration.

I hope VMWare can come up with a good way to compete with this juggernaut.  I think in time KVM will silently become the standard, especially in large production data centers, much like Apache and Linux are today.  But for other corporate uses, VMWare is going to have to complete heavily with HyperV.

For now, for us, KVM is our standard VM platform, and we will start using ESXi to fill in some holes (especially with older equipment).  By comparison, HyperV just doesn’t meet our requirements… yet.  But I have seen it already working well for others in production environments, and can only guess at what the future holds.

SAN for ALL!

The paradigm has shifted (some time ago), and centralized storage is now a requirement for any organization to most efficiently run their infrastructure.  The ability to pool your storage resources, especially as data storage requirements explode, is invaluable.  No longer being tied to how many hard drives a single server can hold, but instead having expandable storage easily accessible via a network.  Today, iSCSI is the standard bearer (sorry FC, AoE), and many organizations see the need for implementing it.  However, the most common roadblock to implementing a SAN…

There are other impediments, but cost is easily the biggest.  It almost hurts to read about such great technology, to work with it in the business world every day, but not be able to afford it for your church.  The features and the capabilities that are gained from a well implemented SAN add so much value to IT, they cannot be ignored.  When IT works more efficiently, that helps them fulfill their role to support the ministries of the Church, which in the end is all that matters.

The cost factor is especially true for a small church (like mine), where there is no IT budget, where the infrastructure is built almost exclusively on donated, older computers.  In those situations, the necessity to be good stewards of limited resources is highlighted.  For a church of any size good stewardship is important, but for a church with no financial resources to contribute to the cause, it seems as there is no opportunity.  But that isn’t the case…

This is for those church’s who see the need for the technology, and have the time, but almost no resources to devote to the effort.  This is also for those that see the need, have the time, and the resources, but want to save some money, and gain a greater level of understanding, experience, and flexibility - the typical DIY-er.  You too can have a SAN, with every feature available to you in commercial SANs, for little to no cost.  The only thing you sacrifice is performance - at least for the SAN that costs nothing.  However performance can be improved to be on par with commercial equivalents for minimal amount of cost.

I’m going to break this down into several posts covering multiple options for configuration, features, costs, and performance.  Let me start with an example:

At my church, I recently implemented a new SAN for $476, using two donated PCs.  You could get by for less $$ if your PCs already have SATA hard disks of the capacity you want, but these were oooolllllddddd computers, so some minor upgrades were needed:

• 4 * 500GB SATA II drives @ $80/ each
• 2 * SATA II 4 port @ $18/ each
• 4 * Intel Gigabit PCI@ $40/ each

So - what did we get for all this?

• approximately 1/2 TB of disk space
• RAID 1 (mirror)
• FT/ HA (fault tolerant/ high availability)
• Dynamic resize
• Snapshot
• multi protocol (we use NFS and iSCSI, you could also use AOE, CIFS, etc)
• encryption
• cuts your grass
• makes your coffee
• …i might have forgotten something

How does it work?  The two PCs are each running 2 500GB drive in a RAID 1.  They are also connected to each other in a network RAID1.  Using HA, they are providing network services(NFS, iSCSI) in a primary/secondary fashion.  Dynamic resize and snapshots are provided via LVM, for the NFS shares I used XFS.

The FT/HA adds a bit more complexity, and may be overkill for many.  The rest of the features though are important.  An alternative to HA would be to simply use the 2nd PC as a backup system of the first, using simple replication.

These system were put in place to support the virtual servers that will be running all the back end systems.

In the next post I’ll go into the details of installing and configuring iSCSI on Ubuntu.

So it looks like Travis was right, and an Outlook plug-in is needed - which means this system isn’t 100% compatible, although the ActiveSync part looks interesting.  That means the only100% solution I know of is still Post Path.  It just isn’t open source.  If Cisco would find a way to GPL it, they would literally TAKE OVER the market.  Maybe one day they will.

Original Post:
So someone FINALLY released a 100% MAPI compatible (read - Exchange compatible) system under an open source license.  Zarafa. At first read, it appears to offer full ActiveSync support, and Outlook over HTTPS as well.  The limitation appears to be in the number of Outlook clients that can connect via MAPI, unless you purchase one of the higher-end versions.

I’ll post an update after further evaluation.

http://zarafa.com/?q=en/content/community-0

Other 100% MAPI compatible alternatives include PostPath, which was recently aquired by Cisco.  But they aren’t (yet) open source.

I am using mod_bandwidth.  There are several other Apache2 modules for managing bandwidth.  This one is a great quick fix.  In fact, I usually install this by default, but unfortunately forgot to on this website - until my bandwidth was maxed.  I highly recommend installing something along these lines to protect your system from this problem.

My system is Ubuntu Server and Apache2

This is simple to implement:
- Grab the latest version of the source code
- Extract and compile the Apache module
- change apache2.conf

To begin:

Find the latest source code download link at: http://ivn.cl/#bandwidth  Currently it is version 0.8

In a Terminal windows, run these commands in a temp folder:

sudo su
apt-get install apache2-threaded-dev
cd /tmp
wget http://ivn.cl/files/source/mod_bw-0.8.tgz
tar xvzf mod_bw-0.8.tgz
cd mod_bw

note: you may get an error after running this next command - you can ignore it

apxs2 -i -a -c mod_bw.c

nano /etc/apache2/apache2.conf

add the following to the top of the file

LoadModule bw_module /usr/lib/apache2/modules/mod_bw.so
BandWidthModule On
BandWidth all 1000000
MinBandWidth all 100000
ForceBandWidthModule On

This will limit the apache server to 1 Mb of bandwidth with each connections getting at least 100k

For about a year after September 11th, 2001 this quote was in my signature at the bottom of every  e-mail, regardless of who it went to - a friend, a customer, anyone.

As I sat in my car this morning talking to my wife, I lamented that so many have already forgotten, or pushed into the back of their minds, the events of this day.  The very real threat that exists in this world from those that hate America and what America has stood for - a country built on the principles of God’s Word (often called Judeo-Christian values).  A country that has been blessed, has prospered, and who’s citizens share that blessing with others in the world, more so than any other country.

Like everyone else, I remember exactly where I was 7 years ago as though it happened today, even if I can’t recall what I ate for lunch Monday.  I recall the odd feeling, simply trying to understand what was happening, “Why has ANOTHER plane flown into the building?”  The abrupt realization that we were like children, who’s innocence was suddenly broken, and our view of the world around us would never be the same.

I also recall where I was on another day, November 7th that same year.  I was working in Chicago.  That evening I was in my hotel room finishing some client notes from the day, with the CMA Awards show on the TV.  I stopped suddenly when Vince Gill announced that Alan Jackson, my favorite country singer, was coming on to perform a never heard before song.  He delivered the song in an appropriate way that only Mr. Jackson could, quite, humble, just him and his guitar (reminds me of my dad).  The words spoke to me, as they did to millions of others.

We have to watch out for complacency.  Certainly on this day, I think of the complacency that might make us think we are no longer vulnerable to attacks of this nature, that it won’t happen to ME.  But we must also be on constant guard for complacency in all areas of our lives, with ourselves, and how we direct the way we live… complacency in the attention we provide to our spouses, our children, our Faith.  They require our constant attention.  My dad once told me that life is a struggle, that you must battle every day against the wickedness of the flesh.  I know what he spoke of, and it comforted me to know that this man, my dad, the man that has earned my love and respect, above all other men that I have known in my life, confided in me that he too, we all, face this battle.  As my dad always does with such ease, he was speaking from God’s Word, where even one such as Paul struggled every day, and was not afraid to admit it.

I have struggled with complacency, and reflecting on this today I know that I must work each day going forward to lean on the power of the Holy Spirit within me to win this fight.  Sadly, I do not find myself complacent about worldly things, which are not important as they will pass away, but am too often complacent about the important things that live forever - my wife, my son, my church.  I must endeavor to stand on Gods Word, actively, everyday, to spend my full effort building Gods kingdom in my family, my church, and the world around me.  It is ironic that as I look to the past today to address my own complacency, God’s Word says that I must move forward by forgetting the past, and focusing on what I can do now, and in the future.  This is a blessing, being released from our past, our sins, which enables us to pursue the best that God has in store for us.  I don’t have to waste any more time with sadness over the past, but am freed to focus on the Hope of today, and the future.  But not to forget that it is work, and must be worked out everyday.

As I sat in my car this morning talking to my wife, I lamented that so many have already forgotten…
And as I spoke those words to my wife, I noticed the flag in front of our office building this morning - at half staff.   Not every one has forgotten…

Hope is not lost, when our Hope is in Christ Jesus.

References:
Romans 7:14-25
http://bibleresources.bible.com/passagesearchresults.php?passage1=Rom+7%3A14-25&version1=51
Philippians 3:12-15
http://bibleresources.bible.com/passagesearchresults.php?passage1=Philippians+3%3A12-15&version1=51
Ephesians 6:10-20
http://bibleresources.bible.com/passagesearchresults.php?passage1=Ephesians+6%3A10-20&version1=51
http://www.cafonline.org/Default.aspx?page=12183
http://www.christianitytoday.com/tc/2003/001/14.14.html

Red Hat has been involved with the ongoing efforts to develop and maintain KVM.  I think this will give additional focus, resources, and credibility to the KVM project.  Red Hat’s virtual management project, oVirt will probably pick up the pace as well.

So long as Red Hat keeps development of KVM as free and open-source, this should be a big plus.  I don’t know why they wouldn’t.

It’s been clear to me for over a year now that KVM was the future of virtualization, that VMWare and Xen would fade as the Hypervisor becomes a commodity.  Ubuntu moving to KVM as it’s primary platform for virtualization was the first validation, and now Red Hat.  KVM already has a broader set of capabilities than ESX and Xen.  It’ll blow the doors off of virtualization with the upcoming PCI passthrough capabilties (more news on that to follow).

I’ll keep you apprised of what I see going on in the developer community for KVM

Funny thing is, the PDA she was using was running Microsoft Pocket PC.  I even pointed it out to my buddy, to which the clerk just laughed and said ‘don’t tell anybody’.

I just thought that was ironic.  What do you think?

Read about it here: http://news.zdnet.com/2424-9595_22-218626.html

Aside from copious amounts of prior art, and obviousness, is this shame on Microsoft, shame on the Patent Office, or both?

What do you think?

(note - One of the alternatives would be to use VDE (Virtual Distributed Ethernet), but that is still evolving (KVM-72 just added it native!).  VDE is slow performance wise, but has some interesting uses, such as extending your virtual switch to remote locations)

Ubuntu 8.04.1 64bit Server updated to 2008.08.01 w/ KVM-72

Step 1 - add a fake interface to bind your virtual machine’s network card, by editing the interfaces config file.  I open it to edit in nano:

nano /etc/network/interfaces

and add the following 2 lines at the top:

auto pb1
iface pb1 inet manual

I named it pb1 for ‘private bridge’.  Call it whatever you want.  And restart your network stack (/etc/init.d/networking restart)

Step 2 - create the startup script for this interface:

nano /etc/qemu-pb1-ifup

with the following in your statup script:

#!/bin/sh
sudo /sbin/ifconfig $1 0.0.0.0 promisc up
sudo /usr/sbin/brctl addif pb1 $1
sleep2

Your done with the private bridge config.

Step 3 - To connect two (or more) VMs to that private interface, just start each VM with similar network settings… reference the same script file and use the same vLAN reference.  They are highlighted in my example KVM startup command below:

qemu-system-x86_64 \
-name SQL09 \
-vnc :32 \
-m 1024 \
-smp 1 \
-net nic,vlan=21,macaddr=aa:00:50:50:50:32,model=e1000 \
-net tap,vlan=21,ifname=tapSQL09,script=/etc/qemu-pb1-ifup \
-drive file=sql09root.qcow2,if=ide,index=0,media=disk,boot=on \
-std-vga \
-usb -usbdevice tablet \
-localtime \
-daemonize

Of course, use whatever vLAN you wish.

You are done!  I am going to update my KVM installation articles to include this, always adding a private interface, it is so easy. You could add multiple private interfaces if you wanted.

Uses:
This is great for setting up a completely isolated network of virtual guests.  However, if you need to connect out for Internet access, it is easy enough to multi-home one of your systems, and connect to a bridged interface on your Host.  For me, I like to run my favorite little firewall PFSense in a VM, multi-homed to both the private bridge and a normal bridge (shared with the Host).  For my laptop, I also include a third interface for the built in QEMU user networking (since I can’t bridge the wireless).

Works like a champ.

I didn’t find anyone else who documented this method.  If you have another working solution, please share.  This method is working great in our production and test environments.

Copy hard disk or partition image to another system using a network and netcat (nc)

Posted By vivek On August 12, 2007 @ 10:35 am In Backup, CentOS, Data recovery, Debian Linux, File system, FreeBSD, Gentoo Linux, Howto, RedHat/Fedora Linux, Suse Linux, Sys admin, Tips, Ubuntu Linux | 12 Comments

[1]

netcat utility (nc command) considered as TCP/IP swiss army knife. It reads and writes data across network connections, using TCP or UDP protocol. It is designed to be a reliable “back-end” tool that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need and has several interesting built-in capabilities.

I also install the netcat package for administering a network and you’d like to use its debugging and network exploration capabilities.

One my favorite usage is to migrating data between two server hard drives using netcat over a network. It is very easy to copy complete drive image from one server to another.

You can also use ssh for the same purpose, but encryption adds its own overheads. This is tried and trusted method (hat tip to [2] karl) .

Make sure you have [3] backup of all important data.

Install netcat

It is possible that nc may not be installed by default under Redhat / CentOS / Debian Linux.

Insall nc under Redhat / CentOS / Fedora Linux

Use yum command as follows:
# yum install nc
Output:

Loading "installonlyn" pluginLoading "rhnplugin" pluginSetting up Install ProcessSetting up repositoriesrhel-x86_64-server-vt-5   100% |=========================| 1.2 kB    00:00rhel-x86_64-server-5      100% |=========================| 1.2 kB    00:00Reading repository metadata in from local filesParsing package install argumentsResolving Dependencies--> Populating transaction set with selected packages. Please wait.---> Downloading header for nc to pack into transaction set.nc-1.84-10.fc6.x86_64.rpm 100% |=========================| 6.9 kB    00:00---> Package nc.x86_64 0:1.84-10.fc6 set to be updated--> Running transaction check

Dependencies Resolved

============================================================================= Package                 Arch       Version          Repository        Size=============================================================================Installing: nc                      x86_64     1.84-10.fc6      rhel-x86_64-server-5   56 k

Transaction Summary=============================================================================Install      1 Package(s)Update       0 Package(s)Remove       0 Package(s)         

Total download size: 56 kIs this ok [y/N]: yDownloading Packages:(1/1): nc-1.84-10.fc6.x86 100% |=========================|  56 kB    00:00Running Transaction TestFinished Transaction TestTransaction Test SucceededRunning Transaction  Installing: nc                           ######################### [1/1] 

Installed: nc.x86_64 0:1.84-10.fc6Complete!

Debian / Ubuntu Linux netcat installation

Simply use [4] apt-get command:
$ sudo apt-get install netcat

WARNING! These examples may result into data loss, ensure there are good backups before doing this, as using command wrong way can be dangerous.

How do I use netcat to copy hard disk image?

Our sample setup

-----------------------HostA // 192.168.1.1------------------------           sda        NETWORK           sdb------------------------HostB // 192.168.1.2-------------------------

Your task is copy HostA /dev/sda to HostB’s /dev/sdb using netcat command. First login as root user

Command to type on hostB (receiving end ~ write image mode)

You need to open port on hostB using netcat, enter :
# netcat -p 2222 -l |bzip2 -d | dd of=/dev/sdb
Where,

• -p 2222 : Specifies the source port nc should use, subject to privilege restrictions and availability. Make sure port 2222 is not used by another process.
• -l : Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host.
• bzip2 -d : Compresses image using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding. This will speed up network transfer ( -d : force decompression mode)
• dd of=/dev/sda : /dev/sda is your hard disk. You can also specify partition such as /dev/sda1

Command to type on hostA (send data over a network ~ read image mode)

Now all you have to do is start copying image. Again login as root and enter:
# bzip2 -c /dev/sda | netcat hostA 2222
OR use IP address:
# bzip2 -c /dev/sda | netcat 192.168.1.1 2222

This process takes its own time.

A note about latest netcat version 1.84-10 and above

If you are using latest nc / netcat version above syntax will generate an error. It is an error to use -l option in conjunction with the -p, -s, or -z options. Additionally, any timeouts specified with the -w option are ignored. So use nc command as follows.

On hostA, enter:
# nc -l 2222 > /dev/sdb
On hostB, enter:
# nc hostA 2222< /dev/sda
OR
# nc 192.168.1.1 2222< /dev/sda

Using a second machine (hostB), connect to the listening nc process at 2222 (hostA), feeding it the file (/dev/sda)which is to be transferred. You can use bzip2 as follows.
On hostA, enter:
# nc -l 2222 | bzip2 -d > /dev/sdb
On hostB, enter:
# bzip2 -c /dev/sda | nc 192.168.1.1 2222

Further readings

• netcat command other usage - [5] Linux port scanning with netcat (nc) command (you can also use [6] nmap command for port scanning)
• nc and bzip2 command man page
• [7] A good collection of netcat command usage
• [8] Another good netcat disk image howto

How do I improve performance?

As suggested by anonymous user:

You should definitely use bs=16M or something like that. Otherwise, the copy will take forever. Copying a 300 GB hard drive over a 1 Gbps cross-over cable took about 1 1/2 hours or so using bs=16M Without this option, the same thing would have taken about 7 hours.

In short use command as follows:
# netcat -p 2222 -l |bzip2 -d | dd of=/dev/sdb bs=16M

Updated for accuracy.

Article printed from nixCraft: http://www.cyberciti.biz/tips

URL to article: http://www.cyberciti.biz/tips/howto-copy-compressed-drive-image-over-network.html

URLs in this post:
[1] Image: http://www.cyberciti.biz/tips/category/networking/
[2] karl: http://www.meme.com/
[3] backup of all important data: http://www.cyberciti.biz/tips/perform-backups-for-the-linux-operating-system.html
[4] apt-get command: http://www.cyberciti.biz/tips/linux-debian-package-management-cheat-sheet.html
[5] Linux port scanning with netcat (nc) command: http://www.cyberciti.biz/faq/linux-port-scanning/
[6] nmap command for port scanning: http://www.cyberciti.biz/tips/linux-scanning-network-for-open-ports.html
[7] A good collection of netcat command usage: http://www.terminally-incoherent.com/blog/2007/08/07/few-useful-netcat-tricks/
[8] Another good netcat disk image howto: http://www.novell.com/coolsolutions/feature/19486.html

To use USB:

Start your Virtual Machine, and then connect to the VM Monitor
In the VM Monitor, issue the following command:

info usbhost

Now you will see a listing of all the available USB devices, very similar to the command ‘lsusb’ you can run from the host

Device 5.2, speed 1.5 Mb/s
    Class 00: USB device 045e:0040, Microsoft 3-Button Mouse with IntelliEye(TM)
Device 3.2, speed 12 Mb/s
    Class 00: USB device 0bda:8150, Linksys USB LAN Adapter
Device 1.3, speed 480 Mb/s
    Class 00: USB device 0457:0151, USB Mass Storage Device

Find the one in the list that you want to add to your running guest.  In my example above, I want to add the Mass Storage Device.  Note the hexadecimal code for the device, and type the following:

usb_add host:0457:0151

The 0457:0151 matches the code for the device I want to add.

You’re Done!
Your guest OS will now recognize the USB device in it’s normal fashion, as though you had just plugged it in.

Troubleshooting:
Did you receive the following error?

Could not open /proc/bus/usb/devices

There is an easy fix. But some background…At some point in time, the location for the USB mount point changed from /proc/bus/usb to /dev/bus/usb. However, many application still reference the older location (such as usbview, and in this case QEMU). There is an easy work around though:
NOTE: This is reported as a problem for VMWare Server and Virtual Box as well - the fix is the same regardless.

You need to simply re-enable the old mount point. Using your favorite editor, open the file at: ‘/etc/init.d/mountdevsubfs.sh’

Look for the code after line 38 that is similar to this:

#
# Magic to make /proc/bus/usb work
#
#mkdir -p /dev/bus/usb/.usbfs
#domount usbfs “” /dev/bus/usb/.usbfs -obusmode=0700,devmode=0600,listm$
#ln -s .usbfs/devices /dev/bus/usb/devices
#mount –rbind /dev/bus/usb /proc/bus/usb

and those last four lines so that it looks like this:

#
# Magic to make /proc/bus/usb work
#
mkdir -p /dev/bus/usb/.usbfs
domount usbfs “” /dev/bus/usb/.usbfs -obusmode=0700,devmode=0600,listm$
ln -s .usbfs/devices /dev/bus/usb/devices
mount –rbind /dev/bus/usb /proc/bus/usb

Then run this command:

sudo /etc/init.d/mountdevsubfs.sh start

(Note again: The above fix is useful for other applications as well, including VMWare Server and Virtual Box)

Other info:
At you Host’s terminal prompt, type:

lsusb

This will provide a listing of all USB devices on your host system.

In the VMMonitor, type:

info usbhost

This will provide a listing of USB devices available to your VM Guest

What an exciting 3 minutes and 8 seconds… Jason Lezak bested the French anchor in the final leg of the Men’s 4 x 100m relay to set a new world record, and return the Gold medal to America in this event, of which the US had won 9 consecutive times since 1964, with a Silver in 2000, and a Bronze in 2004.

The Frenchman was quoted as saying, “The Americans? We’re going to smash them [ in the 4x100]. That’s what we came here for.”  Even the NBC announcers were saying that the French were going to win, and beat the world record (of which the other US relay team had set just the day before).

Ahh, the drama of the Olympics, when 32-year-old Lezak, the oldest member of the USA Men’s swim team had the fastest EVER split time to win by a fingertip, 8 one-hundreths of a second. 3:08.24 set a new world record.

My wife and I joke in the weeks before the Olympics about our own ‘training’, to stay up late to watch it on TV.  Well all that ‘training’ payed off last night, as we watched in amazement as Lezak came from behind in the final leg to win the Gold!  We were jumping up and down, cheering and high-fiving!  Now we are going to be up late every night for the next two weeks…

Congratulations to Team USA, and to Jason Lezak, Cullen Jones, Garrett Weber-Gale, and Michael Phelps for a well deserved victory!

Did you watch the race?  What did you think?

I try to keep my posts on IT issues, but the excitement of the Olympics last night has me briefly diverting.  I found this uncredited photo of the celebration:

First, make sure you enable USB when you start your VM.  An very basic example startup script might be:

qemu-system-x86_64 \
-usb \
-m 1024 \
-hda /home/dave/vm/myubuntu/mynewroot.qcow2

The key part being the -usb command.

Plug the USB device into your VM Host system.

At a terminal prompt, type:

lsusb

This will provide a listing of all USB devices on your host system.  It is necessary to run the command here, but I thought it would be a useful viewpoint.

Now, start your VM, and switch to the VM Monitor (Press Ctrl-Alt-2).  If the VM is already running, just switch to the VM Monitor.

In the VMMonitor, type:

info usbhost

You will likely receive the following error:

Could not open /proc/bus/usb/devices

There is an easy fix.  But some background…At some point in time, the location for the USB mount point changed from /proc/bus/usb  to /dev/bus/usb (probably because USB should be used out of userspace).  However, many application still reference the older location (such as usbview, and in this case QEMU).  There is an easy work around though:
NOTE: This is reported as a problem for VMWare Server and Virtual Box as well - the fix is the same reguardless.

You need to simply re-enable the old mount point.  Using your favorite editor, open the file at: ‘/etc/init.d/mountdevsubfs.sh’

Look for the code after line 38 that is similar to this:

#
# Magic to make /proc/bus/usb work
#
#mkdir -p /dev/bus/usb/.usbfs
#domount usbfs “” /dev/bus/usb/.usbfs -obusmode=0700,devmode=0600,listm$
#ln -s .usbfs/devices /dev/bus/usb/devices
#mount –rbind /dev/bus/usb /proc/bus/usb

and those last four lines so that it looks like this:

#
# Magic to make /proc/bus/usb work
#
mkdir -p /dev/bus/usb/.usbfs
domount usbfs “” /dev/bus/usb/.usbfs -obusmode=0700,devmode=0600,listm$
ln -s .usbfs/devices /dev/bus/usb/devices
mount –rbind /dev/bus/usb /proc/bus/usb

Then run this command:

sudo /etc/init.d/mountdevsubfs.sh start

(Note again: The above fix is useful for other applications as well, including VMWare Server and Virtual Box)

Now you can return to the VM Monitor:

info usbhost

Now you will see a listing of all the available USB devices.  Find the one in the list that you want to add to your running guest, and type the following:

usb_add host:04f9:0028

OpenSSH is probably the most important IT administration tool - ever.  It allows for secure remote access to a system, with many tools built on top of the underlying encryption.  SSHfs is one of those tools.  SSHfs allows you to mount a remote file system, over the secure SSH protocol.  That is just plain cool.  Let me show you how you can be cool too:

My example is using Ubuntu 8.04.1 x64 Server w/ OpenSSH installed:

On the host, install the sshfs package.  On Ubuntu it is easy:

  sudo apt-get install sshfs

And load the kernel module:
 
  sudo modprobe fuse

Now from a remote system you can mount the host with:
  sudo sshfs <yoursshuser>@<ipaddressofHostServer>:/path/on/remote/system /local/mount/point

You now have secure access to the remote file system that you mounted in /local/mount/point.  I use this regularly.  I find it indispensable.  Do you?

Virtualization Note:

This is very useful in making it easy to copy ‘Raw’ virtual disks using the ‘cp’ command’s ’sparse’ options.  I’ll link to that upcoming article shortly.

I recommended using Raw files for your Virtual Machine’s virtual disks (see other post for the reasons), but it was brought to my attention, how do I copy these huge files?  Let’s look at that:

A Raw file is in a format that mimics a hard disk (block level).  It is supported in “sparse” format on file systems that support “holes” (NTFS, Ext3 included).  What does that mean?  When I create a Raw format file, let’s say of 100 GB in size, and then do a directory listing, it reports it as using 100GB of space:

dave@itpc11:~$ qemu-img create -f raw myvirtualfile.raw 100GB
dave@itpc11:~$ ls -l
-rw-r–r– 1 dave dave 107374182400 2008-08-01 15:26 myfile.raw

But wait?  My entire hard drive is only 40GB, how is this possible?  That is the sparse part of the file format, and the fact that Ext3 supports holes.  Run this command instead:

dave@itpc11:~$ ls -ls
0 -rw-r–r– 1 dave dave 107374182400 2008-08-01 15:26 myfile.raw

Note the first value of 0.  The file is actually taking up no space (currently) on the hard disk.

Here is the problem that was posed to me:

When you try to copy the file, either to another computer, or even on the same computer, the system creates a copy that is 100GB in size!

That is true.  You need to tell the application you are using to copy the file with “sparse” file recognition enabled.  Only then will it copy just the data in use.

There are two ways that I do this.  I either use “cp” or “rsync”.  They both have support for sparse files.  However, I find that cp is a bit more efficient for simple copies.  Rsync must be used when you only want to copy changed bits (or any of the many other reasons you would use Rsync).

Some examples:

To simple copy the file on the same computer from location A to location B:

  cp –sparse=always /mydir/myfile.raw /mydir/mycopy.raw

This will be a fast copy

What if I need to copy to another computer, and I want to connect to that computer over SSH?

Typically, you would simple use Rsync, as it supports sparse and SSH.  However, I found Rsync’s sparse copy to be very slow.  I prefer to mount the remote location by a cool technique using SSHFS.

SSHFS

OpenSSH is probably the most imporatant IT administration tool - ever.  SSHfs allows you to mount a remote file system, over the secure SSH protocol.  That is just plain cool.  We’ll use it here.

First, install the sshfs package.  On Ubuntu it is easy:

  sudo apt-get install sshfs

Now load the kernel module:
 
  sudo modprobe fuse

Now simply mount the remote file system to a local folder:

  sudo sshfs <yoursshuser>@<ipaddressofserver>:/path/on/remote/system /local/mount/point

Now you can use the cp command to copy the file to the remote server:
  sudo cp –sparse=always /mydir/myfile.raw /local/mount/point/mycopy.raw

Other methods:
As you begin to deal with more virtual disks, and they are ever increasing in size, you may want to convert them over to QCOW2 format.  Much like the VMDK format, they report there size based on the actual space they are using.  In the above example, if it was converted to a QCOW2 format, the file size would not longer be listed as 100GB:

dave@itpc11:~$