Wednesday 5 September 2007

Workbench Update Bandwidth Saver - ISA 2000

A while back, Susan Bradley mentioned a service called AutoPatcher: AUTOPATCHER TOLD TO CLOSE UP.

To be honest, we did not realize that there was even a need for this type of service! Who was to know that in certain parts of the world, bandwidth is paid for on a per bit basis!

With that realization, we can see why AutoPatcher would be totally relevant.

In our case, we have a workbench server setup that mitigates our bandwidth costs by caching updates locally. It is also isolated from our internal network.

The workbench servers used to run on two VIA EPIA miniITX boards crammed into an old Compaq case with a couple of tiny power supplies and a couple of laptop hard drives to provide local storage.

Both machines ran Windows Server 2000. One was DC and one was a domain member with ISA 2000 installed on top. Why ISA 2000? Because, in our experience ISA 2004 (we haven't even bothered with ISA 2006) could not come close to caching the content we wanted cached: Microsoft/Windows Updates via the Microsoft Update site.

So, what exactly does this mean for us and our bandwidth costs? Well, for example, we just finished rebuilding three Toshiba Tecra laptops. They were all XP Professional Service Pack 1 versions. We then needed to install Office 2003 Professional on top of that!

Whenever we deal with a machine that is using older media, we have an unpacked XP Service Pack 2 folder resident on the workbench DC. We then have a shortcut in the root of the share:

\\WorkbenchDC01\Company\Microsoft\XP\SP2Unpacked\i386\update\update.exe /passive /forcerestart /n /f

Once the shortcut is double clicked on the machine that is pre-SP2, the SP runs on its own with the reboot happening automatically at the end.

From there, we go to Microsoft's Update Site and upgrade the system to Microsoft Update right away. A reboot later and we are on to all of the critical and optional updates to download and install.

Any guesses on the volume of data needed to update each of these laptops' Windows install only as of this blog post?

At last count, it was in the neighbourhood of 210MB on the first run of critical and optional updates! Never mind Office 2003 Service Pack 2's additional 102MB. That would be a combined total of close to 1GB for the three machines first run! There were more to come after that.

When we work with situations like this often enough, we hit close to 95% of all updates cached locally in ISA 2000. The subsequent updates, as well as the Office Service Pack are also cached locally. When we have multiple units to run updates on without any updates being run lately, we always let one run through first to catch any new updates into the cache. From there, the rest of the units will pull from cache.

Do them at the same time, and they all will pull from the Web and thus cost us extra bandwidth and time.

In the above situation, all three laptops were running their post download update install routine within a short period of time.

Recently, the workbench VIA EPIA W2K DC had its hard drive blow up. So, we moved everything into a virtual setup on one box.

Here is what we did for hardware:

  • Intel D945GTP Main Board
  • Intel Pentium D 950 3.4 GHz
  • 3GB Kingston DDR2
  • 320GB Seagate RAID 1 (Software)
  • 10/100 Built In NIC
  • Gigabit D-Link DGE-530T
  • Gigabit D-Link DGE-530T
  • Antec Minuet 300
And the software to do it:

  • Windows Server 2003 (Host)
  • Windows Server 2000 Standard x 2 (Guests)
  • ISA 2000 (member server)
Memory for the virtual machines is set to 512MB each. The DC has a dynamic VHD of 72GB and is currently using all of 3.4GB. The member server has 2 VHDs attached: One is a dynamic 72GB for the OS and ISA install and is using 3.0GB while the second is a dynamic 120GB that is using 1.5GB.

ISA 2000 has the following cache settings:

The "Less frequently..." lets the objects sit in the cache longer. Thus, we have those updates staying put instead of being pulled from the Microsoft download site.

There is 75GB available to ISA to cache updates. So far, it is holding about 1.5GB.

The 3 NICS are physically setup in such a way as to isolate the workbench setup as follows:
  • Gb NIC 1: Internal IP for Virtual Server and VM management - No VMs attached.
  • Gb NIC 2: Static IP 192.168.x.x is bogus but plugged into the Workbench Gigabit Switch (File & Printer Sharing off).
  • Mb NIC 3: Static IP 192.168.x.x is bogus as it is plugged into the Internet (File & Printer Sharing off).
The workbench VM DC and ISA box both share NIC 2 for "internal" connectivity to the Gigabit switch that we connect any machines we need to run updates on or keep isolated due to some sort of infection. The ISA 2K VM also has a second virtual NIC that is tied to NIC 3 and pulling an IP from our ISP.

We have had great success with this arrangement as well as the previous VIA based one. Due to that success, other than our endeavouring to get ISA 2004 to do the same thing and failing a few years ago, we are leaving things status quo.

Some of you may have a similar arrangement, or know whether ISA 2004 or 2006 in their current iterations would actually accomplish what we are doing with ISA 2000. Is it possible? If so, please feel free to let us know.

And to our friends in Australia, we do hope that you will be able to utilize this kind of setup to facilitate a huge reduction in your bandwidth overhead. It works for us.

UPDATE: 2007-09-22: A sample update run on multiple HP systems:
  • HP Pavilion a605x with XP Pro 32bit SP2 OLP fresh install
  • Update to MS Update = Reboot
  • Update run 1: 202MB includes Critical+Optional+3 hardware
    • Total download time for 202MB: 8 minutes (keep in mind the age of these machines)
    • Install of 202MB including IE7: 31 minutes
  • Update run 2: 43.2MB includes Critical+Optional+1 hardware
    • Total download time for 43.2MB: 3 minutes
    • Install time: ~10 minutes (keep in mind the large .NET updates)
  • Update run 3: 8.8MB (.NET 1.1 SP1)
    • Total download time: less than a couple of seconds
    • Install time: ~2 minutes
Total time spent on updating all of these machines: ~54 minutes plus a little time for setting them up and taking them off of the bench. Our client's site has the rest of the installs setup to deliver the balance of the needed software via Group Policy Software Install. So, we will let WSUS take care of the balance of updates post domain install.

For Windows Vista Business and Ultimate systems that we have received from our System Builder, they seem to be always up to date. Every time we run Windows Update, upgrade the Windows Update Service to Microsoft Update, there are usually no updates to apply.

We will make a point of visiting them to see just how they do all of that since our OPK/OEM Preinstall requirements and experiences are virtually nil.

Philip Elder
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.


stryqx said...

One of the poor pay-per-bit Aussies here! (However, my quota is generous and I only pay extra for excess.)

I used to use squid - either on a separate box running FreeBSD, in a FreeBSD VM or natively (SquidNT).
Now I use a local WSUS server (was 2.0, now 3.0) with an appropriate approval structure set for Unassigned Computers. I then inject the WSUS settings (ie faking up the Group Policy registry settings for AU), run wuauclt.exe /detectnow repeatedly until all patches are installed.

This could be scripted to build an AutoPatcher-like package that could be run locally.

The reason we pay-per-bit is that all the Australian ISPs get to pay for both ingress and egress traffic across the trans-Pacific submarine fibres. That's the way the Yanks want to peer, which sounds more like being held to ransom rather than peering Sucky.

Philip Elder Cluster MVP said...

We considered using WSUS as the basis for the setup.

I do believe we found that, even with scripting, there was more babysitting involved.

We wanted to have a setup that had the least amount of modification on the stock setup as possible. The ISA setup seems to fit that bill.

When we have a number of machines on the bench, it is just a matter of keeping an eye open for what each one is doing.

Run 1= ~300MB of Critical and Optional Updates
Run 2= ~75MB of Critical and Optional Updates
Run 3= 20MB iirc.

That is it. There is an initial reboot required for Windows Installer 3.1 & the update to Microsoft Update capability.

Considering that we are on billable time, and things run a heck of a lot faster via the locally cached updates, it is a win-win situation.

A cool thing is, as the machines and laptops get faster, we have seen them run through the update downloads a lot more rapidly, sometimes they go by as rapid flashes ... it is neat to watch.

We have some older machines that we will be rebuilding soon, we can do some timed start to finish stats if you are interested. Would be good to see a time comparison between the two setups.


Absoblogginlutely! said...

interesting idea to use a caching server to make things faster! As for the office service pack, why not just slipstream it into your original office installation share so that it's already installed with sp3 (or whatever they are up to nowadays)
Out of curiosity how do you do the licence handling of office if you use a shared install to install? With the typical shared install you have to put the product key in the installation share which is not very helpful.