Friday, April 2, 2010

Slow MultiCast Imaging with Symantec Ghost

I'm running MultiCast Imaging with Symantec Ghost. The image I'm pushing out is 500GB. It was taking me over 24 hours to push the image out as I was only getting 200MB per minute.

My network was gigabit with a HP ProCurve Switch 2810-48G switch and gigabit NIC's in all the workstations.

Why was it going so slow?

After doing some research I found out that the HP switches have IGMP (Internet Group Management Protocol) disabled by default. IGMP is used to manage the membership of Internet Protocol multicast groups. A ghost cast session is an IGMP group, workstations need to join the IGMP session on the switch to participate in IGMP group traffic. When IGMP is disabled on the switch, the switch treats the multicast traffic as broadcast traffic sending it out over all ports in the specified VLAN - meaning every PC on the network receives the communication. Every PC or device on the VLAN recieves every packet sent from the ghost cast session as they do not have an application listening on the specified port; they will drop the packets.

This network is one big VLAN. Some devices on the VLAN could not support high speed network connectivity even though they had a gigabit NIC. One of these devices was a PACom WebGuard Camera Video Surveillance system. This Camera System was also receiving every packet from the ghost cast session - however because it is not designed to deal with such a large amount of data it was dropping packets meaning the ghostcast server needed to slow down to ensure every PC on the network received the data. A broadcast will only go the speed of the slowest device on the network. Bad cables will also effect the speed of the broadcast. Multicasts have the same principal, they will only go the speed of the slowest device in its multicast group, however you have narrowed it down to a group of devices instead of all devices in a specified VLAN.

To enable IGMP on a VLAN you just enter ip igmp into the switches config under the right area:

ProCurve Switch 2810-48G# show run

Running configuration:

; J9022A Configuration Editor; Created on release #N.11.15

hostname "ProCurve Switch 2810-48G"
snmp-server contact "Clint Boessen"
ip default-gateway 192.168.30.2
snmp-server community "public" Unrestricted
vlan 1
name "DEFAULT_VLAN"
untagged 1-48

ip address 192.168.30.50 255.255.255.0
ip igmp
exit
spanning-tree
password manager


You can also enable IGMP using the HP Web Interface if you have it enabled. Notice when I made the config change in the switches command shell it also updated it in the GUI:





If you have IGMP turned on but you are still experiancing slow ghosting speeds there may be another feature enabled called Automatic Broadcast Control (ABC). When you enable ABC, it automatically sets the broadcast limit (Bcast Limit) on all switch ports to 30% (except for any ports for which you have already set the Bcast Limit manually to a nonzero value). Please do some research into ABC if you have IGMP enabled but are still experiancing slow speeds.

Now with IGMP on we are getting awesome speeds:



The below performance monitor graph shows my Disk Speed in Megabytes per second (MBPS) on the ghost server. As you see my ghost session is getting between 20 to 30.

4 comments:

  1. In addition to this, the other thing about IGMP that I have repeated endlessly on our official forums is that if you turn on IGMP snooping, you need to ensure that your network has an IGMP querier configured - when switches learn about the associations between IGMP groups and switch ports, they forget them unless the association is periodically refreshed, such as by a querier (note that queriers are part of the IGMP protocol specification, which is written around routing; snooping by switches is conceptually parasitic on that routing interchange).

    Typically the IGMP querier device is something that can also act as a multicast (e.g. PIM-capable) router; you're allowed several potential queriers serving a network and in principle they elect one amongst themselves based on the lowest IP address. Note however that not all querier devices do this properly, so that multiple queriers is also a source of problems with switches from some vendors - see https://www-secure.symantec.com/connect/forums/ghost-115-flooding-our-network

    ReplyDelete
  2. Hey Nigel Bree,

    Have you come across this time-out problem with Symantec Ghost Suite 2.5 when pushing out large images?

    Please see:
    http://clintboessen.blogspot.com/2010/04/connection-timed-out-symantec-ghost.html

    ReplyDelete
  3. How would a multicast sender know that a receiver was dropping packets, as multicast is a one way protocol?

    ReplyDelete
  4. Multi-Cast is controlled by the IGMP protocol which has backwards/forwards communication with clients in the IGMP session. The data stream is one way - your correct however is controlled by IGMP.

    ReplyDelete