Wednesday, February 29, 2012

Catalyst 6500/6000 Switch High CPU Utilization


On Catalyst 6500/6000 Switches, there are two CPUs. One CPU is the supervisor engine CPU, which is called the Network Management Processor (NMP) or the Switch Processor (SP). The other CPU is the Layer 3 routing engine CPU, which is called the MSFC or the Route Processor (RP).
The SP CPU performs functions that include:
  • Assists in MAC address learning and aging
    Note: MAC address learning is also called path setup.
  • Runs protocols and processes that provide network control
    Examples include Spanning Tree Protocol (STP), Cisco Discovery Protocol (CDP), VLAN Trunk Protocol (VTP), Dynamic Trunking Protocol (DTP), and Port Aggregation Protocol (PAgP).
  • Handles network management traffic that is destined to the CPU of the switch
    Examples include Telnet, HTTP, and Simple Network Management Protocol (SNMP) traffic.
The RP CPU performs functions that include:

Tuesday, February 28, 2012

Cisco 7600 router commands

On a Cisco 7600 router we have these commands:

- "show ibc brief" which shows how much traffic goes to the CPU
instead of being hardware switched

- "show catalyst6000 traffic-meter" which shows the percentage of the
shared bus utilization
 
Reference:http://puck.nether.net/pipermail/cisco-nsp/2006-May/030802.html 

Understanding and Using Selective Packet Discard

Summary

This document provides an overview of the operation, configuration, and monitoring of Selective Packet Discard (SPD). It builds on this fundamental information with actionable technical details that can help network engineers implement SPD in an environment composed of differing hardware platforms and IOS releases.
In today's complex internetworks, various types of network traffic compete for the finite resources of a router. These might include an interior routing protocol such as Open Shortest Path First (OSPF) or Enhanced Interior Gateway Routing Protocol (EIGRP), network management traffic, and Border Gateway Protocol (BGP). SPD was designed to help ensure that network stability was not undermined during periods of high CPU-bound traffic. Although the safeguards added by SPD are typically applied to the BGP reconvergence use case, they are relevant to security when viewed in the context of availability.
SPD achieves its goals using two techniques: the provisioning of additional queuing capacity for control plane traffic and the implementation of a simple congestion-control mechanism for the interface input queues.

Reference:http://www.cisco.com/web/about/security/intelligence/spd.html
http://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186a008012fb87.shtml

Monday, February 20, 2012

IPC Terminology Fundamentals


Background Information

The Cisco IOS software Inter-Process Communication (IPC) module provides a communication infrastructure by which processes in a distributed system can interact with each other. It also provides transparent communication across backplanes, networks and shared memory.
IPC services serve as the means by which line cards (LCs) and the central route processor (RP) in a distributed system communicate with each other through an exchange of IPC messages sent from the RP to the LCs, and also between active and standby RPs. These messages include configuration commands and responses to those commands, and also "events" that need to be reported by an LC to the RP.
The Cisco 12000 Series, Cisco 10000 Series, Cisco 7600 Series, and the Cisco 7500 Series use a distributed architecture based on IPC messages. Under rare conditions, these routers may report these IPC-related log messages:
  • Cisco 12000 Series – %IPC-3-NOBUFF: The main IPC message header cache has emptied
  • Cisco 7500 Series – %IPC_RSP_CBUS-3-NOBUF: No more IPC memd buffers to transmit IPC message
Note: IPC is also used on Cisco 6400 series and Cisco 7304 series.

IPC Terminology Fundamentals

The more common IPC terminologies are:
  • IPC – Inter-Process Communication.
  • IPC Address – A 32-bit word that is composed of a 16-bit seat ID and a 16-bit port ID.
  • IPC Client – A software module that uses IPC services.
  • IPC Port – A communication endpoint within IPC used as the source and destination of all communication.
  • IPC Seat – An IPC seat is a computational element, such as a processor, that can be communicated with the help of IPC. An IPC seat is where IPC clients and ports reside.
  • IPC Session – An IPC session is an active simplex communication channel between two IPC ports.
All communication that uses IPC happens between IPC ports. A port is a communication endpoint in IPC. Each IPC port is associated with a logical address called an IPC address. IPC uses the IPC address of an IPC port as a return address when it sends IPC messages, or a destination address when it receives IPC messages.
Reference:Cisco 12000, 10000, 7600, and 7500 Series Routers: Troubleshooting IPC-3-NOBUFF Messages

Cisco process list

Reference:The show processes Command


The Processes


The table below explains the individual processes in the show processes, show processes cpu, and show processes memory outputs. This is not an exhaustive list.
Process Explanation
ARP Input Handles incoming Address Resolution Protocol (ARP) requests
BGP I/O Handles reading, writing, and executing Border Gateway Protocol (BGP) messages
BGP Scanner Scans the BGP and main routing tables to ensure consistency (this is a separate process since it can be quite time-consuming)
BGP Router Main BGP process which starts when the configuration is fully loaded
BOOTP Server The gateway's Bootstrap Protocol (BOOTP) server process
CallMIB Background Deletes the call history if the call history ages out and gathers call information
CDP Protocol
  • Main Cisco Discovery Protocol (CDP) - handles the initialization of CDP for each interface
  • If incoming packet, monitors the CDP queue and timers, then processes it
  • If timer event, sends update
Check heaps Checks the memory every minute. It forces a reload if it finds processor corruption.
Compute load avgs
  • Computes the five minute, exponentially-decayed output bit rate of each network interface, and the loading factor of the entire system. The load average is computed using the following formula: average = ((average - interval) * exp (-t/C)) + interval where t = 5 seconds and C = 5 minutes, exp (-5/60*5)) = .983
  • Computes the load of each interface (one by one), and checks the back-up interface's load (enables them or shuts them down according to the load).
*Dead* Processes as a group that is now dead. See Troubleshooting Memory Problems for more details.
Exec Handles console exec sessions; has a high priority
Hybridge Input Handles incoming transparent bridge packets that slip through the fast paths
*Init* System initialization
IP Background
  • Called upon when you change the encapsulation (for example, when an interface moves to a new state, an IP address changes, when you add a new Data Exchange Interface (DXI) map, or when some dialer timers expire)
  • Does the periodic aging of the Internet Control Message Protocol (ICMP) redirect cache
  • Modifies the routing table according to the status of the interfaces
IP Cache Ager Ages the routing cache and heals stale recursive routes. The ager runs once every time interval (once a minute by default) and checks to make sure that a recursive routing change has not made the entry invalid. Another function of this ager is to make sure that the entire cache gets refreshed approximately every 20 minutes.
IP Input Process-switched IP packets
IP-RT Background Periodically revises the gateway of last resort and IP static routes. This process is called on demand, right after the static routes (which the gateway of last resort may depend on) have been revised.
ISDNMIB Background Sends ISDN trap service and deletes the call queue if it ages out
ISDN Timers Handles ISDN carrier timer events
Load Meter Computes the load average for the different processes every five seconds, and the five minute exponentially-decayed busy time. The load average is computed using the following formula: average = ((average - interval) * exp (-t/C)) + interval, where:
  • t = 5 seconds and C = 5 minutes, exp (-5/(60*5)) = .983~= 1007/1024
  • t = 5 seconds and C = 1 minute, exp (-5/60)) = .920~= 942/1024
Multilink PPP out Processes multilink packets that have been queued from fast-switching (outbound half fast-switching)
Net Background
  • Performs a variety of network-related background tasks. These tasks must be performed quickly and may not block for any reason. The tasks that are called in the net_background process (for example, interface dethrottling) are time critical.
  • Executes the "Compute load avgs", "Per-minute Jobs", and "Net Input" processes
  • Handles interface throttling
Net Input
  • Handles otherwise unknown packets. This is done at process level so that input queuing comes into play. If you operate at interrupt level, you could very easily lock up the router.
  • Handles some known protocols which you may decide should be offered to bridging. In this case, net_input either sends the packet to NULL, or bridges it.
Net Periodic Performs interface periodic functions every second such as:
  • resetting the periodic counter
  • clearing the input error rate counter
  • checking serial lines for restarting from glitches
  • performing any periodic keep-alive functions
  • checking protocol routing table consistency
  • doing bridge state consistency checking
  • announcing line protocol up or down events
Per-minute Jobs Performs the following tasks once a minute:
  • analyzes stack usage
  • announces low stacks
  • executes registered one_minute jobs
Per-second Jobs Performs a variety of tasks every second; executes registered one_second jobs
Pool Manager Manager process for managing growth and discarding requests from dynamic pools at the interrupt level
PPP Manager
  • Manages all PPP Finite State Machine (FSM) operations by processing PPP input packets and interface transitions
  • Monitors the PPP queue and the PPP timers (negotiation, authentication, idle, and others) Note: By serializing events that might be detected which interrupt routines in other processes, many common bugs can be avoided.
OSPF Router Main Open Shortest Path First (OSPF) process
OSPF Hello The OSPF process which receives hello
*Sched* The Scheduler
Serial Background Watches events and branches to the correct service routine for each expired event (mainly reset of interfaces)
Spanning Tree
  • Executes the Spanning Tree Protocol (STP), a single process that handles the multiple spanning tree algorithm
  • Monitors the STP Queue:
    • Process incoming STP packets
  • Monitors the STP timers:
    • Hello timer
    • Topology change timers
    • Digital Equipment Corporation (DEC) short age out timer
    • Forward delay timer
    • Message age timer
Tbridge Monitor
  • Dispatches "interesting packets" to the appropriate handler ("interesting traffic" is Cisco Group Management Protocol (CGMP), Internet Group Management Protocol (IGMP), OSPF packets [multicasts]
  • Monitors multicast timers which check station entry age-outs and circuit group active circuits
TCP Driver Handles the sending of packet data over a Transmission Control Protocol (TCP) connection. It includes opening or closing connections, or dropping packets when queues are full. Remote Source-Route Bridging (RSRB), serial tunneling (STUN), X.25 switching, X.25 over TCP/IP (XOT), Data-link Switching (DLSW), translation, and all TCP connections starting or ending at the router currently use TCP Driver.
TCP Timer Handles retransmission of timeout packets
Virtual exec Handles virtual type terminal (vty) lines (for example, Telnet sessions on the router).

Friday, February 17, 2012

PPPoE session

PPPoE discovery stage

2.1.1 The PPPoE Active Discovery Initiation (PADI) packet
2.1.2 The PPPoE Active Discovery Offer (PADO) packet
2.1.3 The PPPoE Active Discovery Request (PADR) packet
2.1.4 The PPPoE Active Discovery Session-confirmation (PADS) packet



PPPoE session

Wednesday, February 15, 2012

MPLS LDP-IGP Synchronization

How MPLS LDP-IGP Synchronization Works

Packet loss can occur because the actions of the IGP and LDP are not synchronized. Packet loss can occur in the following situations:

•When an IGP adjacency is established, the router begins forwarding packets using the new adjacency before the LDP label exchange completes between the peers on that link.

•If an LDP session closes, the router continues to forward traffic using the link associated with the LDP peer rather than an alternate pathway with a fully synchronized LDP session.

The MPLS LDP-IGP Synchronization feature:

•Provides a means to synchronize LDP and IGPs to minimize MPLS packet loss.

•Enables you to globally enable LDP-IGP Synchronization on each interface associated with an IGP Open Shortest Path First (OSPF) or IS-IS process.

•Provides a means to disable LDP-IGP Synchronization on interfaces that you do not want enabled.

•Prevents MPLS packet loss due to synchronization conflicts.

•Works when LDP is enabled on interfaces using either the mpls ip or mpls ldp autoconfig command.

To enable LDP-IGP Synchronization on each interface that belongs to an OSPF or IS-IS process, enter the mpls ldp sync command. If you do not want some of the interfaces to have LDP-IGP Synchronization enabled, issue the no mpls ldp igp sync command on those interfaces.

If the LDP peer is reachable, the IGP waits indefinitely (by default) for synchronization to be achieved. To limit the length of time the IGP session must wait, enter the mpls ldp igp sync holddown command. If the LDP peer is not reachable, the IGP establishes the adjacency to enable the LDP session to be established.

When an IGP adjacency is established on a link but LDP-IGP Synchronization is not yet achieved or is lost, the IGP advertises the max-metric on that link.

###################################################################################

Synchronized here means that the local label bindings have been sent over the Label Distribution Protocol session to the Label Distribution Protocol peer. However, when the synchronization is turned on at router A and that router has only one link to router B and no other IP connectivity to router B via another path (this means via other routers), the OSPF adjacency never comes up. OSPF waits for the Label Distribution Protocol session to come up, but the Label Distribution Protocol session cannot come up because router A cannot have the route for the Label Distribution Protocol router ID of router B in its routing table.
The OSPF and Label Distribution Protocol adjacency can stay down forever in this situation! If router A has only router B as a neighbor, the Label Distribution Protocol router ID of router B is not reachable; this means that no route exists for it in the routing table of router A. In that case, the Label Distribution Protocol-IGP synchronization detects that the peer is not reachable and lets OSPF bring up the adjacency anyway. In this instance, the link is advertised with maximum metric until the synchronization occurs.
This makes the path through that link a path of last resort. In some instances, the problem with the Label Distribution Protocol session might be a persistent one; therefore, it is probably not desirable to keep waiting for the IGP adjacency to be established. The solution for this is to configure a Holddown timer for the synchronization. If the timer expires before the Label Distribution Protocol session is established, the OSPF adjacency is built anyway. If everything is fine with Label Distribution Protocol across that link, Label Distribution Protocol also forms a session across the link. While OSPF is waiting to bring up its adjacency until Label Distribution Protocol synchronizes, the OSPF interface state is down and OSPF does not send Hellos onto that link.

Reference: MPLS LDP-IGP Synchronization
http://salfarisi25.wordpress.com/2011/01/18/multiprotocol-label-switching-label-distribution-protocol/

Advanced VRF Import and Export Features

Selective VRF Import

    Selective route import uses a route map that can filter the routes selected by the RT import filter.
        The routes imported into a VRF are BGP routes, so you can use match conditions in a route map to match any BGP attribute of a route.
        The import route map is deployed in the receiving VRF.
        A route has to pass the RT import filter first and then the import route map.
        First, at least one of the RTs attached to the route needs to match one of the import RTs configured in the VRF.
        Second, the route is permitted by the import route map.
    import map route-map-name attaches a route map to the VRF import process.
        A route is imported into the VRF only if at least one RT attached to the route matches one RT configured in the VRF AND the route is accepted by the route map.



Selective VRF Export

    Some advanced MPLS VPN topologies are easiest to implement if you can attach a variety of RTs to routes exported from the same VRF.
        This capability allows only a subset of the routes exported from a VRF to be imported into another VRF.
        The export route map is deployed in the originating VRF.
       A route map can be specified for each VRF to attach additional RTs to routes exported from that VRF.
        The export route map performs only the attachment of RT's. It does not perform any filtering function.
    Attributes attached to a route with an export route map are combined with the export RT attributes.
        If you specify export RTs in a VRF and set RTs with an export route map, all specified RTs will be attached to the exported route.
    set extcommunity rt extended-community-value [additive] sets the BGP extended community attribute for a RT.
    export map route-map-name attaches a route map to the VRF export process.

Copied from:  http://mynetworkingwiki.com/index.php/Advanced_VRF_Import_and_Export_Features

Tuesday, February 14, 2012

IS-IS adjacency is not formed due to MTU mismatch

IS-IS adjacency is not formed due to MTU mismatch

Solution: 

1. change the MTU under interface

2. change the clns mtu under the interface

Reference:MTU Mismatch Problem in IS-IS 

IS-IS adjacency is not formed due to MTU mismatch 

http://cisco.iphelp.ru/faq/5/ch11lev1sec1.html 

show stacks Command

show stacks Command

show stacks is an exec command that is commonly used to diagnose system crash situations. The first section of this command's output displays stack utilization of processes and interrupt routines, and the reason for the last system reboot. When a system crash happens, failure type, failure program counter (PC), address (operand address), and a stack trace are saved by the ROM Monitor. The show stacks command displays the data saved by the ROM Monitor. The stack trace is displayed in the second section of the show stacks command output (if there has been a system failure).
In the past, support engineers would submit the stack trace of their router to Cisco System's technical support representatives, who had access to symbol tables, object files, source code, and the stack decoder software. Today, the stack decoder is available online (from the CCO) and you can cut your router's stack trace from the output of the show stacks command and paste it in the input field of the stack decoder software. Stack decoder decodes the stack trace and creates a symbol file. The symbol file (perhaps along with other information in the trace) usually provides enough information to isolate the cause of any problems that were experienced.

Reference: http://www.ciscopress.com

Monday, February 13, 2012

root guard bpdu guard

Why root guard

The administrator can set the root bridge priority to 0 in an effort to secure the root bridge position. But there is no guarantee against a bridge with a priority of 0 and a lower MAC address.

"root guard " will protect your switch against unplanned spanning tree changes.  (Superior information received)

"bpdu guard" will protect your switch against unplanned spanning tree period!  (ANY information received)

best practice, use BPDU Guard on all access ports unless there is a special situation where you shoud use ROOT Guard only on that port or few ports and BPDU ROOT Guard on trunks in your domain, to preserve your ROOT Bridge position.

 Reference: https://learningnetwork.cisco.com
 Configuring Spanning Tree PortFast, BPDU Guard, BPDU Filter, UplinkFast,
Understanding Rapid Spanning Tree Protocol (802.1w)
Spanning Tree Protocol Root Guard Enhancement

Creation and Management of Catalyst 3750 Switch Stacks

Creation and Management of Catalyst 3750 Switch Stacks

Sunday, February 12, 2012

Basic Concept of Multicast VPN (MVPN )

Basic Concept of MVPN

The basic concept of MVPN is as follows:
The Service Provider has an IP Network with its own unique IP multicast domain (ie: P-Network).
The MVPN customer has an IP Network with its own unique IP multicast domain (ie: C-Network).
The Service Provider MVPN network forwards the customer IP multicast data to remote customer sites. To achieve this, customer traffic (C-packets) is encapsulated at the Service Provider PE inside P- packets. The encapsulated P-packet is then forwarded to remote PE sites as native multicast inside the P-Network
During this process, the P-Network has no knowledge of the C-Network traffic. The PE is the device that participates in both networks. Note there may be more than one Customer Network per PE. 

Multicast Routing Inside the VPN Versus Multicast Routing Inside the Provider Network

A PE router an MVPN network has multiple multicast routing tables, as well as multiple instances of PIM, IGMP, and MSDP. There is one global table and a table per MVRF.
Multicast Domains is based on the principle of encapsulating multicast packets from a VPN in multicast packets to be routed in the core. As multicast is used in the core network, PIM must be configured in the core.
PIM-SM, PIM-SSM, and PIM-BIDIR are all supported inside the provider core for MVPN.
PIM-SM or PIM-SSM is the recommended PIM option in the provider core, because PIM-BIDIR is not yet supported by all platforms, PIM-SM, PIM-SSM, PIM-BIDIR and PIM-DENSE-MODE are supported inside the MVPN.
MVPN has the concepts of Multicast Distribution Trees (MDT). An MDT is sourced by a PE router and has a multicast destination address. PE routers that have sites for the same MVPN will all source to a Default-MDT and also join to receive traffic on it.
There is a distinction between Default-MDTs and Data-MDTs. A Default-MDT is a tree that is `always-on' and will transport PIM control-traffic, dense-mode traffic and rp-tree (*,G) traffic. All PE routers configured with the same default-MDT will receive this traffic.
Data-MDTs are trees that are created on demand and will only be joined by the PE routers that have interested receivers for the traffic. They can be created either by a traffic rate threshold and/or source-group pair.
Default-MDTs must have the same group address for all VRFs that comprise a MVPN. Data-MDTs may have the same group address if PIM-SSM is used. If PIM-SM is used, they must have a different group address, as providing the same one could result in the PE router receiving unwanted traffic. This is a PIM-SM protocol issue, not an implementation issue. 





Source Specific Multicast (PIM-SSM)

Copied from packetlife

Source-Specific Multicast (SSM), defined in RFC 4607, extends this concept to identify a set of multicast hosts not only by group address but also by source.

SSM brings several important benefits over ASM. Because an SSM channel is defined by both a source and a group address, group addresses can be re-used by multiple sources while keeping channels unique.

One of the biggest advantages SSM holds over ASM is that it does not rely on the designation of a rendezvous point (RP) to establish a multicast tree.

Reference: http://packetlife.net/
Cisco
WIKI

Set duplicate IPs on a Cisco router?

Can you set duplicate IPs on a Cisco router?

Yes, you if disable ip routing.

With ip routing enabled,

A Cisco router will ONLY complain duplicate IP on the LAN interfaces

A Cisco rotuer wil NOT complain duplicate IP on the WAN interfaces

Troubleshooting multicast sparse Mode

Multicast traffic always send to group address, never from

IGMP (router to client communication)
IGMP testing commands - simulate multicast receiver
  • ip igmp join-group - response icmp echo, process switching mpackets
  • ip igmp static-group - not response icmp echo

PIM – Protocol Independent Multicast - Router to router communication

Sparse Mode or Dense modes - this ONLY affect the traffic SENDING OUT interface; an interface will always receive multicast traffic once enble either "ip pim XXX mode".  

Sparse Mode process:

multicast server - [S,G] fisrt hop router - unicast - RP - [*,G] - last hop router - receiver
multicast server - [S,G] fisrt hop router - [S,G] - RP - [*,G] - last hop router - receiver
multicast server - [S,G] fisrt hop router - [S,G] - last hop router - receiver 





Useful command:


ip igmp join-group multicast-address - simulate a join host

sh ip mr
multicast-address - check [S,G] and [*,G]

sh ip mr count - check multicast traffic stats, look for failures such as rpf or oil-null

sh ip rpf source-ip-address


Reference: IPEXPERT
Basic Multicast Troubleshooting Tools 
IP Multicast Troubleshooting Guide 
IP Multicast Routing Commands

Saturday, February 11, 2012

IP Virtual Reassembly

Virtual Reassembly is special IOS feature that allows the router to obtain full picture of a fragmented packet on the fly. When you activate virtual-reassembly on interface, using the command ip virtual-reassembly, IOS starts tracking all incoming fragmented packets. The code delays fragmented packets until it receives all of them, or until the maximum reassembly timeout expires (there are some other thresholds, discussed below). After this, the router performs “virtual” datagram reassembly. Here “virtual” means the packet is not getting actually assembled into a single entity, but rather IOS views it as a whole for subsequent processing. If the router does not receive all fragments during the reassembly timeout, the incomplete packet is dropped.

Ref: http://blog.ine.com/2008/11/05/dealing-with-fragmented-traffic/
http://www.cisco.com/en/US/docs/ios/12_3t/12_3t8/feature/guide/gt_vfrag.html

Friday, February 10, 2012

NAT - Ability to Use Route Maps with Static Translations


Previous to this feature, route mapping was supported only with dynamic Network Address Translation (NAT) translation.
The NAT—Ability to Use Route Maps with Static Translations feature enables NAT multihoming capability with static address translations. Multihomed internal networks now can host common services such as the Internet and Domain Name System (DNS), which are accessed from different outside networks. 

Tuesday, February 7, 2012

Traceback/Crashinfo

Traceback/Crashinfo

Traceback/Crashinfo

Traceback is a record of abnormal function calls that is usually shown on the console of the PIX firewall, when an abnormal situation occurs. Problems with PIX normal functionality may produce a console traceback message. Not every traceback is serious; some are cosmetic. But, every traceback should be decoded and analyzed. Because the traceback is in hexadecimal values, you will not be able to decode it. Therefore, you need to engage the Cisco Support team for decoding and analyzing it. The problematic function (routines) that causes the traceback might have severe effects, such as crashing the whole PIX and thereby requiring a reboot.