rb-ccie

[rb-ccie]

Ansible A3: Configuring a Vagrant Lab

In this article, we'll cover creating your target VMs to be managed with Ansible in Vagrant.

Create a working directory, drop into it, initialise a new Vagrant file, then download and start a new Ubuntu VM.

Log into the new VM with the autogenerated SSH key with the 'vagrant ssh' command.

You can access the second part in this series by clicking this link You can access the first part in this series by clicking this link.

I hope you found this useful.

[rb-ccie]

Ansible A2: Installing Ansible

As I went for a limited install there are a number of dependencies that need to be installed to get Ansible up and running.

Install git, clone the Ansible repo, then drop into the new ansible working directory.

Install python-setuptools.

Install the gcc compiler.

Install the python development tools.

Install the libffi and openssl development packages.

Run the sudo make command.

Run the sudo make install command You can access the third part in this series by clicking this link You can access the first part in this series by clicking this link

[rb-ccie]

Ansible A1: Installing Virtual Lab Software

Welcome to the first in a series of Ansible tutorials.

Pre-reqs: In this instance I'll be building the virtual lab on a Windows 8 PC and VMs from a numbers of OS's, including CentOS (my personal preference for Linux), Ubuntu and Arista EoS (Maybe even FortiOS if I find the time). In order to perform the actions detailed in some of the forthcoming articles, you'll need access to a Windows 8 PC and some virtual Arista and Fortinet images (Please do not ask me for copyrighted software, I will not provide it as I have no desire to jeopardise my customer agreements).

NB: Please note that this series is a result of my part time endeavours and constitues a basic walkthrough of creating a virtual lab, installing Ansible and managing remote hosts. It is absolutely not a comprehensive beginners guide and assumes some foundation level familiarity with Windows, Linux and Networking. As such, I do not baby step through each command, or cover OS installation in this guide.

That said, if you spot any errata in this series or have any questions, please feel free to (contact me)[http://www.rb-cc.ie/ask].

Download Oracle VirtualBox (v5.1.22 at the time of writing) from https://www.virtualbox.org/. I'll be selecting the Windows host package.

Download Vagrant (v1.9.7 at the time of writing) from https://www.vagrantup.com/.

. I'll be selecting the Windows 64 bit package.

Install VirtualBox.

Install Vagrant and restart the computer, when prompted.

Verify successful installation by running the vagrant version command.

Download your preferred Linux image (E.g. CentOS 7), create a new virtual machine, then set the new VM to boot from the Linux ISO image. Once the OS boots, verify basic network connectivity. You can access the next part in this series by clicking this link. I hope you found this useful.

[rb-ccie]

Quick Tip: Auto-generating a list of numbers in Windows Command Prompt

Sometimes you just need to generate large lists of numbers. Now, you could open Excel, type in a number and tediously drag down the required number of cells, but who has time to waste those valuable seconds? Here's how to do it with a couple of quick CLI commands.

NB: @echo off and @echo on aren't required. I just used them to avoid the increased output on the screen as the for loop iterates.

To auto generate a range of IP addresses in a /24, open the command prompt and type:

for /l %i in (1,1,254) do echo 10.0.0.%i >> ipaddresses.txt

To display the list of IP addresses you've generated, type:

type ipaddresses.txt

If you don't have Quick Edit mode enabled in the Command Prompt properties, it's just a matter of right clicking, choosing Select All from the context menu, then right clicking the highlighted text to copy it to your clipboard.

If you do have Quick Edit mode enabled, you simply need to click the Command Prompt icon in the top left of the window, choose Edit, Select All from the menu, then right click the highlighted text to copy it to your clipboard.

Ref: Guide to FOR /L on Windows

I hope you find this article useful.

[rb-ccie]

Adding Linux Style Aliases to Windows 8

If like me, you spend a lot of time typing file paths in the command prompt, you'll appreciate the usefulness of aliases. Linux style aliases allow you to collapse full command strings into a single word and even allow you to pass arguments from the cmd prompt. This article describes how to add one (or more) custom alias functions to the Windows 8 command prompt.

Pre-reqs: For this iteration of the batch file, Notepad++ and PSTools must be installed. It also requires the following directories %USERPROFILE%\Documents\Logs\, %USERPROFILE%\Documents\Tools\ & %USERPROFILE%\Documents\scripts. Although these can be customised to suit your needs.

Add the functions you want to a .cmd file:

NB: The $* operator allows you to pass arguments to the alias.

Save the cmd file to the directory of your choice. I chose: %USERPROFILE%\Documents\scripts\cmd\

Upon every subsequent launch of the command prompt you will have the convenience of whatever alias you configure.

Example .cmd file with your configured DOSKEY alias functions:

`@echo off

:: Temporary system path at cmd startup

set PATH=%PATH%;"%USERPROFILE%\Documents\Tools"

:: Commands

DOSKEY ls=dir /O /Q /X DOSKEY alias="C:\Program Files (x86)\Notepad++\notepad++.exe" %USERPROFILE%\Documents\scripts\cmd\alias.cmd DOSKEY loggedon=psloggedon \$ DOSKEY rc=psexec \$ cmd DOSKEY ~=cd "%USERPROFILE%" DOSKEY home=cd "%USERPROFILE%" DOSKEY clear=cls DOSKEY ns=nslookup $*

:: Common directories

DOSKEY docs=cd "%USERPROFILE%\Documents\$*" DOSKEY logs=cd "%USERPROFILE%\Documents\Logs\" DOSKEY tools=cd "%USERPROFILE%\Documents\Tools\" DOSKEY scripts=cd "%USERPROFILE%\Documents\scripts\"`

Example.reg file to add your alias cmd file:

Windows Registry Editor Version 5.00 [HKEY_CURRENT_USER\Software\Microsoft\Command Processor] "AutoRun"="%USERPROFILE%\\Documents\\scripts\\cmd\\alias.cmd"

Ref: Linux alias command

Disclaimer: I hope you find this article useful, but be warned; you should never alter the registry unless you are comfortable doing so. Incorrect registry edits can permanently bork your computer.

[rb-ccie]

In case you ever forget just how awesome the Internet really is!

NTT have kindly put together a fantastic interactive visualisation of the IPv4 Internet from their AS' perspective that makes you feel like you're piloting a deep space mission (or mebbe that's just me?). Fly through the stars Autonomous System & IXP links with the greatest of ease by using the following keys:

W = Move forward.

S = Move backward.

A = Move left.

D = Move right.

Q = Roll right.

E = Roll left.

L = Toggle links (To make it look more like a space scene).

R = Fly up.

F = Fly down.

Up arrow = Rotate up.

Down arrow = Rotate down.

Left arrow = Rotate left.

Right arrow = You should be seeing a pattern by now!

Shift = Move faster.

Space bar = Toggle steering.

Have fun on your flight Cap'n.

[rb-ccie]

Python: Bulk DNS Lookups from a Text File

Bulk IPv4 DNS lookups are pretty common, so here's a Python script to load up hostnames from a text file and execute them. The output can be redirected to a text file for examination later using the > operator.

Pre-reqs

You have a valid, responsive DNS server configured on your NIC resolver.

If you're using single label hostnames rather than FQDNs in your source file, you have valid DNS suffixes in your NIC resolvers suffix search list.

Script

#!/usr/bin/env python # Author: Prometheus - at - rb-cc.ie # dnsl-list.py - Bulk DNS lookups from text file. import socket # Update file path to reflect your source file. file = "data/servers.txt" # Function to verify IP address format. def isIP(address): try: socket.inet_aton(address) ip = True except socket.error: ip = False return ip # Open the file to read, pass to variable then close. f = open(file, 'r') lines = f.readlines() f.close() # Lookup each hostname in turn. for i in lines: host = i.strip() try: ipaddr = socket.gethostbyname(host) if isIP(ipaddr): print("%s - %s" % (host, ipaddr)) except: # Handle error when lookup fails. print("%s - No IP address found." % (host)) pass

Create the source data file:

prometheus@linux-host (~)$ vi data/servers.txt

Press the i key to insert then paste your list of servers in.

host10 host11 host12 host13 host14 host15

Press the Esc key and type :wq! to save and quit.

Usage

prometheus@linux-host (~)$ data/./dnsl-list.py host10 - No IP address found. host11 - No IP address found. host12 - 192.168.42.12 host13 - No IP address found. host14 - 192.168.42.14 host15 - 192.168.42.15

HTH.

[rb-ccie]

bash Scripting: Reverse DNS lookup of multiple hosts from a text file.

It's pretty common to need to lookup multiple hosts by IP address, so I put together a little bash script that will do just that.

Just copy and paste the script below into a Linux prompt (bash) and it will create the reverse lookup script for you in a scripts directory.

Pre-reqs:

You're running the bash shell in your Linux OS.

A valid DNS server is configured in your /etc/resolv.conf file.

Script:

mkdir scripts mkdir data echo '#/bin/bash' > scripts/bulk-rnsl.sh echo '# Author: Prometheus - at - rb-cc.ie' >> scripts/bulk-rnsl.sh echo '# Accept file parameter and set IP address comparison mask.' >> scripts/bulk-rnsl.sh echo 'FILE="$1"' >> scripts/bulk-rnsl.sh echo 'OCTET="(25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])"' >> scripts/bulk-rnsl.sh echo 'IPADDRESSMASK="^$OCTET\.$OCTET\.$OCTET\.$OCTET$"' >> scripts/bulk-rnsl.sh echo '' >> scripts/bulk-rnsl.sh echo '# Verify filetype.' >> scripts/bulk-rnsl.sh echo 'if [[ $FILE == *.txt ]]' >> scripts/bulk-rnsl.sh echo 'then' >> scripts/bulk-rnsl.sh echo ' # Perform a reverse lookup of the IPs in the file' >> scripts/bulk-rnsl.sh echo ' while read line' >> scripts/bulk-rnsl.sh echo ' do' >> scripts/bulk-rnsl.sh echo ' # Verify file contents are in the correct format.' >> scripts/bulk-rnsl.sh echo ' if [[ $line =~ $IPADDRESSMASK ]]' >> scripts/bulk-rnsl.sh echo ' then' >> scripts/bulk-rnsl.sh echo ' # Print status IP address and hostname if found.' >> scripts/bulk-rnsl.sh echo ' STATUS=$(dig -x $line | grep status | cut -f6 -d'"' '"')' >> scripts/bulk-rnsl.sh echo ' RESULT=$(dig -x $line +short)' >> scripts/bulk-rnsl.sh echo ' echo -e "${STATUS}\t${RESULT}\t$line"' >> scripts/bulk-rnsl.sh echo ' else' >> scripts/bulk-rnsl.sh echo ' echo "Not a valid IP address!"' >> scripts/bulk-rnsl.sh echo ' fi' >> scripts/bulk-rnsl.sh echo ' done > scripts/bulk-rnsl.sh echo 'else' >> scripts/bulk-rnsl.sh echo ' echo "Not a valid text file!"' >> scripts/bulk-rnsl.sh echo 'fi' >> scripts/bulk-rnsl.sh chmod 755 scripts/bulk-rnsl.sh echo "alias rev-lookup='scripts/./bulk-rnsl.sh'" >> .bash_profile alias rev-lookup='scripts/./bulk-rnsl.sh'

Creating your data file:

prometheus@linux-host (~)$ nano data/ip-list.txt

Copy and paste the list of IP addresses you want to query into the new file, then press the CTRL + X keys, Y to save and the Enter key to finish.

Usage:

Once complete, you can run the script and pass the filename to it with syntax: rev-lookup <PATH/FILENAME.EXT> and it will tell you if a record is found (NOERROR) or if there is no PTR found (NXDOMAIN).

prometheus@linux-host (~)$ rev-lookup data/ip.txt NXDOMAIN, 192.168.42.10 NXDOMAIN, 192.168.42.11 NOERROR, host12.dns-suffix.tld. 192.168.42.12 NXDOMAIN, 192.168.42.13 NOERROR, host14.dns-suffix.tld. 192.168.42.14 NOERROR, host15.dns-suffix.tld. 192.168.42.15 prometheus@linux-host (~)$

HTH.

[rb-ccie]

Python: Sending a command to multiple devices via SSH

In the event that you wish to run a single command against multiple network devices from a jump box.

NB: You'll need to install the pexpect Python module for this script to work.

Script:

#!/usr/bin/env python # bulk-ssh-cmd.py (2,000 character output limit per device). # Usage: python bulk-ssh-cmd.py 'command string' '/path/file.txt' # Import required module(s). Only pexpect needs to be downloaded and installed, all others should be present. import sys import os import getpass import pexpect # Validate correct number of command line parameters. argList = sys.argv[1:] if len(argList) 2: sys.exit("Please specify a command string AND a valid source file.") # Set RSA key fingerprint expected output string for new connections. cliCMD = sys.argv[1] ssh_newkey = 'Are you sure you want to continue connecting' # Obtain and read in hosts from file specified in CLI and validate filetype and existence. inputFile = sys.argv[2] print inputFile if not inputFile.endswith(".txt"): sys.exit("Error: %s is not a valid text file." % inputFile) elif os.path.exists(inputFile): hostFile = open(inputFile, "r") else: sys.exit("Error: %s file not found." % inputFile) # Get credentials. userID = os.getenv("USER") openSsme = getpass.getpass(prompt="What is your password? \n") # Handle RSA key fingerprint, login prompt and EOF output. for hostBox in hostFile: # Specify connection string & expected responses. connStr=pexpect.spawn('ssh %s@%s' % (userID, hostBox), timeout = 5) resVal=connStr.expect([ssh_newkey,'assword:',pexpect.EOF]) if resVal==0: print "Sending yes to accept RSA key fingerprint." connStr.sendline('yes') resVal=connStr.expect([ssh_newkey,'assword:',pexpect.EOF]) if resVal==1: print "Password sent. Command:", connStr.sendline(openSsme) connStr.expect("#") connStr.sendline(cliCMD) connStr.expect("#") connStr.sendline("exit") print connStr.before , "\n" # Print the command output. else: print "Key or connection timeout received." pass

Data file (can be IP addresses or hostnames):

admin@linux-host (~/scripts)$ cat hosts.txt 192.168.42.5 192.168.42.6 192.168.42.7

Usage:

python bulk-ssh-cmd.py 'command string' '/path/file.txt'

Error handling:

Too few command line parameters:

admin@linux-host (~/scripts)$ python bulk-ssh-cmd.py 'sh ip int br vrf management' Please specify a command string AND a valid source file.

Too many command line parameters:

admin@linux-host (~/scripts)$ python bulk-ssh-cmd.py 'sh ip int br vrf management' '/home/admin/scripts/hostings.txt' test Please specify a command string AND a valid source file.

Invalid file extension:

admin@linux-host (~/scripts)$ python bulk-ssh-cmd.py 'sh ip int br vrf management' '/home/admin/scripts/test.csv' /home/admin/scripts/test.csv Error: /home/admin/scripts/test.csv is not a valid text file.

Non-existant file:

admin@linux-host (~/scripts)$ python bulk-ssh-cmd.py 'sh ip int br vrf management' '/home/admin/scripts/hostings.txt' /home/admin/scripts/hostings.txt Error: /home/admin/scripts/hostings.txt file not found.

NB: This script will work with Cisco IOS & NX-OS as well as Arista EOS. The lines stating connStr.expect("#") will need to be changed to connStr.expect(">") for Juniper JunOS.

[rb-ccie]

ELAM captures on the Cisco 6500.

Sup720#show module 5 Mod Ports Card Type Model Serial No. --- ----- -------------------------------------- ------------------ ----------- 5 5 Supervisor Engine 720 10GE (Active) VS-S720-10G SAL1429N5ST Sup720(config)#service internal Sup720#show platform capture elam asic superman slot 5 Sup720# show platform capture elam trigger dbus ipv4 if ip_sa=10.1.1.100 ip_da=20.1.1.100 Sup720#show platform capture elam start Sup720#show platform capture elam status Active ELAM info: Slot Cpu Asic Inst Ver PB Elam ---- --- -------- ---- --- -- ---- 5 0 ST_SUPER 0 2.2 Y DBUS trigger: FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=10.1.1.100 IP_DA=20.1.1.100 ELAM capture in progress Sup720#show platform capture elam status Active ELAM info: Slot Cpu Asic Inst Ver PB Elam ---- --- -------- ---- --- -- ---- 5 0 ST_SUPER 0 2.2 Y DBUS trigger: FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=10.1.1.100 IP_DA=20.1.1.100 ELAM capture completed

Interpret the Results

Sup720#show platform capture elam data (some output omitted) DBUS: VLAN ............................ [12] = 10 SRC_INDEX ....................... [19] = 0x102 L3_PROTOCOL ..................... [4] = 0 [IPV4] L3_PT ........................... [8] = 1 [ICMP] DMAC ............................ = 0014.f179.b640 SMAC ............................ = 0021.5525.423f IP_TTL .......................... [8] = 255 IP_SA ........................... = 10.1.1.100 IP_DA ........................... = 20.1.1.100 RBUS: FLOOD ........................... [1] = 1 DEST_INDEX ...................... [19] = 0x14 VLAN ............................ [12] = 20 IP_TTL .......................... [8] = 254 REWRITE_INFO i0 - replace bytes from ofs 0 to ofs 11 with seq '00 05 73 A9 55 41 00 14 F1 79 B6 40'.

[rb-ccie]

VLAN SVI sourced telnet and ACLs

Have you ever sourced a telnet from a VLAN SVI and wondered about ACL order of operations when applied to said SVI? There's no doubt that the ability to source from a specific interface is extremely beneficial, but is the switch actually sourcing from the SVI or is it just generating control plane traffic and padding out the source IP with that of the source interface?

Take the following example SVI config (applying the ACL inbound means traffic from the associated VLAN towards the rest of the world):

interface Vlan10 description VLAN10 ip address 10.10.10.1 255.255.255.0 ip access-group VLAN10-ACL in

ACL config is as follows:

SW1#sh ip access-l VLAN10-ACL Extended IP access list VLAN10-ACL 10 permit ip host 10.10.10.10 any (51 matches) 20 permit ip host 10.10.10.11 any (38 matches) 30 deny ip any 22.22.22.0 0.0.0.255 40 permit ip any any (139 matches)

There are no entries permitting traffic from SW1s 10.10.10.1 SVI IP address. Yet I can reach a target IP on another switch when sourcing from that IP address. It doesn't look like the ACL is being processed for this traffic.

SW1#telnet 22.22.22.22 22 /so vlan10 Trying 22.22.22.22, 22 ... Open

I have been all over the ACL articles in the switching software configuration docs on cisco.com and can't find any description of this behaviour. It makes sense to me that this might be control plane traffic sourced from the CPU rather than the SVI, but that brings up two questions to start (more to follow):

Doesn't this negate the reliability of this as a test if you want to verify ACLs without having access to an end host (although its validity remains for connectivity testing)? and

Why isn't this documented anywhere?

If anyone can clarify this for me or point me toward some (seemingly obscure) documentation that I might have missed, I would greatly appreciate it.

[rb-ccie]

One for my US friends: FCC request for comments on locking down Wi-Fi devices.

Apparently, the FCC is considering a proposal to require manufacturers to lock down computing devices (routers, PCs, phones) to prevent modification if they have a "modular wireless radio" or a device with an "electronic label".

Equipment Authorization and Electronic Labeling for Wireless Devices.

In the event, you're an American citizen that would like your say (and you should):

Go to the Federal Register link above before October 9th and press "Submit a formal comment"

Respectfully ask the FCC to not implement rules that take away the ability of users to install the software of their choosing on their computing devices. Additional points of emphasis you should consider adding: Wireless networking research depends on the ability of researchers to investigate and modify their devices. Americans need the ability to fix security holes in their devices when the manufacturer chooses to not do so. Users have in the past fixed serious bugs in their wifi drivers, which would be banned under the NPRM. Billions of dollars of commerce, such as secure wifi vendors, retail hotspot vendors, depends on the ability of users and companies to install the software of their choosing.

Enter your name and address. This is a public comment and your personal information provided will be publicly available.

[rb-ccie]

Perl: Automating emails with sendmail

Say you need to schedule a task to send emails from a Linux based system. Here's a quick & easy script to add a cron job for.

Script (email.pl):

#!/usr/bin/perl # Author: Prometheus @ www.rb-cc.ie # Accept variables from the command line. $RECIPIENT = $ARGV[0]; $SENDER = $ARGV[1]; $SUBJECT = $ARGV[2]; $BODY = $ARGV[3]; $SIGNATURE = $ARGV[4]; # Set email text. $OUTPUT = "To whom it may concern,\n\n$body\n\nRegards,\n\n$name\n$from\nHave a nice day.\n"; # Verify valid email address format of your choosing # and that the subject, body and signature variables are not empty. if ($RECIPIENT !~ /(.*\@rb-cc\.ie)/g){ die("Recipient address invalid.\n"); } if ($SENDER !~ /(.*\@rb-cc\.ie)/g){ die("Sender address invalid.\n"); } if ($RECIPIENT eq ''){ die("Recipient address undefined.\n"); } if ($SENDER eq ''){ die("Sender address undefined.\n"); } if ($SUBJECT eq ''){ die("Subject undefined.\n"); } if ($BODY eq ''){ die("No email text present.\n"); } if ($SIGNATURE eq ''){ die("Signature undefined.\n"); } # Launch sendmail. open(SMAIL, "| /usr/sbin/sendmail -t"); # Pass email parameters to sendmail. print SMAIL "To: $RECIPIENT\n"; print SMAIL "From: $SENDER\n"; print SMAIL "Subject: $SUBJECT\n\n"; # Pass email body to sendmail. print SMAIL $OUTPUT; close(SMAIL);

Usage:

./email.pl test@rb-cc.ie prometheus@rb-cc.ie "Test Email" "The purpose of this email is to advise you that your script works." "Prometheus"

[rb-ccie]

Quantum spooky action at a distance experimentation.

How Quantum "Spookiness" Passes Toughest Test Yet.

It’s a bad day both for Albert Einstein and for hackers. The most rigorous test of quantum theory ever carried out has confirmed that the ‘spooky action at a distance’ that the German physicist famously hated — in which manipulating one object instantaneously seems to affect another, far away one — is an inherent part of the quantum world.

[rb-ccie]

Traceroute aberrant behaviour?

A colleague of mine recently advised a systems admin he was working with that network traffic seemed to be bouncing back and forth between the same switches off the back of a trace route he’d just taken, so I thought I’d do a quick post to clarify what he and I’m sure others were seeing / have seen.

To explain we’ll take a look at traffic from a client in the 10.1.1.0/24 network to a server in the 10.1.6.0/24 network that traverses several layer 3 devices with redundant paths. The topology we’ll be examining is shown above.

Looking at the output below, one could almost be forgiven for thinking that traffic is actually bouncing between R1 and R2:

DSW1#trace 10.1.6.1 Type escape sequence to abort. Tracing the route to 10.1.6.1 1. 10.1.11.1 16 msec 10.1.12.1 8 msec 10.1.11.1 12 msec 2. 10.1.23.1 20 msec 10.1.13.3 32 msec 10.1.23.1 40 msec 3. 10.1.45.1 64 msec 56 msec 40 msec 4. 10.1.6.1 56 msec * 64 msec

So what’s the issue?

If you take a closer look at the output above the clue is in the numbers, the traceroute utility actually tells you how many hops you’re dealing with, but, if that’s the case then why are we seeing multiple IP addresses per hop? A look at the route to Server1 on DSW one should cast a little light on the issue.

DSW1#sh ip ro 10.1.6.0 Routing entry for 10.1.6.0/24 Known via eigrp 100, distance 90, metric 38400, type internal Redistributing via eigrp 100 Last update from 10.1.12.1 on FastEthernet1/2, 00:07:34 ago Routing Descriptor Blocks: * 10.1.11.1, from 10.1.11.1, 00:07:34 ago, via FastEthernet1/1 Route metric is 38400, traffic share count is 1 Total delay is 500 microseconds, minimum bandwidth is 100000 Kbit Reliability 255/255, minimum MTU 1500 bytes Loading 1/255, Hops 4 10.1.12.1, from 10.1.12.1, 00:07:34 ago, via FastEthernet1/2 Route metric is 38400, traffic share count is 1 Total delay is 500 microseconds, minimum bandwidth is 100000 Kbit Reliability 255/255, minimum MTU 1500 bytes Loading 1/255, Hops 4

As you can see, DSW1 has 2 equal cost paths to get to the network Server1 is on.

How / why does traceroute return multiple IPs when there are multiple paths?

The trace route utility sends out 3 UDP packets at a time with incrementing TTLs These UDP probes are process switched packets sourced from router, not actual network traffic. Due to the fact there are multiple paths the first probe is sent via the first path, the second probe via the second path and the third probe via the first path again. For more information you can check out the ever helpful traceroute command on Cisco devices article.

Ultimately, it doesn’t usually matter that there are multiple IPs in the same hop in a traceroute you’ve run from a layer 3 device or which path actual network traffic takes (Unless there’s a firewall in the way, which brings its own packet out-of-state headaches or you have some specific traffic engineering requirements) as long as said traffic reaches to and from the distribution switches connected to our end networks.

[rb-ccie]

Upgrading to new major release to solve that bug.

http://tinyurl.com/on32unt

[rb-ccie]

INEs Brian Man McGahan providing an intro to the CCIE R&S v5.