Monthly Archives: March 2017

Adding Arista vEOS images to UnetLab

Arista provides a number of VM type images that you can use. The one with the software forwarding plane is vEOS-lab. This image has all the control plane features of standard EOS from Arista, with a software dataplane – the downside to this is that features that rely on the hardware e.g. NAT aren’t available.

There are two files required to get a working vEOS switch. The vmdk file and the Aboot ISO. To create a working vEOS on 4.17.5M we need:

  • vEOS-lab-4.17.5M.vmdk
  • Aboot-veos-serial-8.0.0.iso

To create an image that UnetLab will recognise the following steps should turn the trick

  1. Create the directory for the version in question
  2. Copy the two files in there
  3. Convert the vEOS-lab-4.17.5M.vmdk to a qcow image
  4. Rename the Aboot ISO to cdrom.iso file
  5. All done – fire up your Lab and select Arista vEOS as the template
mkdir /opt/unetlab/addons/qemu/veos-4.17.4M/
/opt/qemu/bin/qemu-img convert -f vmdk -O qcow2 ./vEOS-4.17.5M.vmdk hda.qcow2
mv Aboot-veos-serial-8.0.0.iso /opt/unetlab/addons/qemu/veos-4.17.5M/cdrom.iso
rm ./vEOS-4.17.5M.vmdk
/opt/unetlab/wrappers/unl_wrapper -a fixpermissions

 

Painful update of Dell R710 BIOS

As mentioned in the previous post, I recently acquired an R710 as part of a homelab buildout. I’ve now spent the best part of a day trying to figure out why the Lifecycle Controller stopped working, after it updated the BIOS. I don’t particularily need it, but I know if I ignore it something will come back to bite me when I least expect it.

The one clue (after trying multiple Repair images) was that when the BIOS was booting it had a blank rev. e.g. under the Dell logo there was “BIOS Rev. ” and a blank space. Much google-fu later and it turns out that this generally happens if you try and jump too many BIOS revisions at once.

Since the upgrade was done by the Lifecycle Controller, and was from 6.0.4 -> 6.4.0 I didn’t expect this to be an issue, but there you go. The process to recover is actually fairly straightforward when you know how, and in fairness to Dell they have a huge amount of great information in their forums.

This is just the short and quick version of how to fix.

  1. Create a DOS USB disk using the instructions here: https://www.dell.com/support/article/ie/en/iebsdt1/SLN143196/how-to-create-a-bootable-usb-flash-drive-using-dell-diagnostic-deployment-package–dddp-?lang=EN
  2. Download as many revisions of the BIOS between the original release and the target and copy them to the USB drive too from here – you want the non-packaged version.
    14/03/2017  22:24         1,631,701 R710-060007C.exe
    14/03/2017  22:23         1,633,695 R710-060100C.exe
    14/03/2017  22:23         1,650,371 R710-060203C.exe
    14/03/2017  22:24         1,650,583 R710-060300C.exe
    14/03/2017  22:22         1,653,153 R710-060400C.exe

    NOTE: You might want to rename them to fit in DOS’ 8.3 filename limitation unless you like to try and guess what the ~ are hiding

  3. Reboot the server and hit F2 to access the boot options, then choose to boot from the USB stick
  4. When it has loaded, start with the older of the BIOS versions – in my case 6.0.7. If you get a warning about the platform not being recognised add the /forcetype flag after the executable name
  5. Repeat until you are on the BIOS version you were aiming for

At this point the Lifecycle Controller should come up and you can use it for the remainder of your platform/firmware updates

Updating an old firmware on a Dell R710

I recently purchased a Dell R710 as part of a buildout for my homelab, and as part of that process, wanted to update all the software. This consists of;

  • BIOS – running 6.0.4, current is 6.4 so not too bad
  • iDRAC – running 1.10, current is 2.85, so a good leap
  • PERC6 – no idea
  • Lifecycle Controller – no idea

Given that the only OS installed is ESXi I had a bit of googling to figure this one out.

In theory the Lifecycle Controller has an Platform Update option that will just do all this for you… unfortunately this seemed to be missing on mine.

To get everything up and running, the following process worked for me;

  1. Download, and extract the firmimg.d6 file from the iDRAC updater. This will allow your iDRAC to updated to something modern enough not to give certificate/security/browser errors. Use the iDRAC firmware update option to get this installed
  2. Use the iDRAC firmware update page to load the USC/LC repair package (.usc extension)
  3. Reload and the Lifecycle Controller will now have a Platform Update option – use that to update everything else via FTP from dells website

NOTE: The Lifecycle controller uses the inbuilt NIC rather then the iDRAC ethernet interface

Some PTP Essentials

This entry is part 2 of 2 in the series Precision Time Protocol

IEEE 1588 aims at creating a master-slave timing distribution model across PTP clocks. In IEEE 1588, a clock is a node that implements PTP. In that node, a “PTP port” is a physical, logical, or virtual interface supporting PTP and handling PTP communications. To define the state or function of the port, IEEE 1588 defines state machines as part of its Best Master Clock Algorithm [BMCA] process.

Best Master Clock Algorithm

The BMCA actually has two key functions in a clock. Based on the information received on its PTP Port(s), the BMCA of a clock will:

  • define the master-slave hierarchy of the PTP distribution
  • select the best master (the clock source that it will synchronise to)

NOTE: Be aware of the distinction between the port state and the master clock selection, though both are handled as part of BMCA.

The BMCA is designed in such a way that a local clock determines which of all the clocks (including itself) is the “best”, then it decides on the state of all the ports (master/slave/passive). Clocks do not negotiate who should be master or slave – instead each computes the state of it’s own ports. The algorithm, by design, avoids configurations with two masters, no masters, or ones that thrash.

 

PTP ports can be in 1 of 3 states (the above diagram only considers GM, Slave, and intermediate port states. The role and type of the intermediate clocks isn’t considered yet, simply assume that they run the BMCA);

  1. Master – providing timing information to downstream clocks
  2. Slave – learning timing from an upstream (master) clock
  3. Passive – preventing a loop in the PTP topology

The master port of a clock transmits its packet-based clock signal in the form of PTP messages to the slave port or another clock. Slave ports allow a clock to synchronise with the root of the hierarchy, the ultimate master clock e.g. the Grandmaster [GM] clock. A passive port prevents timing loops or conflicting master ports in the same hierarchy, when there are multiple valid transport network paths (consider an L3 network – no ports blocked due to STP). The state of the port, and the selected clock source (best master or master port) can change over time.

In a steady state network, the BMCA processed by the clocks establishes a hierarchy up to the root, i.e. a GM, and ultimately to slave clocks, the leaves of the hierarchy.

The BMCA will avoid creating timing loops through valid transport network paths e.g. ports that are not blocked by spanning tree. In the case of any changes e.g. device or link failure, the BMCA might have to re-establish the hierarchy and/or reselect the master.

How do we select the “best” clock?

The best master clock algorithm compares one clock to another by comparing datasets (my dataset, vs current best dataset, vs. the dataset just received) that represent those clocks. The table below shows the parameters used to compare any two PTP clocks.

GM Priority 1 Configurable
GM Class Like NTP Stratum
GM Accuracy Atomic Clock, GPS, NTP, Hand set, Internal Oscillator, etc.
GM  offsetScaledLogVariance Provided info from GM regarding PTP variance due to temp, etc.
GM Priority 2 Configurable
GM Identity Usually based on the MAC Address
Steps Removed Number of hops to GM. 0 for local clock
Identity of Senders Identity of parent clock. Same as self for single clock
Identity of Receiver Own Identity
Port number of the receiver Own port number

GM selection can be manipulated using Priority1 and Priority2 values. Typically a true Grand Master device (Meinberg, etc.) will have a low Priority1, with the Priority2 value used to select a preferred active Grand Master device.

We would expect devices with a stable clock (for example the -CL variant of Arista’s 7150s), close to the GMs in the hierarchy to have an intermediate Priority1 and 2 values e.g. 127 – this way they would only become the GM if both “true” GM devices fail.

Slave devices (servers, etc.) would have the highest Priority value configured e.g. 255 to ensure that they never become the GM for the timing domain.

PTP Domains

IEEE 1588 defines the idea of a PTP Domain. These domains can be used for different purposes, e.g. technical or administrative, at the convenience of the user.  The standard defines the protocol and rules but does not define usage. A PTP domain corresponds to one GM delivering its clock signal over the hierarchy established for that domain. Simply put, a PTP domain corresponds to a unique hierarchy of two or more clocks synchronised to the same GM.

The spec allows a clock device to support multiple domains, but a PTP port within that device can only be part of a single domain. Logically, because a PTP communication path is established between two PTP ports, a PTP path belongs to one PTP domain and can never span multiple PTP domains.

A transport network supports multiple PTP domains and a physical network interface can convey multiple PTP paths depending on vendor implementation. Remember, a PTP port is a logical description that can be translated into a physical or virtual interface in a clock implementation. The specification allows for a PTP port to be assigned to a physical port, a VLAN or a loopback interface.

A network or a clock device can support multiple domains on the condition that each domain remains separate. This separation can be physical or logical, as distinct communication networks, VLANs or clocks, and could lead to more complex and expensive design implementation, particularly for maintaining different timescales in single clocks.

Introduction to Precision Time Protocol (IEEE 1588)

This entry is part 1 of 2 in the series Precision Time Protocol

This is the first part of a multi-part series that will cover the basic concepts, terminology, behaviours, and applications of PTP.

PTP Goals

PTP was originally defined in the IEEE 1588-2002 standard, officially entitled “Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems” and published in 2002. In 2008, IEEE 1588-2008 was released as a revised standard; also known as PTP Version 2 [PTPv2], it improves accuracy, precision and robustness but is not backward compatible with the original 2002 version.

IEEE 1588 was designed to fill a niche that neither NTP nor GPS could fulfill by solving two problems:

  1. Local systems needing clock accuracy in the sub-microsecond range
  2. Systems needing synchronisation in areas that cannot afford the cost of a GPS receiver at each node, or that cannot receive GPS signals (consider an underground or windowless datacenter)

The origins of IEEE 1588 development come from test and measurement (instrumentation and data acquisition), industrial automation or power generation, and received strong support from the automation industry leading to the adoption of the IEEE 1588-2002 standard by the Industrial Electro-technical Commission [IEC] as IEEE/IEC 61588-2004.

Specific application domains and targets had driven first release of IEEE 1588 work. Applications (e.g. data acquisition, military instrumentation, industrial robots and high-speed printers) and their targets, had sufficient shared objective to define common criteria:

  • Achievable sub-microsecond accuracy (if required).
  • Operation over control localised areas, such as local area network [LAN], with no intent to cover Internet or wide area networks, but supporting a PTP system.
  • Administration-free operation allowing “plug-and-play” deployment of IEEE 1588 clocks
  • Designed for minimal resource requirement in devices supporting PTP (clock filtering or averaging might not be necessary).

Currently one of the main industries using PTP is the High-Frequency Trading [HFT], and financial sector, to ensure highly accurate timing and synchronisation for their market feeds and trading servers.

When reading the spec

Just to note, that as an IEEE spec, 1588 can be far more difficult to read than IETF/RFC documents. A key thing to note in this particular spec is that IEEE 1588 also specifies “the node, system, and communications properties necessary to support PTP” and proposes models of devices, called “clocks” that support the protocol and aim to help and enhance the transfer of packet-based timing references. In IEEE 1588 terms, a PTP system is a set of IEEE 1588 clocks with specialised functions to support the protocol, PTP.

For instance, specifications of an IEEE 1588 clock include some material, such as a port, that is not part of the protocol itself. However, the messages sent and received by each port of an IEEE 1588 clock and the resulting behaviour of that clock is part of the protocol. Nonetheless, the standard does not define any performance objective for transport network device or clock.

To simplify – a PTP clock is a device that runs the PTP protocol, and has one or more network interfaces