Deploying vCenter High Availability with network addresses in separate subnets

VMware vCenter High Availability is a very interesting feature included in vSphere 6.5. Generally, it provides higher availability of vCenter service by having three vCenter nodes (active/passive/witness) all serving the single vCenter service.

This is written in the official vCenter HA documentation

vCenter High Availability (vCenter HA) protects vCenter Server Appliance against host and hardware failures. The active-passive architecture of the solution can also help you reduce downtime significantly when you patch vCenter Server Appliance.
After some network configuration, you create a three-node cluster that contains Active, Passive, and Witness nodes. Different configuration paths are available.   

The last sentence is very true. The simplest VCHA deployment is within the same SSO domain and within the single datacenter with two Layer2 networks, one for management and second for the heartbeat. Such design can be deployed in fully automated manner and you just need to provide dedicated network (portgroup/VLAN) for the heartbeat network and use 3 IP addresses from separated heartbeat subnet. Easy. But is it what you are expecting from vCenter HA? To be honest, the much more attractive use case is to spread vCenter HA nodes across three datacenters to keep vSphere management up and running even one of two datacenters experiences some issue. Conceptually it is depicted in the figure below.

Conceptual vCenter HA Design

In this particular concept, I have embedded PSC controllers because of simplicity and vCenter HA can increase availability even of PSC services. The most interesting challenge in this concept is networking so let's look into the intended network logical design.

vCenter HA - networking logical design

Networking logical design:

• Each vCenter Server Appliance node has two NICs
• One NIC is connected to management network and second NIC to heartbeat network
• Layer 2 Management network (VLAN 4) is stretched across datacenters A and B because vCenter IP address must work without human intervention in datacenter B after VCHA fail-over.
• In each datacenter we have independent heartbeat network (VCHA-HB-A, VCHA-HB-B, VCHA-HB-C) with different IP subnets to not stretch Layer 2 across datacenters, especially not to datacenter C where is the witness. This requires specific static routes in each vCenter Server Appliance node to have IP reachability over heartbeat network.
• Specific VCHA network tcp/udp ports must be allowed among VCHA nodes across a heartbeat network.

Helpful documents:

• FAQ: vCenter High Availability (2148003) - https://kb.vmware.com/s/article/2148003
• Deploying vCenter High Availability with network addresses in separate subnets (2148442) - https://kb.vmware.com/kb/2148442
• Required ports for VCHA 6.5 (52835) - https://kb.vmware.com/kb/52835
• Errors "Failed to ssh connect peer node x.x.x.x" and "sshConnect Authentication (publickey) failed" while configuring vCenter HA (2150715) - https://kb.vmware.com/kb/2150715

Implementation Notes: 

Note 1:

VMware KB 2148442 (Deploying vCenter High Availability with network addresses in separate subnets) is very important to deploy such design but one information is missing there. After cloning of vCenter Server Appliances, you have to go to passive node and configure on eth0 the same IP address you use in active node. Configuration is in file  /etc/systemd/network/10-eth0.network.manual

Note 2:

In case of badly destroyed VCHA cluster use following commands to destroy VCHA from the command line

cd /etc/systemd/network 

mv 10-eth0.network.manual 20-eth0.networkdestroy-vchareboot

The solution was found at https://communities.vmware.com/thread/552084

Link to the official documentation (Resolving Failover Failures) - https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.avail.doc/GUID-FE5106A8-5FE7-4C38-91AA-D7140944002D.html

Note 3:

In case, you will see the error message MethodFault.summary error during the finalization process it is because a hostname mismatch is detected. The hostname assigned to the Passive node must be the same as the hostname of the Active node. The solution was found  at https://www.altaro.com/vmware/how-to-deploy-a-vcenter-ha-cluster-part-2/ but also written in KB https://kb.vmware.com/kb/2148442

How to check I/O device on VMware HCL

VMware has Hardware Compatibility List of supported I/O devices is available here
https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

VMware HCL for I/O devices

The best identification of I/O device is VID (Vendor ID), DID (Device ID), SVID (Sub-Vendor ID), SSID (Sub-Device ID). VID, DID, SVID and SSID can be simply entered into VMware HCL and you will find if it is supported and what capabilities have been tested. You can also find supported firmware and driver.

The get these identifiers you have to log in to ESXi via SSH and use command "vmkchdev -l".  This command shows VID:DID SVID:SSID for PCI devices and you can use grep to filter just VMware NICs (aka vmnic)

vmkchdev -l | grep vmnic

You should get similar output

[dpasek@esx01:~] vmkchdev -l | grep vmnic

0000:02:00.0 14e4:1657 103c:22be vmkernel vmnic0

0000:02:00.1 14e4:1657 103c:22be vmkernel vmnic1

0000:02:00.2 14e4:1657 103c:22be vmkernel vmnic2

0000:02:00.3 14e4:1657 103c:22be vmkernel vmnic3

0000:05:00.0 14e4:168e 103c:339d vmkernel vmnic4

0000:05:00.1 14e4:168e 103c:339d vmkernel vmnic5

0000:88:00.0 14e4:168e 103c:339d vmkernel vmnic6

0000:88:00.1 14e4:168e 103c:339d vmkernel vmnic7

So, in case of vmknic4 there is
·       VID:DID SVID:SSID
·       14e4:168e 103c:339d


The same applies to HBAs and disk controllers.  For HBA and local disk controllers use

vmkchdev -l | grep vmhba

This is the output from my Intel NUC at my home lab

[root@esx02:~] vmkchdev -l | more

0000:00:00.0 8086:0a04 8086:2054 vmkernel 

0000:00:02.0 8086:0a26 8086:2054 vmkernel 

0000:00:03.0 8086:0a0c 8086:2054 vmkernel 

0000:00:14.0 8086:9c31 8086:2054 vmkernel vmhba32

0000:00:16.0 8086:9c3a 8086:2054 vmkernel 

0000:00:19.0 8086:1559 8086:2054 vmkernel vmnic0

0000:00:1b.0 8086:9c20 8086:2054 vmkernel 

0000:00:1d.0 8086:9c26 8086:2054 vmkernel 

0000:00:1f.0 8086:9c43 8086:2054 vmkernel 

0000:00:1f.2 8086:9c03 8086:2054 vmkernel vmhba0

0000:00:1f.3 8086:9c22 8086:2054 vmkernel 

vmhba0 is local disk controller
vmhba32 is USB storage controller
vmnic0 is network interface

Hope this helps.

Storage QoS with vSphere virtual disk IOPS limits

I'm a long time protagonist of storage QoS applied per each VM virtual disk (aka vDisk). In the past, vSphere virtual disk shares and IOPS limits were the only solutions. Nowadays, there are new architectural options - vSphere virtual disk reservations and VVols QoS. Anyway, whatever option you will decide to use, the reason to use QoS (IOPS limits) is the architecture of all modern shared storages. Let's dig a little bit deeper. In modern storage systems, you usually have a single pool of a large number of disks and volumes (aka LUNs) created on top of this pool. This modern storage array approach has a lot of advantages but the key advantage is the positive impact on performance because single LUN is spread across a large number of disks which multiplies available performance. However, you usually have multiple LUNs and all these LUNs resides on the same disk pool so it sits on the same disks, therefore all these LUNs interfere with each other from the performance point of view. Even you would have a single LUN you would have multiple VMs per single LUN (aka VMFS datastore) therefore VMs would affect each other.

Design consideration

What it all means? Well, if you have a typical vSphere and storage configuration depicted in the figure below, single VM can use all performance from the underlying storage system (disk pool) and I/O workload from one VM impacts another VM. We call this situation "noisy neighbor" where one VM has significantly higher storage traffic than others.

Modern Storage System Architecture - Disk Pool

The problem with such design is not only performance interference but the fact, that performance is unpredictable. VMs are getting excellent performance when storage is NOT overloaded but very poor performance during peak times.  From use experience point of view is better to limit VMs to decrease the difference between BEST and WORST performance.

How to design your infrastructure to keep your storage performance predictable? The short answer is to implement QoS. Of course, there is the longer answer.

Storage QoS

vSphere Storage QoS is called Storage I/O Control or simply SIOC. Storage-based per LUN QoS will not help you on VMFS because multiple VM virtual disks (vDisks) are accomodated on single LUN (datastore), however, all modern storages support Virtual Volumes (aka VVols) which is VMware framework how to manage vDisks directly on storage but the implementation is always specific to particular storage vendor. Nevertheless, in general, VVols can do QoS per vDisk (aka VMDK) as each vDisk is represented on storage as independent volume (aka LUN or better say sub-LUN or micro-LUN).

vSphere Storage QoS (SIOC) supports

• IOPS Limits
• IOPS Reservations
• IOPS Shares

Storage I/O Control (SIOC) was initially introduced in vSphere 4.1. Nowadays it is called SIOC V1. vSphere 6.5 has introduced SIOC V2.

vSphere SIOC V1

In case you want to use vSphere SIOC V1 storage QoS, it is good to know, at least some, SIOC V1 "IOPS limit" implementation details

• I/Os are normalized at 32KB so 128KB I/O is counted as 4 I/Os. This is vSphere default which can be changed by advanced parameter Disk.SchedCostUnit (default value is 32768 bytes) to satisfy different customer's specific requirements. Disk.SchedCostUnit is configurable and allowable values range between 4K to 256K.
• In the latest ESXi version  ESXi 6.5 build 7526125) IOPS limit does not interfere with SVmotion (Storage vMotion). However, this is different behavior than in previous ESXi versions where storage vMotion was billed to the VM.
• If VM has multiple disks with IOPS limits on the same datastore (LUN) then all limits are aggregated and enforced on such datastore per the whole VM. If vDisks of single VM are located on different datastores then limits are enforced independently per each datastore. The behavior is different on NFS datastores. All this behavior is explained in VMware KB "Limiting disk I/O from a specific virtual machine" - https://kb.vmware.com/kb/1038241

vSphere SIOC V2

It is worth to mention that SIOC V1 and SIOC V2 can co-exist. SIOC V2 is very different when compared to V1.

SIOC V2 implementation details

• I/Os are NOT normalized at static I/O size like SIOC V1. In other words, SIOC V2 does not have Disk.SchedCostUnit implemented.
• SIOCv2 is implemented using IO Filter framework and is managed by using SPBM Policies. 

VVols QoS

VVols QoS is out of the scope of this article.


Hope it is informative but as always do not hesitate to contact me for further details or discussions.

ESXi - optical diagnostic information from VMNIC

I have been asked by one customer how to get optical diagnostic information from NIC which is Intel X540 10GbE Controller. NIC was identified by ESXi host as vmnic6 and more info about VNMIC can show command

esxcli network nic get -n vmnic6 

 esxcli network nic get -n vmnic6:  
  Advertised Auto Negotiation: true  
   Advertised Link Modes: 1000baseT/Full, 10000baseT/Full  
   Auto Negotiation: true  
   Cable Type: FIBRE  
   Current Message Level: 7  
   Driver Info:  
      Bus Info: 0000:03:00.0  
      Driver: ixgbe  
      Firmware Version: 0x800008fd, 18.0.16  
      Version: 4.4.1-iov  
   Link Detected: true  
   Link Status: Up  
   Name: vmnic6  
   PHYAddress: 0  
   Pause Autonegotiate: false  
   Pause RX: true  
   Pause TX: true  
   Supported Ports: FIBRE  
   Supports Auto Negotiation: true  
   Supports Pause: true  
   Supports Wakeon: false  
   Transceiver: external  
   Virtual Address: 00:50:56:5c:dd:9a  
   Wakeon: None  

On ESXi host optics diagnostic is not implemented. I have found that ethtool in linux has the option -m which displays optic diagnostic information but ESXi ethtool do not support it. Monitoring Switch Port SFP/QSFP Hardware Information Using ethtool in Cumulus linux is described here.

During the research, I have found that linux -m option reads these diagnostic information from EEPROM.  On ESXi host, we can dump EEPROM by following esxcli command.

 esxcli network nic eeprom dump -l 1048560 -o 0 -n vmnic6  

Here is the EEPROM dump from Intel X520 NIC

 Offset Value                        
 ------ ------------------------------------------------  
 0000  61 05 ea ff 00 00 40 00 42 80 60 81 3f 02 55 02   
 0010  5b 02 61 02 69 02 71 02 74 02 77 02 87 02 82 80   
 0020  00 05 ff ff ff ff ff ff ff ff fa fa 2c 06 2d 03   
 0030  40 40 30 40 ff ff ff ff ff ff ff ff ff ff ff ff   
 0040  32 06 ff ff ff ff ff ff ff ff ff ff ff ff 23 06   
 0050  1d 06 51 40 46 00 ff ff f5 ff a0 05 00 80 a7 02   
 0060  20 01 00 40 5a 23 3b 40 21 01 00 40 0d 31 27 03   
 0070  4c 01 ff ff ff ff ff ff ff ff ff ff 02 80 ea 1a   
 0080  fe 01 00 70 06 06 00 7a c0 c0 62 70 02 02 63 70   
 0090  de de 64 70 02 02 66 70 11 11 67 70 90 90 68 70   
 00a0  2c 2c 69 70 78 78 6a 70 40 40 74 70 01 01 06 72   
 00b0  00 00 06 74 00 00 06 76 00 00 06 78 00 00 12 72   
 00c0  e2 e2 12 74 e2 e2 12 76 e2 e2 12 78 e2 e2 13 72   
 00d0  36 36 13 74 36 36 13 76 36 36 13 78 36 36 14 72   
 00e0  98 98 14 74 98 98 14 76 98 98 14 78 98 98 15 72   
 00f0  1c 1c 15 74 1c 1c 15 76 1c 1c 15 78 1c 1c 18 72   
 0100  f2 f2 18 74 f2 f2 18 76 f2 f2 18 78 f2 f2 1b 72   
 0110  0a 0a 1b 74 0a 0a 1b 76 0a 0a 1b 78 0a 0a 1c 72   
 0120  0c 0c 1c 74 0c 0c 1c 76 0c 0c 1c 78 0c 0c 2c 72   
 0130  02 02 2c 74 02 02 2c 76 02 02 2c 78 02 02 5a 72   
 0140  02 02 5a 74 02 02 5a 76 02 02 5a 78 02 02 5b 72   
 0150  02 02 5b 74 02 02 5b 76 02 02 5b 78 02 02 5c 72   
 0160  00 00 5c 74 00 00 5c 76 00 00 5c 78 00 00 5e 72   
 0170  03 03 5e 74 03 03 5e 76 03 03 5e 78 03 03 01 7a   
 0180  98 98 09 7a 22 22 0a 7a 1d 1d 0c 7a 02 02 0f 7a   
 0190  98 98 10 7a 19 19 40 7a 61 61 7f 7a 00 00 80 7a   
 01a0  00 00 84 7a 00 00 85 7a 00 00 86 7a 00 00 87 7a   
 01b0  60 60 88 7a 61 61 89 7a 61 61 8a 7a 61 61 8b 7a   
 01c0  61 61 8c 7a 61 61 8d 7a 61 61 8e 7a 11 11 8f 7a   
 01d0  00 00 90 7a 30 30 91 7a 00 00 92 7a ac ac 93 7a   
 01e0  aa aa 94 7a aa aa 95 7a aa aa 96 7a 00 00 97 7a   
 01f0  00 00 98 7a 11 11 a2 7a 00 00 a3 7a 40 40 a4 7a   
 0200  ad ad a5 7a 60 60 a6 7a a9 a9 a7 7a 5a 5a a8 7a   
 0210  95 95 a9 7a aa aa aa 7a 55 55 ab 7a 01 01 ac 7a   
 0220  f8 f8 ad 7a 03 03 ae 7a 00 00 af 7a 09 09 b0 7a   
 0230  09 09 b1 7a 00 00 b2 7a 00 00 b3 7a 00 00 b4 7a   
 0240  80 80 b5 7a 07 07 b6 7a 40 40 b7 7a 10 10 b8 7a   
 0250  04 04 b9 7a 01 01 bf 7a ff ff c0 7a a3 a3 c1 7a   
 0260  7e 7e 52 72 06 06 52 74 06 06 52 76 06 06 52 78   
 0270  06 06 53 72 1b 1b 53 74 1b 1b 53 76 1b 1b 53 78   
 0280  1b 1b 54 72 1e 1e 54 74 1e 1e 54 76 1e 1e 54 78   
 0290  1e 1e 55 72 53 53 55 74 53 53 55 76 53 53 55 78   
 02a0  53 53 56 72 31 31 56 74 31 31 56 76 31 31 56 78   
 02b0  31 31 57 72 1b 1b 57 74 1b 1b 57 76 1b 1b 57 78   
 02c0  1b 1b 51 72 0c 0c 51 74 0c 0c 51 76 0c 0c 51 78   
 02d0  0c 0c 00 70 07 07 90 00 ff 00 00 00 00 00 00 00   
 02e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 02f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0300  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0310  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0320  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0330  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0340  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0350  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0360  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0370  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0380  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0390  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 03a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 03b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 03c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 03d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 03e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 03f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0400  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0410  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0420  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0430  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0440  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0450  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0460  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0470  00 00 00 00 00 00 00 7a df df 00 00 00 00 15 00   
 0480  3b 3f ba 81 07 00 3f 00 14 00 00 00 07 4e 61 1f   
 0490  28 10 a6 10 02 00 f7 17 15 15 35 00 64 6e 01 00   
 04a0  24 6e 96 72 78 18 e2 ea 07 00 05 00 3f 06 28 15   
 04b0  00 00 04 14 00 00 05 00 3f 16 28 15 00 00 04 14   
 04c0  00 00 07 00 24 6e 96 72 78 18 06 01 82 05 00 00   
 04d0  01 00 07 00 24 6e 96 72 78 1a 06 01 82 05 00 00   
 04e0  01 00 02 00 40 0f 00 00 02 00 40 0f 00 00 0f 00   
 04f0  44 04 00 00 80 00 00 03 ff ff ff ff 00 03 ff ff   
 0500  ff ff 00 03 ff ff ff ff 00 03 ff ff ff ff 0f 00   
 0510  44 04 00 00 80 00 00 03 ff ff ff ff 00 03 ff ff   
 0520  ff ff 00 03 ff ff ff ff 00 03 ff ff ff ff 10 3d   
 0530  20 00 12 ff c6 c4 00 00 05 03 00 03 00 00 00 c8   
 0540  00 00 00 00 00 00 00 00 00 00 a2 02 00 02 82 15   
 0550  00 58 35 34 30 20 31 30 47 62 45 20 43 6f 6e 74   
 0560  72 6f 6c 6c 65 72 90 79 00 56 30 0a 46 46 56 31   
 0570  38 2e 30 2e 31 36 50 4e 06 47 31 34 38 34 33 4d   
 0580  4e 04 31 30 32 38 56 31 12 44 53 56 31 30 32 38   
 0590  56 50 44 52 2e 56 45 52 31 2e 30 56 33 06 44 54   
 05a0  49 4e 49 43 56 34 17 44 43 4d 31 30 30 31 30 30   
 05b0  38 37 44 35 32 31 30 31 30 30 38 37 44 35 56 35   
 05c0  04 4e 50 59 32 56 36 04 50 4d 54 31 56 37 0d 4e   
 05d0  4d 56 49 6e 74 65 6c 20 43 6f 72 70 52 56 03 39   
 05e0  00 00 78 00 ff ff ff ff ff ff ff ff ff ff ff ff   
 05f0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0600  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0610  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0620  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0630  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0640  ff ff ff ff ff ff ff ff ff ff ff ff ff ff 24 6e   
 0650  96 72 78 18 24 6e 96 72 78 1a 69 53 e0 05 01 00   
 0660  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0670  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0680  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0690  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 06a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 06b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 06c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 06d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 06e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 06f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0700  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0710  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0720  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0730  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0740  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0750  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0760  00 12 10 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0770  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0780  00 00 03 3d 00 00 00 00 00 00 00 00 00 00 00 00   
 0790  00 00 00 00 00 00 bc 0c 00 00 00 00 00 00 00 00   
 07a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 07b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 07c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 07d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 07e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 07f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0800  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0810  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0820  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0830  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0840  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0850  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0860  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0870  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0880  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0890  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 08a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 08b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 08c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 08d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 08e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 08f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0900  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0910  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0920  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0930  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0940  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0950  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0960  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0970  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0980  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0990  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 09a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 09b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 09c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 09d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 3d   
 09e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 09f0  00 00 bc 0c 00 00 00 00 00 00 00 00 00 00 00 00   
 0a00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a10  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a20  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a30  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a40  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a50  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a60  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a70  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a80  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0a90  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0aa0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0ab0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0ac0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0ad0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0ae0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0af0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b10  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b20  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b30  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b40  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b50  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b60  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b70  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b80  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0b90  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0ba0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0bb0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0bc0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0bd0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0be0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0bf0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0c00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0c10  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0c20  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
 0c30  00 00 00 00 00 00 00 00 00 00 24 6e 96 72 78 19   
 0c40  24 6e 96 72 78 1b 03 00 24 6e 96 72 78 19 24 6e   
 0c50  96 72 78 1b 00 20 01 20 06 00 35 48 30 34 39 34   
 0c60  30 2d 35 30 56 4c 01 08 00 00 00 00 ff ff ff ff   
 0c70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0c80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0c90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ca0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0cb0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0cc0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0cd0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ce0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0cf0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d00  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d10  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d20  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d30  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d40  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d50  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d60  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0d90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0da0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0db0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0dc0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0dd0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0de0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0df0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e00  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e10  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e20  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e30  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e40  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e50  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e60  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0e90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ea0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0eb0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ec0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ed0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ee0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ef0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f00  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f10  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f20  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f30  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f40  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f50  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f60  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0f90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0fa0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0fb0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0fc0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0fd0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0fe0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
 0ff0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  

Memory maps are described in specification SFF-8472 available on multiple places around the Internet.

• Digital Diagnostic Monitoring Interface for SFP and SFP+ Optical Transceivers at https://www.finisar.com/sites/default/files/resources/an-2030_ddmi_for_sfp_rev_e2-20140404_updated.pdf
• SFF-8472 Specification for Diagnostic Monitoring Interface for Optical Transceivers is available at http://www.10gtek.com/templates/wzten/pdf/SFF-8472-(Diagnostic%20Monitoring%20Interface).pdf

Two 256 bytes memory maps are described in SFF-8472 Specification but the problem is how to locate these maps in EEPROM dump which has, in this particular case (Intel X520) 4,080 bytes.

I have prepared simple Perl script to convert hex numbers into ASCII characters and look what text is available in the dump. Perl script is here

 #!/usr/bin/perl  
 print "EEPROM view\n";  
 my $filename = 'eeprom.dump';  
 open(my $fh, '<', $filename) or die "Could not open file '$filename' $!";  
 $line = 0;  
 while (my $row = <$fh>) {  
  $line++;  
  chomp $row;  
  if ($line>2) {  
   print "$row\n";  
   $data_str = substr($row, 7, 48);  
   @data = split(" ",$data_str);  
   print "    ";  
   foreach $b_str (@data) {  
    $b = hex $b_str;  
    $char = chr($b);  
    if ( ($b>=32) and ($b<=126) ) {  
     print $char;  
     print " ";  
    } else {  
     print "- ";  
    }  
   }   
   print "\n";  
  }   
 }  

... and dump with ASCII translation is here ...

 EEPROM view  
 0000  61 05 ea ff 00 00 40 00 42 80 60 81 3f 02 55 02   
     a - - - - - @ - B - ` - ? - U -   
 0010  5b 02 61 02 69 02 71 02 74 02 77 02 87 02 82 80   
     [ - a - i - q - t - w - - - - -   
 0020  00 05 ff ff ff ff ff ff ff ff fa fa 2c 06 2d 03   
     - - - - - - - - - - - - , - - -   
 0030  40 40 30 40 ff ff ff ff ff ff ff ff ff ff ff ff   
     @ @ 0 @ - - - - - - - - - - - -   
 0040  32 06 ff ff ff ff ff ff ff ff ff ff ff ff 23 06   
     2 - - - - - - - - - - - - - # -   
 0050  1d 06 51 40 46 00 ff ff f5 ff a0 05 00 80 a7 02   
     - - Q @ F - - - - - - - - - - -   
 0060  20 01 00 40 5a 23 3b 40 21 01 00 40 0d 31 27 03   
       - - @ Z # ; @ ! - - @ - 1 ' -   
 0070  4c 01 ff ff ff ff ff ff ff ff ff ff 02 80 ea 1a   
     L - - - - - - - - - - - - - - -   
 0080  fe 01 00 70 06 06 00 7a c0 c0 62 70 02 02 63 70   
     - - - p - - - z - - b p - - c p   
 0090  de de 64 70 02 02 66 70 11 11 67 70 90 90 68 70   
     - - d p - - f p - - g p - - h p   
 00a0  2c 2c 69 70 78 78 6a 70 40 40 74 70 01 01 06 72   
     , , i p x x j p @ @ t p - - - r   
 00b0  00 00 06 74 00 00 06 76 00 00 06 78 00 00 12 72   
     - - - t - - - v - - - x - - - r   
 00c0  e2 e2 12 74 e2 e2 12 76 e2 e2 12 78 e2 e2 13 72   
     - - - t - - - v - - - x - - - r   
 00d0  36 36 13 74 36 36 13 76 36 36 13 78 36 36 14 72   
     6 6 - t 6 6 - v 6 6 - x 6 6 - r   
 00e0  98 98 14 74 98 98 14 76 98 98 14 78 98 98 15 72   
     - - - t - - - v - - - x - - - r   
 00f0  1c 1c 15 74 1c 1c 15 76 1c 1c 15 78 1c 1c 18 72   
     - - - t - - - v - - - x - - - r   
 0100  f2 f2 18 74 f2 f2 18 76 f2 f2 18 78 f2 f2 1b 72   
     - - - t - - - v - - - x - - - r   
 0110  0a 0a 1b 74 0a 0a 1b 76 0a 0a 1b 78 0a 0a 1c 72   
     - - - t - - - v - - - x - - - r   
 0120  0c 0c 1c 74 0c 0c 1c 76 0c 0c 1c 78 0c 0c 2c 72   
     - - - t - - - v - - - x - - , r   
 0130  02 02 2c 74 02 02 2c 76 02 02 2c 78 02 02 5a 72   
     - - , t - - , v - - , x - - Z r   
 0140  02 02 5a 74 02 02 5a 76 02 02 5a 78 02 02 5b 72   
     - - Z t - - Z v - - Z x - - [ r   
 0150  02 02 5b 74 02 02 5b 76 02 02 5b 78 02 02 5c 72   
     - - [ t - - [ v - - [ x - - \ r   
 0160  00 00 5c 74 00 00 5c 76 00 00 5c 78 00 00 5e 72   
     - - \ t - - \ v - - \ x - - ^ r   
 0170  03 03 5e 74 03 03 5e 76 03 03 5e 78 03 03 01 7a   
     - - ^ t - - ^ v - - ^ x - - - z   
 0180  98 98 09 7a 22 22 0a 7a 1d 1d 0c 7a 02 02 0f 7a   
     - - - z " " - z - - - z - - - z   
 0190  98 98 10 7a 19 19 40 7a 61 61 7f 7a 00 00 80 7a   
     - - - z - - @ z a a - z - - - z   
 01a0  00 00 84 7a 00 00 85 7a 00 00 86 7a 00 00 87 7a   
     - - - z - - - z - - - z - - - z   
 01b0  60 60 88 7a 61 61 89 7a 61 61 8a 7a 61 61 8b 7a   
     ` ` - z a a - z a a - z a a - z   
 01c0  61 61 8c 7a 61 61 8d 7a 61 61 8e 7a 11 11 8f 7a   
     a a - z a a - z a a - z - - - z   
 01d0  00 00 90 7a 30 30 91 7a 00 00 92 7a ac ac 93 7a   
     - - - z 0 0 - z - - - z - - - z   
 01e0  aa aa 94 7a aa aa 95 7a aa aa 96 7a 00 00 97 7a   
     - - - z - - - z - - - z - - - z   
 01f0  00 00 98 7a 11 11 a2 7a 00 00 a3 7a 40 40 a4 7a   
     - - - z - - - z - - - z @ @ - z   
 0200  ad ad a5 7a 60 60 a6 7a a9 a9 a7 7a 5a 5a a8 7a   
     - - - z ` ` - z - - - z Z Z - z   
 0210  95 95 a9 7a aa aa aa 7a 55 55 ab 7a 01 01 ac 7a   
     - - - z - - - z U U - z - - - z   
 0220  f8 f8 ad 7a 03 03 ae 7a 00 00 af 7a 09 09 b0 7a   
     - - - z - - - z - - - z - - - z   
 0230  09 09 b1 7a 00 00 b2 7a 00 00 b3 7a 00 00 b4 7a   
     - - - z - - - z - - - z - - - z   
 0240  80 80 b5 7a 07 07 b6 7a 40 40 b7 7a 10 10 b8 7a   
     - - - z - - - z @ @ - z - - - z   
 0250  04 04 b9 7a 01 01 bf 7a ff ff c0 7a a3 a3 c1 7a   
     - - - z - - - z - - - z - - - z   
 0260  7e 7e 52 72 06 06 52 74 06 06 52 76 06 06 52 78   
     ~ ~ R r - - R t - - R v - - R x   
 0270  06 06 53 72 1b 1b 53 74 1b 1b 53 76 1b 1b 53 78   
     - - S r - - S t - - S v - - S x   
 0280  1b 1b 54 72 1e 1e 54 74 1e 1e 54 76 1e 1e 54 78   
     - - T r - - T t - - T v - - T x   
 0290  1e 1e 55 72 53 53 55 74 53 53 55 76 53 53 55 78   
     - - U r S S U t S S U v S S U x   
 02a0  53 53 56 72 31 31 56 74 31 31 56 76 31 31 56 78   
     S S V r 1 1 V t 1 1 V v 1 1 V x   
 02b0  31 31 57 72 1b 1b 57 74 1b 1b 57 76 1b 1b 57 78   
     1 1 W r - - W t - - W v - - W x   
 02c0  1b 1b 51 72 0c 0c 51 74 0c 0c 51 76 0c 0c 51 78   
     - - Q r - - Q t - - Q v - - Q x   
 02d0  0c 0c 00 70 07 07 90 00 ff 00 00 00 00 00 00 00   
     - - - p - - - - - - - - - - - -   
 02e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 02f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0300  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0310  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0320  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0330  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0340  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0350  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0360  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0370  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0380  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0390  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 03a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 03b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 03c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 03d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 03e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 03f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0400  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0410  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0420  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0430  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0440  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0450  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0460  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0470  00 00 00 00 00 00 00 7a df df 00 00 00 00 15 00   
     - - - - - - - z - - - - - - - -   
 0480  3b 3f ba 81 07 00 3f 00 14 00 00 00 07 4e 61 1f   
     ; ? - - - - ? - - - - - - N a -   
 0490  28 10 a6 10 02 00 f7 17 15 15 35 00 64 6e 01 00   
     ( - - - - - - - - - 5 - d n - -   
 04a0  24 6e 96 72 78 18 e2 ea 07 00 05 00 3f 06 28 15   
     $ n - r x - - - - - - - ? - ( -   
 04b0  00 00 04 14 00 00 05 00 3f 16 28 15 00 00 04 14   
     - - - - - - - - ? - ( - - - - -   
 04c0  00 00 07 00 24 6e 96 72 78 18 06 01 82 05 00 00   
     - - - - $ n - r x - - - - - - -   
 04d0  01 00 07 00 24 6e 96 72 78 1a 06 01 82 05 00 00   
     - - - - $ n - r x - - - - - - -   
 04e0  01 00 02 00 40 0f 00 00 02 00 40 0f 00 00 0f 00   
     - - - - @ - - - - - @ - - - - -   
 04f0  44 04 00 00 80 00 00 03 ff ff ff ff 00 03 ff ff   
     D - - - - - - - - - - - - - - -   
 0500  ff ff 00 03 ff ff ff ff 00 03 ff ff ff ff 0f 00   
     - - - - - - - - - - - - - - - -   
 0510  44 04 00 00 80 00 00 03 ff ff ff ff 00 03 ff ff   
     D - - - - - - - - - - - - - - -   
 0520  ff ff 00 03 ff ff ff ff 00 03 ff ff ff ff 10 3d   
     - - - - - - - - - - - - - - - =   
 0530  20 00 12 ff c6 c4 00 00 05 03 00 03 00 00 00 c8   
       - - - - - - - - - - - - - - -   
 0540  00 00 00 00 00 00 00 00 00 00 a2 02 00 02 82 15   
     - - - - - - - - - - - - - - - -   
 0550  00 58 35 34 30 20 31 30 47 62 45 20 43 6f 6e 74   
     - X 5 4 0   1 0 G b E   C o n t   
 0560  72 6f 6c 6c 65 72 90 79 00 56 30 0a 46 46 56 31   
     r o l l e r - y - V 0 - F F V 1   
 0570  38 2e 30 2e 31 36 50 4e 06 47 31 34 38 34 33 4d   
     8 . 0 . 1 6 P N - G 1 4 8 4 3 M   
 0580  4e 04 31 30 32 38 56 31 12 44 53 56 31 30 32 38   
     N - 1 0 2 8 V 1 - D S V 1 0 2 8   
 0590  56 50 44 52 2e 56 45 52 31 2e 30 56 33 06 44 54   
     V P D R . V E R 1 . 0 V 3 - D T   
 05a0  49 4e 49 43 56 34 17 44 43 4d 31 30 30 31 30 30   
     I N I C V 4 - D C M 1 0 0 1 0 0   
 05b0  38 37 44 35 32 31 30 31 30 30 38 37 44 35 56 35   
     8 7 D 5 2 1 0 1 0 0 8 7 D 5 V 5   
 05c0  04 4e 50 59 32 56 36 04 50 4d 54 31 56 37 0d 4e   
     - N P Y 2 V 6 - P M T 1 V 7 - N   
 05d0  4d 56 49 6e 74 65 6c 20 43 6f 72 70 52 56 03 39   
     M V I n t e l   C o r p R V - 9   
 05e0  00 00 78 00 ff ff ff ff ff ff ff ff ff ff ff ff   
     - - x - - - - - - - - - - - - -   
 05f0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0600  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0610  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0620  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0630  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0640  ff ff ff ff ff ff ff ff ff ff ff ff ff ff 24 6e   
     - - - - - - - - - - - - - - $ n   
 0650  96 72 78 18 24 6e 96 72 78 1a 69 53 e0 05 01 00   
     - r x - $ n - r x - i S - - - -   
 0660  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0670  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0680  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0690  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 06a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 06b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 06c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 06d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 06e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 06f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0700  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0710  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0720  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0730  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0740  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0750  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0760  00 12 10 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0770  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0780  00 00 03 3d 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - = - - - - - - - - - - - -   
 0790  00 00 00 00 00 00 bc 0c 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 07a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 07b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 07c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 07d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 07e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 07f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0800  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0810  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0820  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0830  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0840  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0850  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0860  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0870  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0880  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0890  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 08a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 08b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 08c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 08d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 08e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 08f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0900  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0910  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0920  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0930  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0940  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0950  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0960  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0970  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0980  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0990  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 09a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 09b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 09c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 09d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 3d   
     - - - - - - - - - - - - - - - =   
 09e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 09f0  00 00 bc 0c 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a10  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a20  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a30  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a40  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a50  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a60  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a70  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a80  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0a90  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0aa0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0ab0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0ac0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0ad0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0ae0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0af0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b10  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b20  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b30  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b40  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b50  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b60  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b70  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b80  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0b90  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0ba0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0bb0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0bc0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0bd0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0be0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0bf0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0c00  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0c10  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0c20  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   
     - - - - - - - - - - - - - - - -   
 0c30  00 00 00 00 00 00 00 00 00 00 24 6e 96 72 78 19   
     - - - - - - - - - - $ n - r x -   
 0c40  24 6e 96 72 78 1b 03 00 24 6e 96 72 78 19 24 6e   
     $ n - r x - - - $ n - r x - $ n   
 0c50  96 72 78 1b 00 20 01 20 06 00 35 48 30 34 39 34   
     - r x - -   -   - - 5 H 0 4 9 4   
 0c60  30 2d 35 30 56 4c 01 08 00 00 00 00 ff ff ff ff   
     0 - 5 0 V L - - - - - - - - - -   
 0c70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0c80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0c90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ca0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0cb0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0cc0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0cd0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ce0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0cf0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d00  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d10  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d20  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d30  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d40  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d50  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d60  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0d90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0da0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0db0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0dc0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0dd0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0de0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0df0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e00  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e10  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e20  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e30  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e40  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e50  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e60  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0e90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ea0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0eb0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ec0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ed0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ee0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ef0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f00  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f10  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f20  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f30  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f40  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f50  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f60  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f70  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f80  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0f90  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0fa0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0fb0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0fc0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0fd0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0fe0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   
 0ff0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   
     - - - - - - - - - - - - - - - -   

The most interesting part of the dump from text (ASCII) perspective is the part between 0550h and 05E0h.

  0550 00 58 35 34 30 20 31 30 47 62 45 20 43 6f 6e 74  
    - X 5 4 0  1 0 G b E  C o n t  
  0560 72 6f 6c 6c 65 72 90 79 00 56 30 0a 46 46 56 31  
    r o l l e r - y - V 0 - F F V 1    
  0570 38 2e 30 2e 31 36 50 4e 06 47 31 34 38 34 33 4d    
    8 . 0 . 1 6 P N - G 1 4 8 4 3 M    
  0580 4e 04 31 30 32 38 56 31 12 44 53 56 31 30 32 38    
    N - 1 0 2 8 V 1 - D S V 1 0 2 8    
  0590 56 50 44 52 2e 56 45 52 31 2e 30 56 33 06 44 54  
    V P D R . V E R 1 . 0 V 3 - D T  
  05a0 49 4e 49 43 56 34 17 44 43 4d 31 30 30 31 30 30  
    I N I C V 4 - D C M 1 0 0 1 0 0  
  05b0 38 37 44 35 32 31 30 31 30 30 38 37 44 35 56 35   
    8 7 D 5 2 1 0 1 0 0 8 7 D 5 V 5  
  05c0 04 4e 50 59 32 56 36 04 50 4d 54 31 56 37 0d 4e    
    - N P Y 2 V 6 - P M T 1 V 7 - N    
  05d0 4d 56 49 6e 74 65 6c 20 43 6f 72 70 52 56 03 39    
    M V I n t e l  C o r p R V - 9    
  05e0 00 00 78 00 ff ff ff ff ff ff ff ff ff ff ff ff  
    - - x - - - - - - - - - - - - -  

There are adapter model and vendor name but it seems to me that it does not fit into the memory maps defined in the SFF-8472.  Therefore I do not know how to find optic (SFP) diagnostic values in EEPROM memory dump.

If you have any idea if I am in the right track and how to continue, please let me know.

vSphere 6.5 - DRS CPU Over-Commitment

A lot of VMware vSphere architects and engineers are designing their vSphere clusters for some overbooking ratios to define some level of the service (SLA or OLA) and differentiate between different compute tiers. They usually want to achieve something like

• Tier 1 cluster (mission-critical applications) - 1:1 vCPU / pCPU ratio
• Tier 2 cluster (business-critical applications) - 3:1 vCPU / pCPU ratio
• Tier 3 cluster (supporting applications) - 5:1 vCPU / pCPU ratio
• Tier 4 cluster (virtual desktops) - 10:1 vCPU / pCPU ratio

Terminology: 

• vCPU - virtual CPU configured for Virtual Machine
• pCPU - physical CPU core (hyper-threading is not considered) 

 

Before vSphere 6.5 we have to monitor it externally by vROps or some other monitoring tool. Some time ago I have blogged how to achieve it with PowerCLI and LogInsight - ESXi host vCPU/pCPU reporting via PowerCLI to LogInsight.

vSphere 6.5 DRS has introduced additional option to set maximum CPU over-commitment. It limits the number of vCPUs per pCPU in particular DRS cluster. However, it is good to know that there are two different advanced DRS options (configuration parameters) how to specify vCPU:pCPU ratio and each setting behaves differently. See table below ...

DRS Advanced Option	Scope	Min-Max Value
MaxVcpusPerClusterPct	cluster	0% - 500%
MaxVCPUsPerCore	host	0 - 32

It is worth to mention a little bit tricky setting of these additional options via GUI. It is good to know how GUI setting of "CPU Over-Commitment" is mapped to DRS cluster advanced options.

In my lab, I have VCSA 6.5 U1c (build 7119157) so I did some tests.

If I set "CPU Over-Commitment" in vSphere Web Client (Flash/Flex) it sets MaxVcpusPerClusterPct so it is the setting per the whole vSphere Cluster.

However, in vSphere Client (HTML5) it sets MaxVCPUsPerCore so it is per ESXi host.

Therefore, it is good to know what you would like to achieve and double check DRS Advanced Options.

See different behavior in screenshots below

6.5 U1c (build 7119157) vSphere Web Client (Flash/Flex) sets MaxVcpusPerClusterPct

6.5 U1c (build 7119157) vSphere Client (HTML5) sets MaxVCPUsPerCore

So this is how it should work. Now let's do some test to understand real behavior.

TEST 1: MaxVcpusPerClusterPct = 0 

Let's set MaxVcpusPerClusterPct to 0 so we are saying to allow 0 : 1 vCPU / pCPU ratio. In other words, no VM can be running in the cluster. And it works as expected. When I try to run VM I get the error "The total number of virtual CPUs present or requested in virtual machines' configuration has exceeded the limit on the host: 0". Well, it is a little bit misleading because it should be cluster-wide rule but it works as expected.

Error when MaxVcpusPerClusterPct is set to 0

TEST 2: MaxVcpusPerClusterPct = 100 

Let's set MaxVcpusPerClusterPct to 100% so we are saying to allow 1 : 1 vCPU / pCPU ratio. I have 4 node DRS cluster where each ESXi host has two cores (pCPUs), therefore I have 8 pCPUs available in the cluster. And I can really start only four VMs because each has 2 vCPUs so I can run up to 8 vCPUs in DRS cluster.

Error when MaxVcpusPerClusterPct is set to 100

It is great, but it is worth to mention that VMs were started on single ESXi host even I have 4 ESXi hosts in DRS cluster. So vCPU / pCPU ratio is compliant per cluster but not per ESXi host as I have 8 vCPUs on single ESXi hosts having just 2 pCPUs. But that's expected behavior. So far so good.

TEST 3: MaxVcpusPerCore = 0 

MaxVcpusPerCore should solve the problem observed in previous two test because it should set vCPU / pCPU ratio per host which is much better from the predictability point of view.

Let's set MaxVcpusPerCore to 0. My expectation was that I will not be able to start any VM but that was NOT the case. I was able to start a lot of VMs and exceed the expected vCPU / pCPU ratio. This is unexpected behavior.

TEST 4: MaxVcpusPerCore = 1 

Let's set MaxVcpusPerCore to 1.  My expectation was that I will not be able to start more than 2 vCPUs per ESXi host so only one VM with 2 vCPUs. Unfortunately, I was able to start much more vCPUs per ESXi host. This is again unexpected behavior.

TEST 5: MaxVcpusPerCore = 4 

I have been informed by DRS Engineer that the minimum value allowed by host over-commitment ratio option (MaxVCPUsPerCore) is 4:1.

So, let's set MaxVcpusPerCore to 4.  I have prepared nine VMs with 4 vCPUs each and my expectation is that I will not be able to start more than 8 vCPUs per ESXi host so only two VMs with 4 vCPUs per ESXi host. And because I have 4 ESXi hosts per cluster I should be able to start the maximum of 8 VMs.

Expected error message when MaxVcpusPerCore is set to 4 and vCPU:pCPU is over 4:1 per ESXi host.

UPDATE 2018-04-20: The issue with MaxVcpusPerCore is fixed in vSphere 6.5 U2. This is written in Release Notes: The advanced vSphere DRS parameter MaxVcpusPerCore might not work as expected and the desired ratio of virtual CPUs per physical CPU or core will not take effect in configurations below 4:1. MaxVcpusPerCore supported ratios now start from 1:1.
Please note, that MaxVcpusPerCore does not support ratio 0:1 in contrast with MaxVcpusPerClusterPct where 0:1 is possible and it will effectively disable to run any VM on the cluster.

Conclusion

Advanced DRS setting MaxVcpusPerClusterPct supports values between 0 and 500 and represents percentage between vCPUs and pCPUs across the whole DRS cluster. If the value is higher then 500, enforcing does not work. So, vCPU / pCPU percentage ratio can be enforced between 0:1 to 5:1.  vCPU:pCPU ratio 0:1 is a special setting where no VM can be PowerOn on vSphere Cluster. This is little bit risky setting but it can be used to put the whole cluster in kind of "maintenance mode" and forbid anyone to run VMs there.

Advanced DRS setting MaxVcpusPerCore currently supports values between 0 and 32 but it works only with values between 4 and 32. This setting enforces vCPU / pCPU per each ESXi host within DRS cluster. This means that the minimum vCPU / pCPU ration configurable by this option is 4 : 1.

To be honest, I think for vSphere architects/designers MaxVcpusPerCore makes more sense than MaxVcpusPerClusterPct because vCPU/pCPU overbooking ratio defines CPU quality per ESXi host.

VMware is internally considering to align MaxVcpusPerClusterPct and MaxVcpusPerCore behavior and allow lower vCPU / pCPU ratios when MaxVcpusPerCore is used. I will track how this topic will evolve in the future. Stay tuned.

VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown)

Since January 3, 2017, the whole IT industry is mitigating the impact of SPECTRE and MELTDOWN vulnerabilities and administrators are updating their infrastructures.

Three different CVEs have been identified related to the media described issues:

• CVE-2017-5753 (Spectre - Variant 1) - Branch target injection
• CVE-2017-5715 (Spectre - Variant 2) - Bounds check bypass
• CVE-2017-5754 (Meltdown - Variant 3) - Rogue data cache load

All three security CVEs (common vulnerabilities and exposures) are abusing multi-pipeline architecture (using multiple CPU pipelines) and performance optimization (speculative execution) used in all modern CPUs. Simplified but IMHO very nice explanation is here. If I understand the issue and resolution correctly, it all boils down to new CPU microcodes which must expose special CPU instructions to operating systems which has to incorporate it into their kernels.

VMware infrastructure is, obviously, impacted as well. It is very important to patch the whole infrastructure stack. All subsystems in the infrastructure stack described below must be patched or upgraded for effective mitigation. The virtual infrastructure stack is composed of

• Operating systems (OS)
• Virtual Machines / Virtual Appliances
• Hypervisors
• CPU microcode (usually part of BIOS)

GUEST OS
For guest OS you should obtain patches from your OS software vendors. However, guest operating systems are using new CPU instructions (capabilities IBRS, IBPB, STIBP) to mitigate Spectre/Meltdown vulnerabilities. These capabilities are exposed from physical CPU microcode, therefore, patches from top to down (OS-HYPERVISOR-CPU microcode) is necessary for successful vulnerability mitigation. To expose those new CPU capabilities to guest OS, Virtual Machine hardware 9 and newer must be used otherwise those new capabilities are masked by VM hardware and Guest OS kernel would not be able to use it.

VIRTUAL APPLIANCES

Another consideration should be given to VMware Virtual Appliances. VMware officially released a new Knowledge Base article regarding the investigation of the impact on the VMware Virtual Appliances (KB 52264) and there is also VMware Security Advisory VMSA-2018-0007.

HYPERVISOR
Hypervisor remediation can be classified into the two following categories:

• Hypervisor-Specific Remediation (documented in VMSA-2018-0002) - Result of exploitation may allow for information disclosure from one Virtual Machine to another Virtual Machine that is running on the same host. 
• Hypervisor-Assisted Guest Remediation (documented in VMSA-2018-0004) - This issue may allow for information disclosure between processes within the VM.

For Hypervisor-Specific remediation, all relevant patches already exist and are described in security advisory VMSA-2018-0002. VMware recommends applying described patches.

For Hypervisor-Assisted Guest Remediation, all relevant patches exist and are described in security advisory VMSA-2018-0004.

Rogue Data Cache Load or “Meltdown” (CVE-2017-5754) does not affect ESXi, Workstation, and Fusion because ESXi does not run untrusted user mode code, and Workstation and Fusion rely on the protection that the underlying operating system provides.

PERFORMANCE IMPACT

The virtualization specific performance impact of these mitigations should be negligible but performance impact of workloads inside guest OS is expected but workload dependent. There are rumors it might be somewhere between 10% and 30%.

The VMware performance team evaluated the performance costs of the Meltdown/Spectre mitigations for vSphere and official messaging of performance impact is published in VMware KB 52337.

Customers having vROps 6.6.0 and above can leverage specific dashboards to assess performance impact of Spectre & Meltdown patches using vRealize Operations. For more information read this blog post.

HOW TO UPDATE?

Please note, the update order is important.

To enable hardware support for branch target mitigation in vSphere, you should apply these steps, in the order shown below:

1. Ensure you have EVC enabled on vSphere clusters
2. For vCenter 6.x, in case you have external PSC, update all external PSCs to the latest patches of PSC 5.5 – 6.5 
3. Update to the latest patches of vCenter 5.5 – 6.5 
4. Apply the latest ESXi 5.5 – 6.5 patches
5. Apply the CPU Microcode by BIOS or alternatively by VMware patch mentioned in VMSA-2018-0004 where VMware distributes CPU Microcode. Please note, CPU Microcode distributed by BIOS is highly recommended.

An ESXi host that is running both a patched CPU microcode and a patched hypervisor will see new CPU features that were not previously available. These new features will be exposed to all Virtual Hardware Version 9+ VMs that are powered-on by that host. Because these VMs now see additional CPU features, vMotion to ESXi without the microcode or hypervisor patches applied will be prevented. There are three new CPU Features

• IBRS
• IBPB
• STIBP

These three new CPU capabilities are not known to existing EVC baselines, therefore it seems to ESXi hosts as newer CPU model and prevents vMotion migration. EVC baselines are defined in vCenter and that's the reason why vCenter patch must be applied before ESXi patch and CPU microcode update. Patched EVC on the latest vCenter Server is leveraged to temporarily mask new CPU Features until all hosts in the cluster have new CPU Microcode. This allows vMotion of VMs from unpatched to patched ESXi hosts.

HOW TO VERIFY YOUR UPDATE WAS SUCCESSFUL AND YOU ARE NOT VULNERABLE?

You should verify it in different layers.

Hypervisor level

• You can verify it manually in vmware.log of particular VM as is described in KB 52085
• William Lam has published blog post "Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI" with PowerCLI script to test vulnerability status on particular ESXi host or vSphere Cluster. 
• Vikas Shitole has published the blog post "PowerCLI script : How to confirm a vCenter server, ESXi VMware hypervisor & microcode patches are applied?: Spectre vulnerability" with PowerCLI automation script to test all ESXi host within vCenter.

If you want to know if you have Intel microcode updated you can run following shell command on your ESXi host

if [ `vsish -e get /hardware/msr/pcpu/0/addr/0x00000048 2&>1 > /dev/null ;echo $?` -eq 0 ]; then echo -e "\nIntel Security Microcode Updated\n";else echo -e "\nIntel Security Microcode NOT Updated\n";fi 

On Intel machine with microcode update, you get the output "Intel Security Microcode Updated".
On Intel machine with microcode update, you get the output "Intel Security Microcode NOT Updated".

If you want to do the same for AMD microcode following command should work

if [ $(($((`vsish -e get /hardware/cpu/cpuList/0 | grep -i EBX | head -5 | tail -1 | awk -F: '{print $2}'` & 0x00001000)) >> 12)) -eq 1 ]; then echo -e "\nAMD Security Microcode Updated\n";else echo -e "\nAMD Security Microcode NOT Updated\n";fi 

Guest OS - Linux

• Very nice blog post with the shell script is here “How to check Linux for Spectre and Meltdown vulnerability". Please note, that if you run the shell script inside guest OS on top of update ESXi host, the script will report that CPU microcode is updated even the physical CPU microcode is vulnerable. However, the information if Guest OS is patched is still valuable.

Guest OS - Microsoft Windows

• Microsoft has published following article "Windows Client Guidance for IT Pros to protect against speculative execution side-channel vulnerabilities" with PowerShell module SpeculationControl where it the command-let Get-SpeculationControlSettings.

OTHER VENDORS RESPONSES AND MITIGATIONS

Intel official response is available here
https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr
Intel Analysis of Speculative Execution Side Channels is available here https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Intel-Analysis-of-Speculative-Execution-Side-Channels.pdf

DELL official response: Meltdown and Spectre Vulnerabilities
DELL KB article with technical information: http://www.dell.com/support/article/SLN308588

HP response I have found so far: Resources to help mitigate Speculative Execution vulnerability in Intel and other processors
HP Support Customer Bulletin: a00039267en_us
Hewlett Packard Enterprise Product Security Vulnerability Alerts: http://h22208.www2.hpe.com/eginfolib/securityalerts/SCAM/Side_Channel_Analysis_Method.html

RedHat: Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715

FreeBSD: How to patch Meltdown and Spectre vulnerabilities on FreeBSD

If you know links to other or better server vendors responses, please leave it in comments.

Hope this article is helpful. And as always, if you see some wrong information or you just have a question or some other experience, do not hesitate to write the comment. I will try to update this article in case something new is available.

Sources:

• VMSA-2018-0002 - VMware ESXi, Workstation and Fusion updates address side-channel analysis due to speculative execution - https://www.vmware.com/security/advisories/VMSA-2018-0002.10.html
• VMSA-2018-0004 - VMware vSphere, Workstation and Fusion updates add Hypervisor-Assisted Guest Remediation for speculative execution issue - https://www.vmware.com/security/advisories/VMSA-2018-0004.10.html
• VMSA-2018-007 - VMware Virtual Appliance updates address side-channel analysis due to speculative execution - https://www.vmware.com/us/security/advisories/VMSA-2018-0007.10.html
• VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52245) - https://kb.vmware.com/kb/52245 
• Hypervisor-Assisted Guest Mitigation for branch target injection (52085) - https://kb.vmware.com/s/article/52085
• VMware Virtual Appliances and CVE-2017-5753, CVE-2017-5715 (Spectre), CVE-2017-5754 (Meltdown) (52264) - https://kb.vmware.com/s/article/52264
• VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52337) - https://kb.vmware.com/kb/52337
• Intel Sightings in ESXi Bundled Microcode Patches for VMSA-2018-0004 (52345) - https://kb.vmware.com/s/article/52345
• Assess performance impact of Spectre & Meltdown patches using vRealize Operations Manager - https://blogs.vmware.com/management/2018/01/assess-performance-impact-spectre-meltdown-patches-using-vrealize-operations-manager.html
• WHY RASPBERRY PI ISN’T VULNERABLE TO SPECTRE OR MELTDOWN
• Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715 and CVE-2017-5753 using Red Hat Enterprise Linux Tunables

Admission Control "Dedicated fail-over hosts"

More then three years ago I published the blog post about "vSphere HA Cluster Redundancy". There are three algorithms

• Define fail-over capacity by static number of hosts
• Define fail-over capacity by reserving a percentage of cluster resources
• Use dedicated fail-over hosts

I discussed first two algorithms very well but the third one "dedicated fail-over hosts" was described briefly by following words ...

This algorithm simply dedicates specified hosts to be unused during normal conditions and used only in case of ESXi host failure. Multiple fail-over dedicated hosts are supported since vSphere 5.0. This algorithm will keep your capacity and performance absolutely predictable and independent on VM reservations. You'll get exactly what you configure.

I have been asked recently by one my customer for some details behavior of  "dedicated fail-over hosts" method.

Question #1: How will be VMs restarted in case of single host failure when two hosts are dedicated for fail-over?

Answer: All impacted VMs are restarted and spread across both dedicated fail-over hosts.

Question #2: Is it possible to vMotion VMs to dedicated hosts for fail-over?

Answer: Of course not. These hosts are dedicated just for fail-over and vSphere Cluster is aware about it, therefore, it will not allow administrator nor DRS to migrate VMs there.

Question #3: What will happen with VMs when the failed host is back?

Answer: VMs will stay on on hosts dedicated for fail-over unless DRS will move them to other hosts. Based on my testing, DRS will do it probably just in case of lack of resources, therefore, some VMs can stay on dedicated fail-over hosts, which is not good. Therefore, vSphere administrator should check the cluster state after host failure and move all VMs out of dedicated fail-over hosts if DRS did not do it before.

Hope this helps broader VMware community to better understand VMware Admission Control. And as always, if you have some other question, opinion or different experience, please, feel free to leave the comment below.

SDRS Initial Placement - interim storage lease between recommendation and provisioning

Every day we learn something new. In the past, I blogged about SDRS behavior on these blog posts

• Storage DRS Design Considerations
• VMware vSphere SDRS - test plan of SDRS initial placement
• VMware vSphere SDRS VM provisioning process
• Storage DRS integration with storage profiles

Recently (a few months ago),  I have been informed about interesting SDRS behavior which is not exposed through standard GUI nor advanced settings but available through API. Such functionality was not very well known even within VMware so I have decided to blog about it.

Long story short ...

vSphere API Call for SDRS Initial Placement can lease recommended storage resource for some time. 

What does it mean? Just after recommendations, SDRS can lease the storage space on recommended datastores to have an interim reservation for somebody who is, most probably, going to do provisioning. By default, SDRS do not lease storage space on recommended datastores, therefore, you can observe provisioning failures in some situations. I have simulated such situation in Test #3 of test plan available here. Such situations are not very common when you do manual provisioning but there is higher probability when automated provisioning is in use so you can experience such issues on environments with VMware vRealize Automation (vRA) or vCloud Director (vCD).

And now the secret I did not know ... SDRS has the solution for such issues since vSphere 5.1. When somebody (vRA, vCD, anybody else who wants to deploy VM) asked for SDRS recommendation via API call, that API call can include a specific parameter (resourceLeaseDurationSec) which will instruct vSphere to block the recommended storage space on datastores only for provisioning of that specific SDRS recommendation. It's worth to mention, the resource leasing is released immediately after provisioning, therefore the time defined in resourceLeaseDurationSec is actually the maximum reservation time of the resource just in case somebody who wanted to do provisioning change his mind and decided to not deploy VM. This is to avoid unnecessary storage space reservations.
 
If you want to know details check API documentation. Here is what is written in vSphere API documentation about placeSpec.resourceLeaseDurationSec - Resource lease duration in seconds. If the duration is within bounds, Storage DRS will hold onto resources needed for applying recommendations generated as part of that call. Only initial placement recommendations generated by storage DRS can reserve resources this way.

Parameter resourceLeaseDurationSec is used in StoragePlacementSpec which encapsulates all of the inputs passed to the VcStorageResourceManager method recommendDatastores. VcStoragePlacementSpec is documented here.

So that sounds good, right? Well, there is one issue with this approach. SDRS can give provisioning application more recommendations (multiple datastores) which would lead to blocking more storage space than is really needed. VMware engineering is aware of this issue and at the moment works at least with vRA BU to solve it. As far as I know, the final solution will be a special SDRS setting to return a single recommendation. However, this is planned as a specific integration optimization between SDRS and vRA provisioning. 

UPDATE: details about special SDRS setting is described at https://www.vcdx200.com/2018/06/undocumented-sdrs-advanced-options.html

The challenge with VRA storage (vSphere vDisk) provisioning and SDRS is depicted in the figure bellow

To be honest, there is another design consideration and potential risk associated with this solution. If resourceLeaseDurationSec is used and an external application (VRA, vCloud Director, or other) is using it incorrectly, it can eventually block a storage space in the Datastore cluster and cause Deny of Service (DoS). Incorrect usage would be to leverage SDRS for recommendations that would block the recommended storage space but do not provision anything, therefore storage would be in block state for some defined time and not available for other provisionings until the lease expires.