Skip to main content

Accurately recreating the partition table of a VMFS volume

 This article need an expert to use, Please reach to VMware support to not run into more complex situation.


 Details
  • You cannot mount a VMFS volume which you were previously able to mount.
  • The partition table of the volume no longer contains any partitions.
  • When recreating a partition table on a volume, this error appears in /var/log/vmkernel:

    WARNING: LVM: 1650: [vmhba1:0:2:1] Device size mismatch (actual 16777083 blocks, stored 16777088 blocks)
 Solution
In some cases, the original partition table was not created at the end of the volume. You must calculate what the end boundary of the lost partition table entry is, and re-create the new one with the same ending.

Notes:
The procedure in this article uses this sample:

00110000 0d d0 01 c0 03 00 00 00 10 00 00 00 00 0a 00 00 |................|
00110010 00 03 56 4d 77 61 72 65 2c 20 56 4d 77 61 72 65 |..VMware, VMware|
00110020 20 56 69 72 74 75 61 6c 20 53 31 2e 30 20 76 6d | Virtual S1.0 vm|
00110030 68 62 61 30 3a 31 3a 30 00 00 00 00 00 00 00 00 |hba0:1:0........|
00110040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00110050 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 |................|
00110060 ff 7f 00 00 00 00 01 00 00 00 07 00 00 00 06 00 |................|
00110070 00 00 09 00 00 00 00 00 00 00 00 00 10 01 00 00 |................|
00110080 00 00 d8 9b 6f 47 42 5d 03 db c8 6d 00 0c 29 05 |....oGB]...m..).|
00110090 e9 39 e7 8a c6 c3 06 42 04 00 39 3a c7 c3 06 42 |.9.....B..9:...B|
001100a0 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001100b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|


Perform these steps:
 
  1. Collect the first two (2) megabytes of the volume using the dd command:

    # dd if=/dev/sdX of=/root/sdX.dump bs=1M count=2

    The file sdX.dump is created in /root. If copying a file from the ESX console is a problem, substitute /root with /vmfs/volumes/volume_name/sdX.dump.

    You can download the file using the VMware Infrastructure / vSphere Client datastore browser.

    Note: You can view the information live with this command:

    # hexdump -C -n 2048000 /dev/sdX
     
  2. Identify the necessary values:
     
    1. To view the 2 MB dump, run the command:

      # hexdump -C sdX.dump | less
       
    2. Search for the magic number (it marks the beginning offset of the LVM device header offset): 0d d0 01 c0

      In the sample, it is located at 0x00110000 or 1114112.

      Note: The conversion of the 0x00110000 (Hex) to 1114112 (Decimal) can be done using the Windows Calculator in Programmer Mode. Activate Programmer mode, input in HEX format the value "0x00110000" and change to decimal in the Calculator to get the converted value.

      The starting sector is calculated by:

      (<LVM device header offset in decimal> - 1048576) / 512

      In this sample, the starting sector = (1114112 - 1048576) / 512 = 128

      Note: 1048576: This number is always the same (1048576 <=> 1MB). It corresponds to the 1 MB padding (zeros) at the beginning of a VMFS partition. This is true for vmfs3 and vmfs5.
       
    3. To get the total size (in bytes) of the volume, extract from position 0x5e (94) the next 8 bytes (00 00 ff 7f 00 00 00 00).

      As the value is encoded in little endian, it resolves to 0x000000007fff0000 or 2147418112 bytes.

      Note: To find position 0x5e, go to byte 0x110000 + 0x5e, or find 110050 in the first column and then go right 0xe (15) bytes.

      Little endian: 00 00 ff 7f 00 00 00 00
      Big endian: 00 00 00 00 7f ff 00 00
      Bits: 00000000 00000000 00000000 00000000 01111111 11111111 00000000 00000000
      Decimal: 0*2^0 + 0*2^1 + ... + 1*2^16 + ... + 0*2^47 = 2147418112
       
    4. Calculate the ending block:

      ((<Total size in bytes> + (<Starting offset> * 512)) / 512) -1

      In this sample, the ending sector = ((2147418112 + (128 * 512)) / 512) - 1 = 4194303
       
  3. Use the fdisk utility to correct the partition:
     
    1. Start fdisk on the device using the -u option to be able to set the starting block:

      # fdisk -u /dev/sdX
       
    2. If the partition exists, remove it:
      1. Press d and press Enter.
      2. Press 1 and press Enter.
         
    3. Create the partition:
      1. Press n and press Enter.
      2. Enter the calculated value of the starting offset (in this sample: 128) and press Enter.
      3. Enter the calculated value of the ending block (in this sample: 4194303) and press Enter.
         
    4. Change the partition type to FB:
      1. Press t and press Enter.
      2. Press 1 and press Enter.
      3. Enter fb and press Enter.
         
    5. Save the changes and exit. Press w and press Enter.
       
  4. Rescan the existing volumes to see if the datastores can be mounted with the command:

    # vmkfstools -V

    The missing volume should be mounted now. If not, check /var/log/vmkernel for further evidence of:

    WARNING: LVM: 1650: [vmhba1:0:2:1] Device size mismatch (actual 16777083 blocks, stored 16777088 blocks)

    Verify your numbers and that the partition was created correctly.

Note: For large LUNs, fdisk seems to have a display issue (that is, it displays the ending block as a negative number). In this case, you can use parted or another tool to create the partition.

If this is not an option and you are still getting Device size mismatch errors, you can determine the ending with this calculation:

<Adjusted End> = -(abs(<Previous End>) + (<Actual> - <Stored> + <Starting Offset>) )

Where:
  • <Adjusted End> is the new ending block adjusted to work around the fdisk bug.
  • <Previous End> is the ending block fdisk displayed for the volume that generated the above error.
 Confidential or Internal Information

A VMFS5 sample for reference:

00200000 0d d0 01 c0 05 00 00 00 15 00 00 00 02 16 04 00 |................|
00200010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00200020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60 03 |..............`.|
00200030 0d 90 1b 3c 19 02 4a 3d 96 45 96 1a 91 43 56 69 |...<..J=.E...CVi|
00200040 72 74 75 61 00 00 00 00 00 00 00 00 00 00 00 00 |rtua............|
00200050 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 be |................|
00200060 ef ff 76 01 00 00 01 00 00 00 6f 17 00 00 6e 17 |..v.......o...n.|
00200070 00 00 05 00 00 00 00 00 00 00 00 00 10 01 00 00 |................|
00200080 00 00 c7 fc 2c 51 50 ab d5 bb 3d 3a 18 a9 05 5e |....,QP...=:...^|
00200090 33 00 40 0a 89 f9 a1 cb 04 00 78 89 1b 6e a9 d6 |3.@.......x..n..|
002000a0 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
002000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 |................|
002000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
 

In this example,
the magic number can be found on the line starting 00200000 , the total size is located on the line starting 00200050 and the little endian value is: 00 be ef ff 76 01 00 00 .

Convert the little endian value to big endian: 00 00 01 76 ff ef be 00 .
Convert Hex to Decimal: 1610611670528
The size of the volume is: 1610611670528 bytes = ~1500GB

Collect the first two (2) megabytes of the volume using the dd command:

# dd if=/dev/sdX of=/root/sdX.dump bs=1M count=2

The file sdX.dump is created in /root . If copying a file from the ESX console is a problem, substitute /root with /vmfs/volumes/volume_name/sdX.dump.

We can also use /vmfs/devices/disks/naa.X for this command This is valid for 5.x and 6.x

 

 

 

Recreating the volume with partedUtil
 
For VMFS3, using MBR:
partedUtil set /path/to/device "1 <starting offset> <ending offset> 251 0"
 
In this example: partedUtil set /dev/sdX "1 128 4194303 251 0"
 
 
For VMFS5, using GPT:
Note: You can use GPT partition type if the partition size is more than 2TB
partedUtil setptbl /path/to/device gpt "1 <starting offset> <ending offset> AA31E02A400F11DB9590000C2911D1B8 0"
 
example:
partedUtil setptbl /vmfs/devices/disks/naa.6f01faf000dcedf3000005a6558a589f gpt "1 2048 35156901854 AA31E02A400F11DB9590000C2911D1B8 0"
 
For VMFS 6 
SR = 19126886203 

Example Calculation goes as below 
 

LVM header :- 0d d0 01 c0

Shows this offset 00200000 = 2048 = VMFS 6 

00 02 e0 ff ff 05 00 00 = Hex 

000005ffffe00200 = Decimal Value :- 6597067670016

As per the Calculation and formula :- ((<Total size in bytes> + (<Starting offset> * 512)) / 512) -1

6597067670016 + (2048*512)/512 – 1 = 12884899840

12884899841 - 1 = 12884899840

Partition Table of the Datastore 'partedUtil getptbl' output below which matches the calculation :

802048 255 63 12884901888
1 2048 12884899840 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Hexdump of the LVM header output of datastore :

00200000 0d d0 01 c0 06 00 00 00 15 00 00 00 01 26 03 00 |.............&..|
00200010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00200020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 62 31 |..............b1|
00200030 37 37 62 33 30 64 37 61 37 62 36 39 35 31 36 63 |77b30d7a7b69516c|
00200040 39 63 65 39 30 30 35 64 66 33 64 33 39 66 53 65 |9ce9005df3d39fSe|
00200050 72 76 65 72 00 00 00 00 00 00 00 02 00 00 00 02 |rver............|
00200060 e0 ff ff 05 00 00 01 00 00 00 02 00 00 00 01 00 |................|
00200070 00 00 07 00 00 00 00 00 00 00 00 00 10 01 00 00 |................|
00200080 00 00 77 b4 43 5b 6c e2 7c a0 45 98 00 ca fe 01 |..w.C[l.|.E.....|
00200090 00 85 9c 9a 72 d8 95 70 05 00 f8 b8 5c 30 ed 84 |....r..p....\0..|
002000a0 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
002000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
002000c0 00 00 00 00 00 00 00 00 00 00 00 10 00 00 ff 5f |..............._|
002000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

Comments

Post a Comment

Popular posts from this blog

Integration with vCloud Director failing after NSXT upgrade to 4.1.2.0 certificate expired

  Issue Clarification: after upgrade from 3.1.3 to 4.1.2.0 observed certificate to be expired related to various internal services.   Issue Verification: after Upgrade from 3.1.3 to 4.1.2.0 observed certificate to be expired related to various internal services.   Root Cause Identification: >>we confirmed the issue to be related to the below KB NSX alarms indicating certificates have expired or are expiring (94898)   Root Cause Justification:   There are two main factors that can contribute to this behaviour: NSX Managers have many certificates for internal services. In version NSX 3.2.1, Cluster Boot Manager (CBM) service certificates were incorrectly given a validity period of 825 days instead of 100 years. This was corrected to 100 years in NSX 3.2.3. However any environment originally installed on NSX 3.2.1 will have the internal CBM Corfu certs expire after 825 regardless of upgrade to the fixed version or not. On NSX-T 3.2.x interna...
Post a Comment
Read more

Calculate how much data can be transferred in 24 hours based on link speed in data center

  In case you are planning for migration via DIA or IPVPN link and as example you have 200Mb stable speed so you could calculate using the below formula. (( 200Mb /8)x60x60x24) /1024/1024 = 2TB /per day In case you have different speed you could replace the 200Mb by any rate to calculate as example below. (( 5 00Mb /8)x60x60x24) /1024/1024 =  5.15TB  /per day So approximate each 100Mb would allow around 1TB per day.
Post a Comment
Read more

Device expanded/shrank messages are reported in the VMkernel log for VMFS-5

    Symptoms A VMFS-5 datastore is no longer visible in vSphere 5 datastores view. A VMFS-5 datastore is no longer mounted in the vSphere 5 datastores view. In the  /var/log/vmkernel.log  file, you see an entry similar to: .. cpu1:44722)WARNING: LVM: 2884: [naa.6006048c7bc7febbf4db26ae0c3263cb:1] Device shrank (actual size 18424453 blocks, stored size 18424507 blocks) A VMFS-5 datastore is mounted in the vSphere 5 datastores view, but in the  /var/log/vmkernel.log  file you see an entry similar to: .. cpu0:44828)LVM: 2891: [naa.6006048c7bc7febbf4db26ae0c3263cb:1] Device expanded (actual size 18424506 blocks, stored size 18422953 blocks)   Purpose This article provides steps to correct the VMFS-5 partition table entry using  partedUtil . For more information see  Using the partedUtil command line utility on ESX and ESXi (1036609) .   Cause The device size discrepancy is caused by an incorrect ending sector for the VMFS-5 partition on the ...
Post a Comment
Read more