VMware ESXi 5.5 Purple Diagnostic Screen Exception 14 in SEsparse and LibAIO (LibAIODrainMergeQueue, LibAIOMergedIODone, SESparseAsyncDataDone) (2073516)

Symptoms

• VMware ESXi 5.5 host fails with a purple diagnostic screen
• You see backtrace similar to:
  
  cpu0:33101)@BlueScreen: #PF Exception 14 in world 33101:memMap-0 IP 0x4180182f4948 addr 0x4108fffffff0 PTEs:0x100088063;0x80000020ad5bf063;0x0;
  cpu0:33101)Code start: 0x418018000000 VMK uptime: 1:09:27:02.593
  cpu0:33101)0x4123c535cb20:[0x4180182f4948]LibAIODrainMergeQueue@vmkernel#nover+0x150 stack: 0x4130002a85c0
  cpu0:33101)0x4123c535cb80:[0x4180182f53fd]LibAIOMergedIODone@vmkernel#nover+0x211 stack: 0x412ec622ef90
  cpu0:33101)0x4123c535cbb0:[0x41801802d21f]AsyncPopCallbackFrameInt@vmkernel#nover+0xe7 stack: 0x1
  cpu0:33101)0x4123c535cbe0:[0x418018bb9798]SESparseAsyncDataDone@esx#nover+0x15c stack: 0x41300007b0c0
  cpu0:33101)0x4123c535cc10:[0x41801802d21f]AsyncPopCallbackFrameInt@vmkernel#nover+0xe7 stack: 0x4123c535cc70
  cpu0:33101)0x4123c535cc40:[0x4180182cdc22]FS_IOAccessDone@vmkernel#nover+0x56 stack: 0x412ec043e340
  cpu0:33101)0x4123c535cc70:[0x41801802d21f]AsyncPopCallbackFrameInt@vmkernel#nover+0xe7 stack: 0x4123c535cc90
  cpu0:33101)0x4123c535cca0:[0x4180182fd5e1]FDSAsyncTokenIODone@vmkernel#nover+0xdd stack: 0x4123c535cdb0
  cpu0:33101)0x4123c535cda0:[0x418018320593]SCSICompleteDeviceCommand@vmkernel#nover+0x46f stack: 0x4123c535cdc0
  cpu0:33101)0x4123c535cdd0:[0x4180187dbebe]nmp_CompleteCommandForDevice@com.vmware.vmkapi#v2_2_0_0+0x82 stack:
  cpu0:33101)0x4123c535cee0:[0x4180187dc18e]nmp_CompleteCommandForPath@com.vmware.vmkapi#v2_2_0_0+0x12a stack: 0
  cpu0:33101)0x4123c535cf60:[0x41801834153a]SCSICompletePathCommand@vmkernel#nover+0x292 stack: 0xe958503533d2
  cpu0:33101)0x4123c535cfd0:[0x41801832eaf3]SCSICompleteAdapterCommand@vmkernel#nover+0x13f stack: 0x4180184a69f
  cpu0:33101)0x4123c535d0f0:[0x4180186d9c13]SCSILinuxWorldletFn@com.vmware.driverAPI#9.2+0x3e3 stack: 0x4123c535
  cpu0:33101)0x4123c535d250:[0x4180180eb521]WorldletProcessQueue@vmkernel#nover+0x779 stack: 0x417f00000000
  cpu0:33101)0x4123c535d290:[0x4180180eb93c]WorldletBHHandler@vmkernel#nover+0x54 stack: 0x4123c535d2d0
  cpu0:33101)0x4123c535d320:[0x41801802e94f]BH_DrainAndDisableInterrupts@vmkernel#nover+0xf3 stack: 0x4123c535d4
  cpu0:33101)0x4123c535d360:[0x418018063e83]IDT_IntrHandler@vmkernel#nover+0x1af stack: 0x4123c535d480
  cpu0:33101)0x4123c535d370:[0x4180180f1064]gate_entry@vmkernel#nover+0x64 stack: 0x4018
  cpu0:33101)0x4123c535d480:[0x4180183a329a]Power_HaltPCPU@vmkernel#nover+0x1fe stack: 0x0
  cpu0:33101)0x4123c535d4f0:[0x41801824e859]CpuSchedIdleLoopInt@vmkernel#nover+0x4bd stack: 0x4123c535d580
  cpu0:33101)0x4123c535d650:[0x418018254930]CpuSchedDispatch@vmkernel#nover+0x1630 stack: 0x0
  cpu0:33101)0x4123c535d6c0:[0x418018255c65]CpuSchedWait@vmkernel#nover+0x245 stack: 0x1
  cpu0:33101)0x4123c535d740:[0x4180182565c4]CpuSched_TimedWait@vmkernel#nover+0xec stack: 0x0
  cpu0:33101)base fs=0x0 gs=0x418040000000 Kgs=0x0
• You observe this issue on a View environment with ESXi 5.5 using SESparse or Reclaim disk space.

Purpose

This article provides information on troubleshooting hosts that fail with a purple diagnostic screen referencing Linux native Asynchronous I/O Interface library LibAIO in the stack trace.

Cause

This issue occurs due to a new feature introduced in the AIO Library to improve SESparse performance and may occur when using SESparse snapshots.

Resolution

This is a known issue affecting VMware ESXi 5.5.

Currently, there is no resolution.

Work around

To work around this issue, you must disable LibAIO coalesce Read and Write.

To disable LibAIO coalesce Read and Write:

1. Run this command on the affected ESXi host to verify current settings for coalesce Read and Write:
   
   esxcfg-advcfg -g /FDS/FDSEnableCoalesceWrites
   esxcfg-advcfg -g /FDS/FDSEnableCoalesceReads
   
   Note: Value 1 indicates Enabled and value 0 indicates Disabled. 
2. If you see the value 1, run this command on the ESXi host to disable the setting:
   
   esxcfg-advcfg -s 0 /FDS/FDSEnableCoalesceWrites
   esxcfg-advcfg -s 0 /FDS/FDSEnableCoalesceReads
   
   Note: Changes to these advanced parameters can be changed in a live system and they are persistent.

From <http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2073516>