Wednesday, November 21, 2012

Windbg SOS CheatSheets


A Couple of Cheatsheets (reference: http://geekswithblogs.net/.NETonMyMind/archive/2006/03/14/72262.aspx and http://blogs.msdn.com/b/alejacma/archive/2009/06/30/sos-cheat-sheet-net-2-0-3-0-3-5.aspx):

Cheatsheet #1

Starting, Attaching, Executing and Exiting

Start -> All Programs -> Debugging Tools for Windows -> WinDbg
F6
attach to process
Ctrl-Break
interrupt debugee
.detach
detach from a process
g
continue debugee execution
q
exit WinDbg


Getting Help
?
help on commands that affect the debugee
.help
help on commands that affect the debugger
.hh command
view the on line help file
!help
help on the extension dll at the top of the chain (e. g., SOS)


Issuing Commands
up arrow, down arrow, enter
scroll through command history
Right mouse button
paste into command window


Examining the Unmanaged Environment
lmf
list loaded modules with full path
lmt
list loaded modules with last modified timestamp
~
list unmanaged threads
~thread s
select a thread for thread specific commands
!token -n
view thread permissions
k
view the unmanaged call stack
!runaway
view thread CPU consumption
bp
set a breakpoint
.dump path
dump small memory image
.dump /ma path
dump complete memory image


Working with Extension DLLs (e. g., SOS)
.chain
list extensions dlls
.load clr10\sos
load SOS for debugging framework 1.0 / 1.1
.unload clr10\sos
unload SOS
.loadby sos mscorwks
load SOS for debugging framework 2.0


SOS Commands
!threads
view managed threads
!clrstack
view the managed call stack
!dumpstack
view combined unmanaged & managed call stack
!clrstack -p
view function call arguments
!clrstack –l
view stack (local) variables
!name2ee module class
view addresses associated with a class or method
!dumpmt –md address
view the method table & methods for a class
!dumpmd address
view detailed information about a method
!do address
view information about an object
!dumpheap –stat
view memory consumption by type
!dumpheap –min size
view memory consumption by object when at least size
!dumpheap –type type
view memory consumption for all objects of type type
!gcroot address
view which object are holding a reference to address
!syncblk
view information about managed locks


SOS 2.0 Commands
!bpmd module method
set breakpoint
!DumpArray address
view contents of an array
!PrintException
view information about most recent exception



Cheatsheet #2
========================================================================
HELP
========================================================================

.hh
  -> Open Debugger.chm.
!SOS.help
  -> See all commands in an extension.
!SOS.help !command
  -> Get help for one command in an extension.

========================================================================
PREPARE THE ENVIRONMENT
========================================================================

---------------
SET SYMBOL PATH
---------------

.sympath
    -> Shows current symbol path.
.sympath srv*c:\symbols*\\symbols\symbols
    -> Sets symbol path.
.sympath+ srv*c:\symbolspub*http://msdl.microsoft.com/downloads/symbols
    -> Adds path to symbol path.
 
--------------------
SET SOURCE CODE PATH
--------------------

.srcpath
    -> Shows current source path.
.srcpath \\dubitgfila04\DevDivServicing\whidbey\50727.42\sources
    -> Sets source path.
.srcpath+ C:\path
    -> Adds path to source path.
 
----------------
LOAD EXTENSIONS
----------------

.chain
    -> Shows loaded extensions. 
!EEVersion
    -> Shows .NET version and SOS version.
.load SOS
    -> Loads extension.
sxe ld mscorwks.dll; g; .loadby SOS mscorwks
sxe -c "" clrn; g; .loadby SOS mscorwks
sxe -c ".loadby sos mscorwks; g" ld mscorwks.dll
    -> Loads the right version of SOS when mscorwks.dll gets loaded.
.unload SOS
    -> Unloads extension

========================================================================
SET A BREAKPOINT
========================================================================

-----------
FIND METHOD
-----------

!DumpDomain
    -> Shows all app domains and their assemblies/modules.
!DumpAssembly <Assembly>
    -> Shows assembly info: name, modules...
!DumpModule -mt <Module>
    -> Shows Method Tables (types) defined and referenced by a module.
!DumpMT -md <Method Table>
    -> Shows Method Descriptors of a Method Table.
!DumpMD <Method Descriptor>
    -> Shows Method Descriptor info. We can see Code Address if method
       is jitted.
!Name2EE <Module> <NameSpace.Class.Method>
!Name2EE <Module>!<NameSpace.Class.Method>
    -> Shows Method Descriptor and jitted Code Address of a method.
!Name2EE * <NameSpace.Class.Method>
!Name2EE *!<NameSpace.Class.Method>
    -> Shows Module, Method Descriptor and jitted Code Address of a
       method.

--------------
SET BREAKPOINT
--------------

!BPMD -md <Method Descriptor>
    -> Sets breakpoint on method by Method Descriptor.
bp <Code Address>
    -> Sets breakpoint on method by jitted Code Address.
!BPMD <Module> <NameSpace.Class.Method>
    -> Sets breakpoint on method by module and name.
bl
    -> Lists breakpoints.
bd 0
bd *
    -> Disables breakpoints.
be 1
    -> Enables breakpoint.
bc 0
    -> Removes breakpoints.

-----------------
STEP THROUGH CODE
-----------------

l+t
    -> Enables source code mode. With private symbols and source code,
       Windbg will open source code files automatically.
l-t
    -> Enables assembly mode.
p
    -> Steps over (F10).
t
    -> Steps into (F11 or F8).

========================================================================
CALL STACKS
========================================================================

--------------------------
INSPECT CURRENT CALL STACK
--------------------------

k
    -> Shows unmanaged + managed call stack.
kp
    -> Shows unmanaged + managed call stack with unmanaged parameters.
!DumpStack
    -> Shows raw call stack with unmanaged + managed calls and managed
       Method Descriptors.
!CLRStack
    -> Shows managed call stack.
!DumpStack -EE
    -> Shows managed call stack with Method Descriptors.
!CLRStack -p
    -> Shows managed call stack with parameters.
!CLRStack -l
    -> Shows managed call stack with locals (we will only see memory
       addresses in the stack).
!CLRStack -a
    -> Shows managed call stack with parameters and locals.
   
---------------
INSPECT OBJECTS
---------------

!DumpObj <Object>
!do <Object>
    -> Shows object info: Method Table, EEClass, size, fields...
!DumpArray <Array>
!da <Array>
    -> Shows array info: # of elements, elements, their type... 

-------------------
INSPECT SOURCE CODE
-------------------

kn
    -> Shows unmanaged + managed call stack with frame numbers.
.frame <Frame Number>
    -> Changes the local context to a specific method frame. With Source
       Code Mode enabled, private symbols and source code, Windbg will
       open the source code file automatically.

------------
INSPECT MSIL
------------

!IP2MD <Instruction Pointer>
!CLRStack; !IP2MD <Instruction Pointer>
!DumpStack -EE
!IP2MD @eip
    -> Gets a Method Descriptor that we can use with !DumpIL.
!DumpIL <Method Descriptor>
    -> Shows IL of a method. ILDASM.exe and Reflector.exe show the same
       IL, but also locals.

-------------------
INSPECT SOURCE CODE
-------------------
lm; !SaveModule <Dll start> c:\sample.dll
    -> Writes a loaded dll to disk.
   
----------------
INSPECT ASSEMBLY
----------------

k
!DumpMD <Method Descriptor>
!DumpStack -EE; !DumpMD <Method Descriptor>
    -> Gets Code Address of an unmanaged or managed jitted method.
u <Code Address>
    -> Shows several lines of assembly code at a given Code Address, and
       source code lines if we have private symbols.
uf <Code Address>
    -> Shows all the assembly code of the managed or unmanaged function
       containing the Code Address, and source code lines if we have
       private symbols.
!u <Code Address>
    -> Shows all the assembly code of the managed method containing the
       Code Address, and marks that address with ">>>".
       If address is in unmanaged method and we have private symbols and
       source code, it shows all the assembly in that method along with
       the source code itself!
      
-----------------
INSPECT REGISTERS
-----------------

r ecx   
    -> When hitting a breakpoint on a managed function, it shows its
       first parameter (_fastcall calling convention). First parameter
       is usually the 'this' pointer.
r edx
    -> When hitting a breakpoint on a managed function, it shows its
       second parameter (_fastcall calling convention).
r esi
    -> Typically used to hold the 'this' pointer as copied from ecx.

========================================================================
THREADS
========================================================================

-------------------
INSPECT ALL THREADS
-------------------

~
    -> Shows all threads (managed and unmanaged).
!runaway
    -> Shows time consumed by each thread.
~0s
    -> Switches to thread 0.
~#s
    -> Switches to the thread where the debugger broke.
~*k100
    -> Shows all unmanaged + managed call stacks in all threads.
!Threads
    -> Shows all managed threads.
~*e !CLRStack
    -> Shows managed calls in all threads.
!Threads -special
    -> Shows all .NET related threads (managed and unmanaged).
!ThreadPool
    -> Shows ThreadPool info: number of work requests in the queue,
       timers and completion port threads.
      
========================================================================
THREAD STACKS
========================================================================

----------------------------
INSPECT OBJECTS IN THE STACK
----------------------------

!DumpStackObjects
!dso
    -> Shows a list of references to objects that are still on the
       thread’s stack.
      
---------------
INSPECT OBJECTS
---------------

!do <Object>
    -> Shows object info: Method Table, EEClass, size, fields...
!do -nofields <Object>
    -> Shows object info but not fields. Good to see i.e. Strings.
!da <Array>
    -> Shows array info: # of elements, elements, their type... 
!DumpVC <Method Table> <Value Type Address>
    -> Shows value type structs ('VT = 1' in '!do').
!DumpClass <EEClass>
    -> Shows the static fields of an object/class ('Attr = static' in
       '!do'). These fields are part of the EEClass, a representation of
       a class created before any method invocations are made on it.
!DumpClass <EEClass>      
!Name2EE <Module> <NameSpace.Class>; !DumpClass <EEClass>
    -> Shows the static fields of a class.
   
========================================================================
MANAGED HEAP
========================================================================

------------
INSPECT HEAP
------------

!EEVersion
    -> Shows version of GC heap (server/workstation) and how many GC
       heaps we have in server mode.
!EEHeap -gc
    -> Shows how many GC heaps there are and how much memory is taken by
       them.
!Threads
    -> Shows if Preemptive GC is enabled on a thread.
   
---------------------------
INSPECT OBJECTS IN THE HEAP
---------------------------

!DumpHeap
    -> Shows all objects in the heap.
!DumpHeap -stat
    -> Shows statistics of all objects in the heap: # of objects of
       a given type/Method Table, etc.
!DumpHeap -mt <Method Table>      
!Name2EE * <NameSpace.Class.Method>; !DumpHeap -mt <Method Table>
    -> Shows all objects in the heap of a given type/Method Table.
!DumpHeap -type <Partial Class Name>
    -> Shows all objects in the heap which class name contains a text.
!DumpHeap -min 85000
!EEHeap -GC; !DumpHeap <LOH begin> <LOH allocated>
    -> Shows all objects in Large Object Heap (LOH/Gen 3).
!FinalizeQueue
    -> Shows which objects are ready for finalization and in general
       which objects in the heap have a Finalize method.
!FinalizeQueue; dd <generation 0 begin> <generation 0 end>-4
    -> Shows all finalizable objects in Gen 0.
!FinalizeQueue; !DumpHeap -mt <Method Table>
    -> Shows all finalizable objects of a given Method Table/type.
!DumpMT -md <Method Table>
    -> Useful to check if an object/class has Finalize and Dispose
       methods.
!FinalizeQueue -detail
    -> Shows SyncBlocks and RuntimeCallableWrappers (RCW) registered
       for finalization.
!GCRoot <Object>
    -> Shows references to an object.
!ObjSize <Object>
    -> Shows the size of an object including its child objects.

------------------------
TROUBLESHOOT HEAP ISSUES
------------------------

!TraverseHeap c:\memory.log
    -> Generates a log file which can be analyzed with CLR Profiler.
!VerifyHeap
    -> Checks the heap for signs of corruption.

---------------------------
TROUBLESHOOT GCHANDLE LEAKS
---------------------------

!GCHandles
    -> Shows statistics of GCHandles.
!GCHandleLeaks
    -> Tracks down GCHandle leaks.
!GCHandleLeaks; !DumpObj poi(<GCHandle>)
    -> Shows the object a GCHandle references.
   
-------------------------------
TROUBLESHOOT LOADER HEAP ISSUES
-------------------------------

!DumpHeap -stat -type Assembly
    -> Shows the number of assemblies in the heap.
!DumpDomain
    -> Shows all assemblies.   
!DumpDomain; !DumpModule <Module>; dc <MetaData start> <MetaData start>+<Metadata size>
    -> Shows metadata of a module in an assembly.   
!EEHeap -loader
    -> Shows Loader Heap size and modules.

========================================================================   
BREAK ON AN EXCEPTION
========================================================================   

-----------------------
BREAK ON CLR EXCEPTIONS
-----------------------

sxe clr
sxe 0xe0434f4d
    -> Break on all CLR exceptions.
!StopOnException -create <Exception Type> 1
!soe -create <Exception Type> 1
    -> Break on a specific CLR exception type.
!soe -derived -create <Base Exception Type> 1
    -> Break on all exceptions which derive from a certain type.
sxn clr
    -> Stops breaking on CLR exceptions. If we used '!soe' before, we
       can still see the exception types of the CLR exceptions that we
       hit, but the debugger won't break on them.
sxr
    -> Resets to defaults.
   
----------------------
INSPECT CLR EXCEPTIONS
----------------------

kp; !do <Object from mscorwks!RaiseTheExceptionInternalOnly>
    -> Shows exception object info.
!PrintException <Object>
!pe <Object>
    -> Shows exception info.
!pe
    -> Shows exception info of last exception thrown in current thread.
!dso; !pe <Object>
    -> Shows exception info of an exception in the stack of current
       thread.
!DumpHeap -type Exception; !DumpHeap -mt <Method Table>; !pe <Object>
    -> Shows exception info of an exception in the heap.
.foreach(ex {!DumpHeap -type <Partial Class Name> -short}){!pe ex;.echo}
    -> Shows exception info of all exceptions of a given type in the
       heap.

-----------------------------------------
INSPECT METHOD WHERE WE GOT THE EXCEPTION
-----------------------------------------

!pe <Object>; !u <Instruction Pointer>; u <Memory Address>
    -> Shows the line of code where the exception happened.
   
========================================================================   
LOCKS
======================================================================== 

---------------------
INSPECT MANAGED LOCKS
---------------------

!SyncBlk
    -> Shows the Index of the syncblock in the syncblock table, the
       address of the syncblock, the thread holding it & the object
       which syncblock we are waiting for. MonitorHeld = 1 for each
       owner and 2 for each waiter. Used with i.e.
       System.Threading.Monitor.
~*kp; ~*e !CLRStack -p; ~*e !dso; !SyncBlk
    -> Helps to find out which threads are waiting on the syncblock of
       which object.

-----------------------
INSPECT UNMANAGED LOCKS
-----------------------

!locks
!critsec
!SIEExt.critlist
    -> Shows info of a critical sections (unmanaged). Use with i.e.      
       System.Threading.Mutex (not based on syncblocks).

SPSolutionExport Exception SecurityTokenService.Issue()

In trying to save a site using the Site Settings -> Save as Template, I received the following error:

11/15/2012 15:57:26.84  w3wp.exe (0x1A64)                        0x1944 SharePoint Foundation          General                        c42u Monitorable SPSolutionExporter: System.InvalidOperationException: The Writer is closed or in error state.     at System.Xml.XmlWellFormedWriter.AdvanceState(Token token)     at System.Xml.XmlWellFormedWriter.WriteEndElement()     at Microsoft.SharePoint.ScopedXmlWriterElement.Closer(XmlWriter writer, ScopedObjectUsage`1 wrapper)     at Microsoft.SharePoint.ScopedObjectUsage`1.Dispose(Boolean isDisposing)     at Microsoft.SharePoint.ScopedObjectUsage`1.Dispose()     at Microsoft.SharePoint.SPSolutionExporter.ExportContentTypes(SPContentTypeCollection contentTypes, String partitionName)     at Microsoft.SharePoint.SPSolutionExporter.ExportLists()     at Microsoft.SharePoint.SPSolutionExporter.GenerateSolutionFiles()     at Microsoft.SharePoint.SPSolutionExporter.ExportWebAsSolution() 2cf55a43-6bc9-47cb-82a2-27fbd38d452b
11/15/2012 15:57:26.86  w3wp.exe (0x1A64)                        0x1944 SharePoint Foundation          Runtime                        tkau Unexpected System.InvalidOperationException: Error generating solution files in temporary directory.    at Microsoft.SharePoint.SPSolutionExporter.ExportWebAsSolution()     at Microsoft.SharePoint.SPSolutionExporter.ExportWebToGallery(SPWeb web, String solutionFileName, String title, String description, ExportMode exportMode, Boolean includeContent, String workflowTemplateName, String destinationListUrl)     at Microsoft.SharePoint.SPSolutionExporter.ExportWebToGallery(SPWeb web, String solutionFileName, String title, String description, ExportMode exportMode, Boolean includeContent)     at Microsoft.SharePoint.ApplicationPages.SaveAsTemplatePage.BtnSaveAsTemplate_Click(Object sender, EventArgs e)     at System.Web.UI.WebControls.Button.OnClick(EventArgs e)     at System.Web.UI.WebControls.Button.Rai... 2cf55a43-6bc9-47cb-82a2-27fbd38d452b
11/15/2012 15:57:26.86* w3wp.exe (0x1A64)                        0x1944 SharePoint Foundation          Runtime                        tkau Unexpected ...sePostBackEvent(String eventArgument)     at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) 2cf55a43-6bc9-47cb-82a2-27fbd38d452b
11/15/2012 15:57:26.87  w3wp.exe (0x1A64)                        0x1944 SharePoint Foundation          Monitoring                     b4ly High     Leaving Monitored Scope (Request (POST:https://workflow-int.curtin.edu.au:443/sites/test/_layouts/savetmpl.aspx)). Execution Time=5500.39376512937 2cf55a43-6bc9-47cb-82a2-27fbd38d452b
In addition it appeared that the SecurityTokenService was also throwing exceptions issueing tokens.

When visiting the service directly I recieved the following message:
The service encountered an error.
An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is:
System.InvalidOperationException: An exception was thrown in a call to a policy export extension.
Extension: System.ServiceModel.Channels.TransportSecurityBindingElement
Error: Security policy export failed. The binding contains a TransportSecurityBindingElement but no transport binding element that implements ITransportTokenAssertionProvider. Policy export for such a binding is not supported. Make sure the transport binding element in the binding implements the ITransportTokenAssertionProvider interface. ----> System.InvalidOperationException: Security policy export failed. The binding contains a TransportSecurityBindingElement but no transport binding element that implements ITransportTokenAssertionProvider. Policy export for such a binding is not supported. Make sure the transport binding element in the binding implements the ITransportTokenAssertionProvider interface.
   at System.ServiceModel.Channels.TransportSecurityBindingElement.System.ServiceModel.Description.IPolicyExportExtension.ExportPolicy(MetadataExporter exporter, PolicyConversionContext policyContext)
   at System.ServiceModel.Description.MetadataExporter.ExportPolicy(ServiceEndpoint endpoint)
   --- End of inner ExceptionDetail stack trace ---

3 Steps were needed to resolve the issue:

Step 1. Ensure that SecurityTokenService is responding to requests on all SharePoint farm servers, then go to IIS manager and edit the SecurityTokenService web.config by replacing it with the content below:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.serviceModel>
    <!-- Behavior List: -->
    <behaviors>
      <serviceBehaviors>
        <behavior name="SecurityTokenServiceBehavior">
          <!-- The serviceMetadata behavior allows one to enable metadata (endpoints, bindings, services) publishing.
               This configuration enables publishing of such data over HTTP GET.
               This does not include metadata about the STS itself such as Claim Types, Keys and other elements to establish a trust.
          -->
          <serviceMetadata httpGetEnabled="true" />
          <!-- Default WCF throttling limits are too low -->
          <serviceThrottling maxConcurrentCalls="65536" maxConcurrentSessions="65536" maxConcurrentInstances="65536" />
  <serviceDebug includeExceptionDetailInFaults="True" httpHelpPageEnabled="True" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <!-- Service List: -->
    <services>
      <service name="Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceContract" behaviorConfiguration="SecurityTokenServiceBehavior">
        <!-- This is the HTTP endpoint that supports clients requesing tokens. This endpoint uses the default
             standard ws2007HttpBinding which requires that clients authenticate using their Windows credentials. -->
        <endpoint address="" binding="customBinding" bindingConfiguration="spStsBinding" contract="Microsoft.IdentityModel.Protocols.WSTrust.IWSTrust13SyncContract" />
        <!-- This is the HTTP endpoint that supports clients requesting service tokens. -->
        <endpoint name="ActAs" address="actas" binding="customBinding" bindingConfiguration="spStsActAsBinding" contract="Microsoft.IdentityModel.Protocols.WSTrust.IWSTrust13SyncContract" />
        <!-- This is the HTTP endpoint that supports IMetadataExchange. -->
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
      </service>
      <service name="Microsoft.SharePoint.Administration.Claims.SPWindowsTokenCacheService">
        <endpoint address="" binding="customBinding" bindingConfiguration="SPWindowsTokenCacheServiceHttpsBinding" contract="Microsoft.SharePoint.Administration.Claims.ISPWindowsTokenCacheServiceContract" />
      </service>
    </services>
    <!-- Binding List: -->
    <bindings>
      <customBinding>
        <binding name="spStsBinding">
          <binaryMessageEncoding>
            <readerQuotas maxStringContentLength="1048576" maxArrayLength="2097152" />
          </binaryMessageEncoding>
          <httpTransport maxReceivedMessageSize="2162688" authenticationScheme="Negotiate" useDefaultWebProxy="false" />
        </binding>
        <binding name="spStsActAsBinding">
          <security authenticationMode="SspiNegotiatedOverTransport" allowInsecureTransport="true" defaultAlgorithmSuite="Basic256Sha256" messageSecurityVersion="WSSecurity11WSTrust13WSSecureConversation13WSSecurityPolicy12" />
          <binaryMessageEncoding>
            <readerQuotas maxStringContentLength="1048576" maxArrayLength="2097152" />
          </binaryMessageEncoding>
          <httpTransport maxReceivedMessageSize="2162688" authenticationScheme="Negotiate" useDefaultWebProxy="false" />
        </binding>
        <binding name="SPWindowsTokenCacheServiceHttpsBinding">
          <security authenticationMode="IssuedTokenOverTransport" />
          <textMessageEncoding>
            <readerQuotas maxStringContentLength="1048576" maxArrayLength="2097152" />
          </textMessageEncoding>
          <httpsTransport maxReceivedMessageSize="2162688" authenticationScheme="Anonymous" useDefaultWebProxy="false" />
        </binding>
      </customBinding>
    </bindings>
  </system.serviceModel>
  <system.webServer>
    <security>
      <authentication>
        <anonymousAuthentication enabled="true" />
        <windowsAuthentication enabled="true">
          <providers>
            <clear />
            <add value="Negotiate" />
            <add value="NTLM" />
          </providers>
        </windowsAuthentication>
      </authentication>
    </security>
    <modules>
      <add name="WindowsAuthenticationModule" />
    </modules>
  </system.webServer>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="10000" />
    </connectionManagement>
  </system.net>
    <system.web>
        <identity impersonate="false" />
    </system.web>
</configuration>

Once the changes are complete


Step 2:

On all IIS servers ensure that under Web Services root => IIS - Authentication => only Windows and Anonymous authentication are "Enabled". Then progress down to the SecureTokenService virtual directory and ensure the same. This had forms authentication enabled on only one of my farm servers.

Step 3:

Delete and reprovision the UserProfileService. You may notice when you delete it, you can already access the sites again. This is because the user profile service was failing to communicate with the SecureTokenService on the app server.

Step 4:

Test this by building a new web application and site collection. Ensuring that no existing content is responsible for the failure to save template.

Note: In my situation I found the content type hub was also responsible publishing broken content types for which the "Save as Template" command could not save.
 

Using WinDBG to Debug SharePoint & Cheatsheet

Loading SOS


For sometime now I have relied on windbg to show me exceptions occuring in SharePoint.

You can attach windbg to any process and view the .NET exceptions occuring at runtime between the process and the CLR, even handled exceptions.

Usually the true source of the error is hidden in a wrapper and rethrown as an irrelevant exception. Thank you SharePoint.

In order to view these exceptions we can attach windbg to either the w3wp.exe related to your IIS website running the SharePoint site by clicking Start => Run => inetmgr => Click on server => Worker processes, or by powershell if you are executing or can execute the relevant command through powershell you can attach the to powershell.exe process.

After launching windbg and attaching to a process we need to load the Sons of Strike SOS.dll to enable viewing managed exceptions in windbg, without the SOS dll we can only view native code.

To do this within the windbg command prompt of the attached process type:

For .NET 2.0:

.load C:\Windows\Microsoft.NET\Framework\v2.50727\SOS.dll
.load C:\Windows\Microsoft.NET\Framework\v2.50727\mscorwks.dll


For .NET 4.0:

.load C:\Windows\Microsoft.NET\Framework\v4.0.30319\SOS.dll
.load C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorwks.dll


Once we are attached, I try to use the notify on exception but you can also choose to break on exception.


To notify on .NET exceptions and print the exception (!pe) we use:
sxn -c "!pe" clr


To break on .NET exceptions and print the exception (!pe) we use:
sxe clr
!pe


We can also use !do <address> to dump objects, this is helpful if you see a System.String variable and want to see its content. We can also view the call stack at the given point by calling:
!clrstack
or
!clrstack -p

The -p switch will print out all the stack including method parameters. You can then use !do to dump the parameter.

A complete dump of all strings in the heap can be achieved by running the following command:

.foreach(myVariable {!dumpheap -type System.String -min 6500 -short}){!do myVariable;.echo *************}

Using a Crash Dump

If you obtain a crash dump instead of a running process you can also use the following commands to view the CLR stack:

Step 1: Start windbg and Load sos.dll, mscorwks.dll and mscordacwks.dll
Step 2: !runaway
Step 3: ~<threadnumber>
Step 4: !clrstack

Example using Dumpmt



If you want to chase down the owning class/method:

!clrstack -all

produces

0:000> !clrstack -all
Thread 0
ESP       EIP    
0012f5e0  77e73887 [FRAME: HelperMethodFrame]
0012f60c  06d3025f [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Doh
                                       (String,ValueClass ExceptApp.Days)
   at [+0x67] [+0x16] c:\junk\cruft\exceptapp\class1.cs:14
    PARAM: this: 0x04a41b5c (ExceptApp.DoSomething)
    PARAM: value class ExceptApp.Days StrParam
    PARAM: unsigned int8 ValueParam: 0x07
0012f630  06d301e2 [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Reh
                                                   (I4,String)
  at [+0x6a] [+0x2b] c:\junk\cruft\exceptapp\class1.cs:23
    PARAM: this: 0x04a41b5c (ExceptApp.DoSomething)
    PARAM: class System.String i: 0x00000042
    PARAM: int8 StrParam: 77863812
    LOCAL: class System.String s: 0x04a45670 (System.String)
    LOCAL: value class ExceptApp.Days e: 0x003e5278 0x0012f63c

 Then to dump the class method descriptor:

!dumpmt 0x003e5278

produces

0:000> !dumpmt 0x003e5278 
EEClass : 06c03b1c
Module : 001521a0
Name: ExceptApp.Days
mdToken: 02000002  (D:\Dev\ExceptApp\bin\Debug\ExceptApp.exe)
MethodTable Flags : 80000
Number of IFaces in IFaceMap : 3
Interface Map : 003e5380
Slots in VTable : 55

 then you can use !do to dump objects such as strings or use !dumpallstackobjects

Some objects in the heap may be of type System.Exception just as with strings you can dump the exception using !do for example:

0:000> !dumpobj 04a45f64 
Name: System.ArgumentException
MethodTable 0x79b87b84
EEClass 0x79b87c0c
Size 68(0x44) bytes
mdToken: 02000038  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 79b87c70
      MT    Field   Offset          Type       Attr    Value Name
79b7fcd4  400001d        4         CLASS   instance 00000000 _className
79b7fcd4  400001e        8         CLASS   instance 00000000
                                                        exceptionMethod
79b7fcd4  400001f        c         CLASS   instance 00000000 
                                                 _exceptionMethodString
79b7fcd4  4000020       10         CLASS   instance 04a456cc _message
79b7fcd4  4000021       14         CLASS   instance 00000000 
                                                        _innerException
79b7fcd4  4000022       18         CLASS   instance 00000000 _helpURL
79b7fcd4  4000023       1c         CLASS   instance 00000000 
                                                            _stackTrace
79b7fcd4  4000024       20         CLASS   instance 00000000
                                                      _stackTraceString
79b7fcd4  4000025       24         CLASS   instance 00000000
                                                _remoteStackTraceString
79b7fcd4  4000026       2c  System.Int32   instance        0
                                                      _remoteStackIndex
79b7fcd4  4000027       30  System.Int32   instance -2147024809 
                                                               _HResult
79b7fcd4  4000028       28         CLASS   instance 00000000 _source
79b7fcd4  4000029       34  System.Int32   instance        0 _xptrs
79b7fcd4  400002a       38  System.Int32   instance -532459699 _xcode
79b87b84  40000d7       3c         CLASS   instance 04a45708 
                                                            m_paramName
 

New Commands in Framework 4.0

We will review the WinDbg Extension SOS.dll in .NET Framework 4.0 CTP. CLR 4.0 has renamed runtime dll from mscorwks.dll to CLR.DLL, that’s really helpful.
loading SOS dll depending on the location of .net 4.0 runtime aka CLR.DLL, execute the following command
.loadby sos clr
1.  DML Support – YES, finally.  SOS supports DML in .NET 1.1 but it was gone in clr 2.0.  Silverlight CoreCLR supports DML and now .NET framework 4.0 supports it as well.
Execute the following command to turn on DMLfor every command or use /D option
0:003> .prefer_dml 1
DML versions of commands on by default
0:003> !dumpheap /D -type Exception -stat
For people new to WinDbg, Why am I so excited about DML support in SOS?
DML Snapshot
If you look at the above snapshot, you have the link for each MethodTable address which you can just click on to execute the command. No need to type, however not every commands will have the DML support but !dumpobject is another important one, you can just click on object address to dump an object from GC Heap.
2. The following additional extension commands are added
Examining code and stacks
!ThreadState
Examining CLR data structures
!DumpSigElem
Diagnostic Utilities
!VerifyObj
!FindRoots
!HeapStat
!GCWhere
!ListNearObj (lno)
!AnalyzeOOM (ao)
Examining the GC history
!HistInit
!HistStats
!HistRoot
!HistObj
!HistObjFind
!HistClear

!ThreadState Command
When you execute !threads command, you will see the similar output as shown below
PreEmptive   GC Alloc           Lock
ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
0    1  310 00161438      a020 Enabled  013b4c64:013b5fe8 00159230     1 MTA
2    2  8c4 0016dab0      b220 Enabled  00000000:00000000 00159230     0 MTA (Finalizer)
First column is your debugger thread id and the second column ID is ManagedThread ID, OSID column is OS thread ID so that means OSID column will be 0 or some garbage when a runtime uses Fiber.
You will see the State column which is a bit flag as shown below(taken from Shared CLI)
TS_Unknown                = 0×00000000,    // threads are initialized this way
TS_AbortRequested         = 0×00000001,    // Abort the thread
TS_GCSuspendPending       = 0×00000002,    // waiting to get to safe spot for GC
TS_UserSuspendPending     = 0×00000004,    // user suspension at next opportunity
TS_DebugSuspendPending    = 0×00000008,    // Is the debugger suspending threads?
TS_GCOnTransitions        = 0×00000010,    // Force a GC on stub transitions (GCStress only)
TS_SuspendUnstarted       = 0×00400000,    // latch a user suspension on an unstarted thread
TS_ThreadPoolThread       = 0×00800000,    // is this a threadpool thread?
TS_TPWorkerThread         = 0×01000000,    // is this a threadpool worker thread?
TS_Interruptible          = 0×02000000,    // sitting in a Sleep(), Wait(), Join()
TS_Interrupted            = 0×04000000,    // was awakened by an interrupt APC. !!! This can be moved to TSNC
TS_CompletionPortThread   = 0×08000000,    // Completion port thread
………………………………………………………………………..
SOS in CLR4.0 has !threadstate command, which tells you exactly the state of the thread given the bit field, the following output shows you the threadstate bit for Worker Thread, Completion Port Thread and Finalizer Thread
0:000> !ThreadState 1009220
Legal to Join
Background
CLR Owns
In Multi Threaded Apartment
Thread Pool Worker Thread
0:000> !ThreadState 800a220
Legal to Join
Background
CoInitialized
In Multi Threaded Apartment
Completion Port Thread
0:000> !ThreadState b220
Legal to Join
Background
CLR Owns
CoInitialized
In Multi Threaded Apartment
Other Important Commands
!findroots – This is a very powerful and interesting command, because it allows you to break into debugee when CLR garbage collect generational objects.
!GCWhere - tells you the generation number along with the GC heap segment, you no longer need to map the object address with the GC heap segment or use any other extension dll
!HeapStat- This is another cool command, this command displays the stat on generational heap including generation sizes
!AnalyzeOOM – displays the detailed informatin on Last System.OutOfMemoryException
I can’t do justice on detailed documentation for each of these commands because SOS !help documentation has done a very good job. You can either look at !help documentation  or read below. I am just copying and pasting from SOS Help documentation
0:020> !help ThreadState
——————————————————————————-
!ThreadState value
The !Threads command outputs, among other things, the state of the thread.
This is a bit field which corresponds to various states the thread is in.
To check the state of the thread, simply pass that bit field from the
output of !Threads into !ThreadState.
Example:
0:003> !Threads
ThreadCount:      2
UnstartedThread:  0
BackgroundThread: 1
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
PreEmptive   GC Alloc           Lock
ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
0    1  250 0019b068      a020 Disabled 02349668:02349fe8 0015def0     0 MTA
2    2  944 001a6020      b220 Enabled  00000000:00000000 0015def0     0 MTA (Finalizer)
0:003> !ThreadState b220
Legal to Join
Background
CLR Owns
CoInitialized
In Multi Threaded Apartment
Possible thread states:
Thread Abort Requested
GC Suspend Pending
User Suspend Pending
Debug Suspend Pending
GC On Transitions
Legal to Join
Yield Requested
Hijacked by the GC
Blocking GC for Stack Overflow
Background
Unstarted
Dead
CLR Owns
CoInitialized
In Single Threaded Apartment
In Multi Threaded Apartment
Reported Dead
Task Reset
Sync Suspended
Debug Will Sync
Stack Crawl Needed
Suspend Unstarted
Aborted
Thread Pool Worker Thread
Interruptible
Interrupted
Completion Port Thread
Abort Initiated
Finalized
Failed to Start
Detached
0:020> !help DumpSigElem
——————————————————————————-
!DumpSigElem <sigaddr> <moduleaddr>
This command dumps a single element of a signature object.  For most circumstances,
you should use !DumpSig to look at individual signature objects, but if you find a
signature that has been corrupted in some manner you can use !DumpSigElem to read out
the valid portions of it.
If we look at a valid signature object for a method we see the following:
0:000> !dumpsig 0x000007fe`ec20879d 0x000007fe`eabd1000
[DEFAULT] [hasThis] Void (Boolean,String,String)
We can look at the individual elements of this object by adding the offsets into the
object which correspond to the return value and parameters:
0:000> !dumpsigelem 0x000007fe`ec20879d+2 0x000007fe`eabd1000
Void
0:000> !dumpsigelem 0x000007fe`ec20879d+3 0x000007fe`eabd1000
Boolean
0:000> !dumpsigelem 0x000007fe`ec20879d+4 0x000007fe`eabd1000
String
0:000> !dumpsigelem 0x000007fe`ec20879d+5 0x000007fe`eabd1000
String
We can do something similar for fields.  Here is the full signature of a field:
0:000> !dumpsig 0x000007fe`eb7fd8cd 0x000007fe`eabd1000
[FIELD] ValueClass System.RuntimeTypeHandle
Using !DumpSigElem we can find the type of the field by adding the offset of it (1) to
the address of the signature:
0:000> !dumpsigelem 0x000007fe`eb7fd8cd+1 0x000007fe`eabd1000
ValueClass System.RuntimeTypeHandle
!DumpSigElem will also work with generics.  Let a function be defined as follows:
public A Test(IEnumerable<B> n)
The elements of this signature can be obtained by adding offsets into the signature
when calling !DumpSigElem:
0:000> !dumpsigelem 00000000`00bc2437+2 000007ff00043178
__Canon
0:000> !dumpsigelem 00000000`00bc2437+4 000007ff00043178
Class System.Collections.Generic.IEnumerable`1<__Canon>
The actual offsets that you should add are determined by the contents of the
signature itself.  By trial and error you should be able to find various elements
of the signature.
0:020> !help VerifyObj
——————————————————————————-
!VerifyObj <object address>
!VerifyObj is a diagnostic tool that checks the object that is passed as an
argument for signs of corruption.
0:002> !verifyobj 028000ec
object 0x28000ec does not have valid method table
0:002> !verifyobj 0680017c
object 0x680017c: bad member 00000001 at 06800184
0:020> !help FindRoots
——————————————————————————-
!FindRoots -gen <N> | -gen any | <object address>
The “-gen” form causes the debugger to break in the debuggee on the next
collection of the specified generation.  The effect is reset as soon as the
break occurs, in other words, if you need to break on the next collection you
would need to reissue the command.
The last form of this command is meant to be used after the break caused by the
other forms has occurred.  Now the debuggee is in the right state for
!FindRoots to be able to identify roots for objects from the current condemned
generations.
!FindRoots is a diagnostic command that is meant to answer the following
question:
“I see that GCs are happening, however my objects have still not been
collected. Why? Who is holding onto them?”
The process of answering the question would go something like this:
1. Find out the generation of the object of interest using the !GCWhere
command, say it is gen 1:
!GCWhere <object address>
2. Instruct the runtime to stop the next time it collects that generation using
the !FindRoots command:
!FindRoots -gen 1
g
3. When the next GC starts, and has proceeded past the mark phase a CLR
notification will cause a break in the debugger:
(fd0.ec4): CLR notification exception – code e0444143 (first chance)
CLR notification: GC – end of mark phase.
Condemned generation: 1.
4. Now we can use the !FindRoots <object address> to find out the cross
generational references to the object of interest.  In other words, even if the
object is not referenced by any “proper” root it may still be referenced by an
older object (from an older generation), from a generation that has not yet been
scheduled for collection.  At this point !FindRoots will search those older
generations too, and report those roots.
0:002> !findroots 06808094
older generations::Root:  068012f8(AAA.Test+a)->
06808094(AAA.Test+b)
0:020> !help HeapStat
——————————————————————————-
!HeapStat [-inclUnrooted | -iu]
This command shows the generation sizes for each heap and the total, how much free
space there is in each generation on each heap.  If the -inclUnrooted option is
specified the report will include information about the managed objects from the
GC heap that are not rooted anymore.
Sample output:
0:002> !heapstat
Heap     Gen0         Gen1         Gen2         LOH
Heap0    177904       12           306956       8784
Heap1    159652       12           12           16
Total    337556       24           306968       8800
Free space:                                                 Percentage
Heap0    28           12           12           64          SOH:  0% LOH:  0%
Heap1    104          12           12           16          SOH:  0% LOH:100%
Total    132          24           24           80
0:002> !heapstat -inclUnrooted
Heap     Gen0         Gen1         Gen2         LOH
Heap0    177904       12           306956       8784
Heap1    159652       12           12           16
Total    337556       24           306968       8800
Free space:                                                 Percentage
Heap0    28           12           12           64          SOH:  0% LOH:  0%
Heap1    104          12           12           16          SOH:  0% LOH:100%
Total    132          24           24           80
Unrooted objects:                                           Percentage
Heap0    152212       0            306196       0           SOH: 94% LOH:  0%
Heap1    155704       0            0            0           SOH: 97% LOH:  0%
Total    307916       0            306196       0
The percentage column contains a breakout of free or unrooted bytes to total bytes.
0:020> !help GCWhere
——————————————————————————-
!GCWhere <object address>
!GCWhere displays the location in the GC heap of the argument passed in.
0:002> !GCWhere 02800038
Address  Gen Heap segment  begin    allocated size
02800038 2    0   02800000 02800038 0282b740  12
When the argument lies in the managed heap, but is not a valid *object* address
the “size” is displayed as 0:
0:002> !GCWhere 0280003c
Address  Gen Heap segment  begin    allocated size
0280003c 2    0   02800000 02800038 0282b740  0
0:020> !help ListNearObj
——————————————————————————-
!ListNearObj <object address>
!ListNearObj is a diagnostic tool that displays the object preceeding and
succeeding the address passed in:
The command looks for the address in the GC heap that looks like a valid
beginning of a managed object (based on a valid method table) and the object
following the argument address.
0:002> !ListNearObj 028000ec
Before: 0x28000a4           72 (0×48      ) System.StackOverflowException
After:  0×2800134           72 (0×48      ) System.Threading.ThreadAbortException
Heap local consistency confirmed.
0:002> !ListNearObj 028000f0
Before: 0x28000ec           72 (0×48      ) System.ExecutionEngineException
After:  0×2800134           72 (0×48      ) System.Threading.ThreadAbortException
Heap local consistency confirmed.
The command considers the heap as “locally consistent” if:
prev_obj_addr + prev_obj_size = arg_addr && arg_obj + arg_size = next_obj_addr
OR
prev_obj_addr + prev_obj_size = next_obj_addr
When the condition is not satisfied:
0:002> !lno 028000ec
Before: 0x28000a4           72 (0×48      ) System.StackOverflowException
After:  0×2800134           72 (0×48      ) System.Threading.ThreadAbortException
Heap local consistency not confirmed.
0:020> !help AnalyzeOOM
——————————————————————————-
!AnalyzeOOM
!AnalyzeOOM displays the info of the last OOM occured on an allocation request to
the GC heap (in Server GC it displays OOM, if any, on each GC heap).
To see the managed exception(s) use the !Threads command which will show you
managed exception(s), if any, on each managed thread. If you do see an
OutOfMemoryException exception you can use the !PrintException command on it.
To get the full callstack use the “kb” command in the debugger for that thread.
For example, to display thread 3′s stack use ~3kb.
OOM exceptions could be because of the following reasons:
1) allocation request to GC heap
in which case you will see JIT_New* on the call stack because managed code called new.
2) other runtime allocation failure
for example, failure to expand the finalize queue when GC.ReRegisterForFinalize is
called.
3) some other code you use throws a managed OOM exception
for example, some .NET framework code converts a native OOM exception to managed
and throws it.
The !AnalyzeOOM command aims to help you with investigating 1) which is the most
difficult because it requires some internal info from GC. The only exception is
we don’t support allocating objects larger than 2GB on CLR v2.0 or prior. And this
command will not display any managed OOM because we will throw OOM right away
instead of even trying to allocate it on the GC heap.
There are 2 legitimate scenarios where GC would return OOM to allocation requests -
one is if the process is running out of VM space to reserve a segment; the other
is if the system is running out physical memory (+ page file if you have one) so
GC can not commit memory it needs. You can look at these scenarios by using performance
counters or debugger commands. For example for the former scenario the “!address
-summary” debugger command will show you the largest free region in the VM. For
the latter scenario you can look at the “Memory\% Committed Bytes In Use” see
if you are running low on commit space. One important thing to keep in mind is
when you do this kind of memory analysis it could an aftereffect and doesn’t
completely agree with what this command tells you, in which case the command should
be respected because it truly reflects what happened during GC.
The other cases should be fairly obvious from the callstack.
Sample output:
0:011> !ao
———Heap 2 ———
Managed OOM occured after GC #28 (Requested to allocate 1234 bytes)
Reason: Didn’t have enough memory to commit
Detail: SOH: Didn’t have enough memory to grow the internal GC datastructures (800000 bytes) -
on GC entry available commit space was 500 MB
———Heap 4 ———
Managed OOM occured after GC #12 (Requested to allocate 100000 bytes)
Reason: Didn’t have enough memory to allocate an LOH segment
Detail: LOH: Failed to reserve memory (16777216 bytes)
0:020> !help FAQ
——————————————————————————-
>> Where can I get the right version of SOS for my build?
If you are running version 1.1 or 2.0 of the CLR, SOS.DLL is installed in the
same directory as the main CLR dll (CLR.DLL). Newer versions of the
Windows Debugger provide a command to make it easy to load the right copy of
SOS.DLL:
“.loadby sos clr”
That will load the SOS extension DLL from the same place that CLR.DLL is
loaded in the process. You shouldn’t attempt to use a version of SOS.DLL that
doesn’t match the version of CLR.DLL. You can find the version of
CLR.DLL by running
“lmvm clr”
in the debugger.  Note that if you are running CoreCLR (e.g. Silverlight)
then you should replace “clr” with “coreclr”.
If you are using a dump file created on another machine, it is a little bit
more complex. You need to make sure the mscordacwks.dll file that came with
that install is on your symbol path, and you need to load the corresponding
version of sos.dll (typing .load <full path to sos.dll> rather than using the
.loadby shortcut). Within the Microsoft corpnet, we keep tagged versions
of mscordacwks.dll, with names like mscordacwks_<architecture>_<version>.dll
that the Windows Debugger can load. If you have the correct symbol path to the
binaries for that version of the Runtime, the Windows Debugger will load the
correct mscordacwks.dll file.
>> I have a chicken and egg problem. I want to use SOS commands, but the CLR
isn’t loaded yet. What can I do?
In the debugger at startup you can type:
“sxe clrn”
Let the program run, and it will stop with the notice
“CLR notification: module ‘mscorlib’ loaded”
At this time you can use SOS commands. To turn off spurious notifications,
type:
“sxd clrn”
>> I got the following error message. Now what?
0:000> .loadby sos clr
0:000> !DumpStackObjects
Failed to find runtime DLL (clr.dll), 0×80004005
Extension commands need clr.dll in order to have something to do.
0:000>
This means that the CLR is not loaded yet, or has been unloaded. You need to
wait until your managed program is running in order to use these commands. If
you have just started the program a good way to do this is to type
bp clr!EEStartup “g @$ra”
in the debugger, and let it run. After the function EEStartup is finished,
there will be a minimal managed environment for executing SOS commands.
>> I have a partial memory minidump, and !DumpObj doesn’t work. Why?
In order to run SOS commands, many CLR data structures need to be traversed.
When creating a minidump without full memory, special functions are called at
dump creation time to bring those structures into the minidump, and allow a
minimum set of SOS debugging commands to work. At this time, those commands
that can provide full or partial output are:
CLRStack
Threads
Help
PrintException
EEVersion
For a minidump created with this minimal set of functionality in mind, you
will get an error message when running any other commands. A full memory dump
(obtained with “.dump /ma <filename>” in the Windows Debugger) is often the
best way to debug a managed program at this level.
>> What other tools can I use to find my bug?
Turn on Managed Debugging Assistants. These enable additional runtime diagnostics,
particularly in the area of PInvoke/Interop. Adam Nathan has written some great
information about that:
http://blogs.msdn.com/adam_nathan/
>> Does SOS support DML?
Yes.  SOS respects the .prefer_dml option in the debugger.  If this setting is
turned on, then SOS will output DML by default.  Alternatively, you may leave
it off and add /D to the beginning of a command to get DML based output for it.
Not all SOS commands support DML output.
0:020> !help HistInit
——————————————————————————-
!HistInit
Before running any of the Hist – family commands you need to initialize the SOS
structures from the stress log saved in the debuggee.  This is achieved by the
HistInit command.
Sample output:
0:001> !HistInit
Attempting to read Stress log
STRESS LOG:
facilitiesToLog  = 0xffffffff
levelToLog       = 6
MaxLogSizePerThread = 0×10000 (65536)
MaxTotalLogSize = 0×1000000 (16777216)
CurrentTotalLogChunk = 9
ThreadsWithLogs  = 3
Clock frequency  = 3.392 GHz
Start time         15:26:31
Last message time  15:26:56
Total elapsed time 25.077 sec
……………………………….
—————————- 2407 total entries —————————–
SUCCESS: GCHist structures initialized
0:020> !help HistStats
——————————————————————————-
!HistStats
HistStat provides a number of garbage collection statistics obtained from the
stress log.
Sample output:
0:003> !HistStats
GCCount    Plugs Promotes   Relocs
———————————–
2296        0       35       86
2295        0       34       85
2294        0       34       85

2286        0       32       83
2285        0       32       83
322        0       23       55
0        0        0        0
Root 01e411b8 relocated multiple times in gc 322
Root 01e411bc relocated multiple times in gc 322

Root 01e413f8 relocated multiple times in gc 322
Root 01e413fc relocated multiple times in gc 322
0:020> !help histroot
——————————————————————————-
!HistRoot <root>
The root value obtained from !HistObjFind can be used to track the movement of
an object through the GCs.
HistRoot provides information related to both promotions and relocations of the
root specified as the argument.
0:003> !HistRoot 01e411b8
GCCount    Value       MT Promoted?                Notes
———————————————————
2296 028970d4 5b6c5cd8       yes
2295 028970d4 5b6c5cd8       yes
2294 028970d4 5b6c5cd8       yes
2293 028970d4 5b6c5cd8       yes
2292 028970d4 5b6c5cd8       yes
2291 028970d4 5b6c5cd8       yes
2290 028970d4 5b6c5cd8       yes
2289 028970d4 5b6c5cd8       yes
2288 028970d4 5b6c5cd8       yes
2287 028970d4 5b6c5cd8       yes
2286 028970d4 5b6c5cd8       yes
2285 028970d4 5b6c5cd8       yes
322 028970e8 5b6c5cd8       yes Duplicate promote/relocs

0:020> !help HistObj
——————————————————————————-
!HistObj <obj_address>
This command examines all stress log relocation records and displays the chain
of GC relocations that may have led to the address passed in as an argument.
Conceptually the output is:
GenN    obj_address   root1, root2, root3,
GenN-1  prev_obj_addr root1, root2,
GenN-2  prev_prev_oa  root1, root4,

Sample output:
0:003> !HistObj 028970d4
GCCount   Object                                    Roots
———————————————————
2296 028970d4 00223fc4, 01e411b8,
2295 028970d4 00223fc4, 01e411b8,
2294 028970d4 00223fc4, 01e411b8,
2293 028970d4 00223fc4, 01e411b8,
2292 028970d4 00223fc4, 01e411b8,
2291 028970d4 00223fc4, 01e411b8,
2290 028970d4 00223fc4, 01e411b8,
2289 028970d4 00223fc4, 01e411b8,
2288 028970d4 00223fc4, 01e411b8,
2287 028970d4 00223fc4, 01e411b8,
2286 028970d4 00223fc4, 01e411b8,
2285 028970d4 00223fc4, 01e411b8,
322 028970d4 01e411b8,
0 028970d4
0:020> !help HistObjFind
——————————————————————————-
!HistObjFind <obj_address>
To examine log entries related to an object whose present address is known one
would use this command. The output of this command contains all entries that
reference the object:
0:003> !HistObjFind 028970d4
GCCount   Object                                  Message
———————————————————
2296 028970d4 Promotion for root 01e411b8 (MT = 5b6c5cd8)
2296 028970d4 Relocation NEWVALUE for root 00223fc4
2296 028970d4 Relocation NEWVALUE for root 01e411b8

2295 028970d4 Promotion for root 01e411b8 (MT = 5b6c5cd8)
2295 028970d4 Relocation NEWVALUE for root 00223fc4
2295 028970d4 Relocation NEWVALUE for root 01e411b8

0:020> !help HistClear
——————————————————————————-
!HistClear
This command releases any resources used by the Hist-family of commands.
Generally there’s no need to call this explicitly, as each HistInit will first
cleanup the previous resources.


An example run through

Lets attach to a w3wp.exe process and run
0:000> !DumpHeap -stat
total 36955 objects
Statistics:
MT Count TotalSize Class Name
7b4ecd7c 1 12
System.Windows.Forms.ButtonInternal.ButtonPopupAdapter
7b481f00 1 12
System.Windows.Forms.LinkLabel+LinkComparer
7b475ca8 1 12
System.Windows.Forms.FormCollection
7b474f8c 1 12
System.Windows.Forms.Layout.DefaultLayout
7b4749e0 1 12
System.Windows.Forms.ClientUtils+WeakRefCollection
7b473ca8 1 12
System.Windows.Forms.Layout.ArrangedElementCollection
7a755834 1 12
System.Diagnostics.PerformanceCounterCategoryType
7a753394 1 12
System.Diagnostics.TraceOptions
7a71a710 1 12 System.Net.TimeoutValidator
.......
00166030 891 169744 Free
054d24d4 3128 187680 System.Web.UI.LiteralControl
0548cbd4 519 197220 ASP.default_aspx

Detailed Example for an ASP.NET Web Application


Lets load a specific class:

!DumpHeap -mt 0548cbd4
Address MT Size
.....
01854ff0 0548cbd4 380
01860130 0548cbd4 380
0186b2b4 0548cbd4 380
018773f8 0548cbd4 380
01882538 0548cbd4 380
0188d6bc 0548cbd4 380
01898840 0548cbd4 380
018a39c4 0548cbd4 380
018aeb48 0548cbd4 380
total 519 objects
Statistics:
MT Count TotalSize Class Name
0548cbd4 519 197220 ASP.default_aspx
Total 519 objects
To see why the class is not being garbage collected we can call GCRoot:

0:000> !gcroot 018aeb48
Note: Roots found on stacks may be false positives. Run
"!help gcroot" for
more info.
Scan Thread 0 OSTHread 3a8
Scan Thread 2 OSTHread e8
Scan Thread 3 OSTHread 1a8
Scan Thread 6 OSTHread 7d4
Scan Thread 7 OSTHread 2b4
Scan Thread 8 OSTHread fdc
Scan Thread 9 OSTHread eac
DOMAIN(001E5E08):HANDLE(Pinned):12312f0:Root:0226c498(Syste
m.Object[])->
018af940(System.EventHandler)->
0186c0ac(System.Object[])->
018af920(System.EventHandler)->
018aeb48(ASP.default_aspx)

To view the event handler details:

0:000> !do 018af920
Name: System.EventHandler
MethodTable: 7910d61c
EEClass: 790c3a7c
Size: 32(0x20) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e
089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr
Value Name
790f9c18 40000f9 4 System.Object 0 instance
018aeb48 _target
79109208 40000fa 8 ...ection.MethodBase 0 instance
00000000 _methodBase
790fe160 40000fb c System.IntPtr 0 instance
88962888 _methodPtr
790fe160 40000fc 10 System.IntPtr 0 instance
0 _methodPtrAux
790f9c18 4000106 14 System.Object 0 instance
00000000 _invocationList
790fe160 4000107 18 System.IntPtr 0 instance
0 _invocationCount
 
invocationList and _invocationCount are both 0 indicating that this is a single delegate  and no chain to follow.

The object _methodPtr has a delegate which is holding onto - my default_aspx in this

 
!ip2md 0n88962888
Failed to request MethodData, not in JIT code range
Not very helpful
0:000> !u 0n88962888
Unmanaged code
054d7748 e862289b74 call
mscorwks!LogHelp_TerminateOnAssert+0x3f5f (79e89faf)
054d774d 5e pop esi
054d774e cc int 3
054d774f cc int 3
054d7750 38c8 cmp al,cl
054d7752 48 dec eax
054d7753 05a0774d05 add eax,54D77A0h
054d7758 0100 add dword ptr [eax],eax
054d775a 0011 add byte ptr [ecx],dl
054d775c 0000 add byte ptr [eax],al
Not very helpful
0:000> dd 0n88962888
054d7748 9b2862e8 cccc5e74
0548c838 054d77a0
054d7758 11000001 90000000 054d77a0 11000002
054d7768 90000004 00000000 054d77a0 00000000
The third and fourth DWORD look interesting because they appear to fall in the managed heap as well -- so let’s dump them out to try to figure out what they are – through trial and error, we’ll find out that the third DWORD is actually a method descriptor:


0:000> !dumpmd 0548c838
Method Name: _Default.OnDatabaseHasChanged(System.Object,
System.EventArgs)
Class: 054ab574
MethodTable: 0548c86c
mdToken: 06000004
Module: 0548c35c
IsJitted: no
m_CodeOrIL: ffffffff


The real method is bound to the delegate instance. The other DWORD appears to be a metadata reference to the event owner itself.

0:000> !dumpmt 054d77a0

EEClass: 0551940c
Module: 048ac9ec
Name: DatabaseMonitor
mdToken: 02000002
(C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary
ASP.NET
Files\leakypage\2a399ab5\b1e04c63\App_Code.onwg1zqj.dll)
BaseSize: 0xc
ComponentSize: 0x0
Number of IFaces in IFaceMap: 0
Slots in VTable: 8



 
From here I can see the module that this code is defined in (the dynamically
generated App_Code directory) and the name (DatabaseMonitor), gives me enough information to stop here and begin looking at the code itself.


 

Video Link