Skip to content

DEP, NXCOMPAT and changes in .NET Framework 2.0 SP1

This topic has been written about but not as much as it should have been. Hence, I am repeating it here again. It will also serve as a future reference for me.

DEP stands for Data Execution Prevention. It is a hardware based feature supported by Intel as well as AMD processors that disables execution of code that resides in memory areas marked as data pages. On AMD processors, this feature is implemented as NX (No eXecute) bit, and on Intel processors it is implemented as XD (eXecute Disable) bit. An OS can expose this feature and enable the application to take advantage of this protection. Once enabled, it becomes very difficult to inject and run malicious code using popular exploits such as buffer overflow which injects code on stack or heap memory and branches the instruction pointer (EIP) to execute the code. Microsoft has made this feature available in its operating systems starting with the Windows XP Service Pack 2. This feature can be enabled either system wide, or per application in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003. In Windows Vista this feature is enabled by default for all the system applications. Vista also has an Opt-out feature that allows you to enable DEP for all the (system as well as user) applications except those on the opt-out list. Robert Hensing explains this is great detail in his post. For VC++ developers there is one more way to take advantage of DEP. Visual Studio 2003 and above provide a linker switch called /NXCOMPAT. This switch sets a bit in the executable that indicates the OS loader that the application is DEP enabled. However, there is one catch. Setting this bit in the executable overrides all the other DEP settings - meaning even if DEP is disabled system wide or the application is opted out of DEP, the OS will still enable DEP for that application!!! This is where the problem starts for applications compiled using .NET 2.0 SP1. The C# compiler was updated to set this bit in the assemblies that it creates, and there is no way to unset it as in VC++ linker. If your application does not use native code, then you have nothing to worry about. If your application uses native then you may be in for a rude shock; your application might suddenly crash with IP_ON_HEAP errors. The old ATL framework generated code on the fly that ran on the stack or heap. If your .NET application uses a component or an ActiveX control that was built using the old ATL framework then your application will fail. The option is to recompile your native code using VS2003 or above. If this is a third party component that the vendor has not updated then you may be out of luck. Don't panic however! There is a ray of hope. You can disable the NXCOMPAT bit set by the C# compiler in the executable using the EditBin utility that is a part of Visual Studio installation. If your assembly is strongly named then you will have to resign it using SN utility. I strongly believe that Microsoft made a horrible mistake by deciding to set this bit by default on all the assemblies. It would have been much better if they had provided a switch just like the C++ linker. This new C# compiler feature is not well publicized by Microsoft. All the .NET developers that I have spoken to so far do not even know what DEP or NXCOMPAT is. Following the pure managed code mantra, most .NET developers do not care about native code any more even if they are using native code in their application. There is still lot of native code out there in the enterprises that is incompatible with DEP. There are lot of .NET applications that use such native code. This creates quite a problem when unknowing developers are faced with a situation when their application(s) suddenly stops working by just recompiling it in .NET 2.0 SP1. There are also other breaking changes as well as new properties, methods and Types added in SP1 as mentioned in a post by Scott Hanselman. This situation is made even worse by the release of Visual Studio 2008 as it installs .NET FX 2.0 SP1. Developers are keen to take advantage of the new features in C# 3.0 in .NET 2.0 applications using VS2008's multi-targeting feature. In an enterprise application development scenario, it is quite possible that developers will have .NET 2.0 SP1 on their development and test machines due to VS2008, but the user desktops will not have SP1. If the application uses new methods or types then it will fail on user desktops. Although one might be inclined to blame developers for developing and testing on environment that differs from the user's environment, such a scenario is highly likely to occur since the changes in .NET 2.0 SP1 are not well known. I have been able to find a list of bug fixes in SP1, but there is no knowledge base article that lists the differences between .NET 2.0 and SP1 other than the few scattered blog entries. It is a very bad idea to make such significant changes in a service pack level release. This is however a reality. . . so developers beware!

{ 6 } Comments

  1. Alessio | January 29, 2008 at 3:22 am | Permalink

    I was going mad about that! I have an application that uses a 3rd party DLL. All OK until I decided to switch to VS2008. Same code: when assembly loads the Interop for that DLL it gives “AccessViolationException”. Strange enough, this happens on some machines, and on others not! One week of desperate debugging, to find out that the new version works only in machines with DEP disabled or with CPU non-DEP. Thank you!

  2. Krishna Joshi | June 3, 2008 at 12:24 am | Permalink

    This is really a good information on DEP. I am working on a project where i am using Window Media Encoder SDK an other DLLImport functions. It is working fine on XP but when i compile it on vista with vs2008. It just crashes and give message vshost.exe stopped working.Even build is successfull but the exe doesn’t execute.When i put my exe under DEP Enable list of vista. It says program must be DEP Enabled. I read the NXCompact bit and Editbin utility. I am just looking the way to use Editbin.

    Thanks,
    Krishna Joshi

  3. aniscartujo | June 3, 2008 at 8:01 am | Permalink

    Very interesting… Without NXCOMPAT switch I was not able to run a .NET console application (that has a delay) in Windows 2008…

  4. James | July 11, 2008 at 11:31 am | Permalink

    Oooggggaaa!!!

    Thank you for this blog. I am doing a twain_32.dll import in a scanning app, and am doing the development on Vista X64 with VS2008. The program will run fine from the debugger, but as soon as I try to manually run the output exe, the program crashes, and gives the:

    xxx has stopped working – a problem caused the application to stop working correctly. Windows will close the program and notify you if a solution is available…(but dont hold breath!)

    Note: this also happens when compiled from VS2005 on x64 vista.

    I target x86 because of the interop issues with the twain driver.

    This is a BIG issue and one I’ve probably spent over 40hrs on trying to resolve. I’m buying another machine and putting x86 XP on it to work around unless I can get editbin.exe (in SDK) to do what I need.

  5. James | July 11, 2008 at 11:59 am | Permalink

    I’ll also add (for googlers), if I turn on unmanged debugging…

    First-chance exception at 0x005816b8 in app.exe: 0xC0000005: Access violation reading location 0x005816b8.

    Program runs in debugger without this option. VS2005 for this test case on x64 vista.

  6. jalilv | July 12, 2008 at 8:42 am | Permalink

    James,
    I am not sure if Editbin.exe is available with the SDK. It comes with Visual Studio installation. On my machine it was in c:\Program Files\Microsoft Visual Studio 9.0\VC\bin\editbin.exe

    HTH,

    Jalil Vaidya

{ 1 } Trackback

  1. [...] recently wrote about the impact of DEP on .NET applications due to changes in the .NET Framework 2.0 SP1. A couple [...]