This topic has been written about but not as much as it should have been. Hence, I am repeating it here again. It will also serve as a future reference for me.
DEP stands for Data Execution Prevention. It is a hardware based feature supported by Intel as well as AMD processors that disables execution of code that resides in memory areas marked as data pages. On AMD processors, this feature is implemented as NX (No eXecute) bit, and on Intel processors it is implemented as XD (eXecute Disable) bit. An OS can expose this feature and enable the application to take advantage of this protection. Once enabled, it becomes very difficult to inject and run malicious code using popular exploits such as buffer overflow which injects code on stack or heap memory and branches the instruction pointer (EIP) to execute the code. Microsoft has made this feature available in its operating systems starting with the Windows XP Service Pack 2. This feature can be enabled either system wide, or per application in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003. In Windows Vista this feature is enabled by default for all the system applications. Vista also has an Opt-out feature that allows you to enable DEP for all the (system as well as user) applications except those on the opt-out list. Robert Hensing explains this is great detail in his post. For VC++ developers there is one more way to take advantage of DEP. Visual Studio 2003 and above provide a linker switch called /NXCOMPAT. This switch sets a bit in the executable that indicates the OS loader that the application is DEP enabled. However, there is one catch. Setting this bit in the executable overrides all the other DEP settings – meaning even if DEP is disabled system wide or the application is opted out of DEP, the OS will still enable DEP for that application!!! This is where the problem starts for applications compiled using .NET 2.0 SP1. The C# compiler was updated to set this bit in the assemblies that it creates, and there is no way to unset it as in VC++ linker. If your application does not use native code, then you have nothing to worry about. If your application uses native then you may be in for a rude shock; your application might suddenly crash with IP_ON_HEAP errors. The old ATL framework generated code on the fly that ran on the stack or heap. If your .NET application uses a component or an ActiveX control that was built using the old ATL framework then your application will fail. The option is to recompile your native code using VS2003 or above. If this is a third party component that the vendor has not updated then you may be out of luck. Don’t panic however! There is a ray of hope. You can disable the NXCOMPAT bit set by the C# compiler in the executable using the EditBin utility that is a part of Visual Studio installation. If your assembly is strongly named then you will have to resign it using SN utility. I strongly believe that Microsoft made a horrible mistake by deciding to set this bit by default on all the assemblies. It would have been much better if they had provided a switch just like the C++ linker. This new C# compiler feature is not well publicized by Microsoft. All the .NET developers that I have spoken to so far do not even know what DEP or NXCOMPAT is. Following the pure managed code mantra, most .NET developers do not care about native code any more even if they are using native code in their application. There is still lot of native code out there in the enterprises that is incompatible with DEP. There are lot of .NET applications that use such native code. This creates quite a problem when unknowing developers are faced with a situation when their application(s) suddenly stops working by just recompiling it in .NET 2.0 SP1. There are also other breaking changes as well as new properties, methods and Types added in SP1 as mentioned in a post by Scott Hanselman. This situation is made even worse by the release of Visual Studio 2008 as it installs .NET FX 2.0 SP1. Developers are keen to take advantage of the new features in C# 3.0 in .NET 2.0 applications using VS2008’s multi-targeting feature. In an enterprise application development scenario, it is quite possible that developers will have .NET 2.0 SP1 on their development and test machines due to VS2008, but the user desktops will not have SP1. If the application uses new methods or types then it will fail on user desktops. Although one might be inclined to blame developers for developing and testing on environment that differs from the user’s environment, such a scenario is highly likely to occur since the changes in .NET 2.0 SP1 are not well known. I have been able to find a list of bug fixes in SP1, but there is no knowledge base article that lists the differences between .NET 2.0 and SP1 other than the few scattered blog entries. It is a very bad idea to make such significant changes in a service pack level release. This is however a reality. . . so developers beware!
{ 6 } Comments