How Many Types are loaded for Hello World

Consider the following super simple C# code

namespace SmartDeviceProject1
{
    class Program
    {
        static void Main(string[] args)
        {
            System.Console.WriteLine("Hello");
        }
    }
}

Can you guess how many managed Type gets loaded to run this? I was doing some profiling the .NET Compact Framework loader (for entirely unrelated reason) and was surprised by the list that got dumped. 87 types**, never could've guessed that...

1. System.Object
   
2. System.ValueType
   
3. System.Type
   
4. System.Reflection.MemberInfo
   
5. System.Delegate
   
6. System.MarshalByRefObject
   
7. System.SystemException
   
8. System.Exception
   
9. System.Attribute
   
10. System.Collections.Hashtable
    
11. System.Enum
    
12. System.Reflection.BindingFlags
    
13. System.MulticastDelegate
    
14. System.Reflection.MemberFilter
    
15. System.Reflection.Binder
    
16. System.Reflection.TypeAttributes
    
17. System.DateTime
    
18. System.Collections.IDictionary
    
19. System.UnhandledExceptionEventHandler
    
20. System.AppDomainManager
    
21. System.Version
    
22. System.Decimal
    
23. System.Runtime.InteropServices.ComInterfaceType
    
24. System.Collections.ICollection
    
25. System.Collections.IEqualityComparer
    
26. System.AppDomainManagerInitializationOptions
    
27. System.ArithmeticException
    
28. System.ArgumentException
    
29. System.MissingMemberException
    
30. System.MemberAccessException
    
31. System.AppDomainSetup
    
32. System.Reflection.AssemblyName
    
33. System.Globalization.CultureInfo
    
34. System.Reflection.Assembly
    
35. System.Configuration.Assemblies.AssemblyHashAlgorithm
    
36. System.Configuration.Assemblies.AssemblyVersionCompatibility
    
37. System.Reflection.AssemblyNameFlags
    
38. System.Globalization.CultureTableRecord
    
39. System.Globalization.CompareInfo
    
40. System.Globalization.TextInfo
    
41. System.Globalization.NumberFormatInfo
    
42. System.Globalization.DateTimeFormatInfo
    
43. System.Globalization.Calendar
    
44. System.Globalization.BaseInfoTable
    
45. System.Globalization.CultureTable
    
46. System.Globalization.CultureTableData
    
47. System.Globalization.NumberStyles
    
48. System.Globalization.DateTimeStyles
    
49. System.Globalization.DateTimeFormatFlags
    
50. System.Globalization.TokenHashValue
    
51. System.Globalization.CultureTableHeader
    
52. System.Globalization.CultureNameOffsetItem
    
53. System.Globalization.RegionNameOffsetItem
    
54. System.Globalization.IDOffsetItem
    
55. System.TokenType
    
56. System.Char
    
57. System.IO.TextReader
    
58. System.IO.TextWriter
    
59. System.IFormatProvider
    
60. System.Console
    
61. System.RuntimeTypeHandle
    
62. System.NotSupportedException
    
63. System.Globalization.EndianessHeader
    
64. System.Reflection.Missing
    
65. System.RuntimeType
    
66. System.Threading.StackCrawlMark
    
67. System.Globalization.CultureTableItem
    
68. System.Int32
    
69. System.Security.CodeAccessSecurityEngine
    
70. System.AppDomain
    
71. System.LocalDataStoreMgr
    
72. System.Threading.ExecutionContext
    
73. System.LocalDataStore
    
74. System.Collections.ArrayList
    
75. System.Threading.SynchronizationContext
    
76. System.Runtime.Remoting.Messaging.LogicalCallContext
    
77. System.Runtime.Remoting.Messaging.IllogicalCallContext
    
78. System.Threading.Thread
    
79. System.Collections.Generic.Dictionary`2
    
80. System.Runtime.Remoting.Messaging.CallContextRemotingData
    
81. System.RuApplication starting
    
82. System.Collections.Generic.IEqualityComparer`1
    
83. Runtime.Remoting.Messaging.CallContextSecurityData
    
84. System.Array
    
85. System.RuntimeFieldHandle
    
86. System.Globalization.CultureTableRecord[]
    
87. System.Text.StringBuilder

**This is for the compact framework CLR. Your mileage will vary if you run the same on the desktop CLR.

Designer for my path finding boards

I'm writing a small application (or rather a screen saver) that animates and demonstrates A* search algorithm. The idea is simple. On screen you see a start and end point and some random obstacles in between them. Then you see animation on how A* algorithm is used to navigate around the board to find a path (close to the shortest possible) between the two.

All of this is fairly standard. However, I got hit by a simple issue. I wanted to have the ability to design this board visually and also let my potential million users do the same. Obviously I don't have time to code up a designer and hence choose the all time manager's favorite technique. Re-use :)

So the final solution I took was to use Microsoft Office Excel as the WYSIWYG editor. I created a xlsx file with the following conditional formatting which colors cells based on the value of the cells.

So in this case

1. w = Wall marking the end of the table
2. b = blocks/bricks/obstacle
3. s = start
4. e = end

Using this I can easily design the board. The designer in action looks as follows

Since the excel sheet has conditional formatting the user just types in s, e, b, w in the cells and they all light up visually. At the end he/she just saves the file using File => Save As and uses CSV format. This saves the file shown above as

w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w
w,,,,,,,,,,,,,,,,,,,,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,b,,,,,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,s,,,,,,,,b,b,b,b,b,b,,,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,,,,,,,b,b,b,b,,b,b,b,b,,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,,,,,,b,b,b,,,,,,b,b,b,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,,,,,,b,b,,,,,,,,b,b,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,,,,,,b,b,,,,,,,,b,b,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,,,,,,b,b,,,,,,,b,b,,,,,b,,,,,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,b,b,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,b,b,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,b,b,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,b,b,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,,,,,,,,b,,,,b,,,,,,,,,,,e,,,w
w,,,,,,,,,,,,,,,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,,,,,,,,b,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,,,,,,,,,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,,,,,,,,,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,,,,,,,,,,,,b,,,,,,,,,,,,,,w
w,,,,,,,,,,,,,,,,,,,,,,,,,,,b,,,,,,,,,,,,,,w
w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w,w

As you see the format is simply a row per line and every column separated by a comma. My simulator reads this file and renders using whatever renderer is configured (console or WPF).

More about the A* simulator to come soon...

It's is easy to claim that the world won't get destroyed today

The clock is ticking and the Large Hadron Collider in Cern is going to get switched on today (9/10/2008). Even though there are speculations, CERN is claiming it's perfectly safe and the world won't end. But it's easy to claim that, who'll be around to prove them wrong in case they are :)

It's one of the coolest device at an operating temperature of < -270°C. But I'd get really angry if there's any disturbance to my Birthday celebrations!!

Team Foundation Server tool dump workspace details

I juggle around with a lot of workspaces. The reason is .NET Compact Framework is consumed in a whole bunch of things like Windows Mobile, Xbox, Zune, Nokia and most of them are on different source branches. On top of this active feature work happens in feature branches and there are other service branches to frequently peek into.

So the net effect is I need to sometime go to different folders and see what it's source control workspace mapping is or I need to dump details of workspaces on other computers and see if I can use that workspace as a template to create a new workspace.

The thing I hate here is to run cd to that folder and do a tf workspace to bring up the GUI and then dismiss the UI. I don't like GUI when things can be done equally well on the console. So I quickly hacked up a tool that dumps workspace details onto the console, something as below

d:\>workspace.exe NETCF\F01
Name:      ABHINAB2_S60
Server:    TfsServer01
Owner:     Abhinaba
Computer:  ABHINAB2

Comment:

Working folders:

Active   $/NetCF/F01/tools     d:\NETCF\F01\tools
Active   $/NetCF/F01/source    d:\NETCF\F01\source

-

This tool can be used either to see the mapping of a folder or given a workspace name and owner dump it details. It uses the exact same format as show in the tf workspace UI.

Source

The source and a build binary is linked below. It is a single source file (Program.cs) and you should be able to pull it into any version of Visual Studio and even build it from the command line using CSC

1. Source
   
2. Binary

Using the tool

Build it or use the pre-built binary and edit the .config file to point it to your TFS server and you are all set to go.

Obsession with desktop continues

This is my new office setup. I have been pushing around a lot of code lately and felt I needed more real-estate to effectively do what I'm doing. So I hooked up another monitor. All  3 are standard HP LP1965 19" monitors.

However, since none of my video cards support 3 monitors and I have a whole bunch of computers to hook up I had to do some complex wiring around :). If I number the screens 1,2,3 from left to right this is how it works out

• 1 and 2 is connected to my main dev box (machine1)
• 2 and 3 is connected to the machine I use for emails and browsing (machine 2)
• 2 actually goes through a 4-port KVM switch using which it can circle through machine1, machine2 and two other less used machines
• 1 also goes through a 2 port KVM switch and connects a xbox and machine 1
• I don't use the laptop at work directly, I connect to it using remote desktop.

Sweet :)

It's not as complex as it sounds. For my normal flow I just use the KVM to switch between machine-1 and machine-2. I rarely need to switch between the xbox and machine1.

PS: Don't try to make much sense of the Vintage books on the shelve, I plan to keep them for some more time before handing them off to some historian.

Back to Basic: Using a System.Threading.Interlocked is a great idea

I just saw some code which actually takes a lock to do a simple set operation. This is bad because taking locks are expensive and there is an easy alternative. The System.Threading.Interlocked class and its members can get the same job done and much faster.

I wrote the following two methods which increments a variable a million times. The first method does that by taking a lock and the other uses the Interlocked.Increment API.

static int IncrementorLock()
{
    int val = 0;
    object lockObject = new object();
    for (int i = 0; i < 1000000; ++i)
    {
        lock (lockObject)
        {
            val++;
        }
    }

    return val;
}

static int IncrementorInterlocked()
{
    int val = 0;
    for (int i = 0; i < 1000000; ++i)
    {
        System.Threading.Interlocked.Increment(ref val);
    }

    return val;
}

-

I then used the Visual Studio Team System Profiler (instrumented profiling) and got the following performance data.

Function Name	Elapsed Inclusive Time	Application Inclusive Time
IncrementorInterlocked()	1,363.45	134.43
IncrementorLock()	4,374.23	388.69

Even though this is a micro benchmark and uses a completely skewed scenario, it still drives the simple point that using the interlocked class is way faster.

The reason the interlocked version is faster is because instead of using .NET locks it directly calls Win32 apis like InterlockedIncrement which ensures atomic updation to variables at the OS scheduler level.

These Interlocked APIs complete depend on the support of the underlying OS. This is the reason you'd see that all the APIs are not available uniformly across all versions of .NET (e.g. there is no uniform coverage over WinCE and Xbox). While implementing these for the Nokia platform (S60) we are hitting some scenarios where there is no corresponding OS API.

String equality

akutz has one of the most detailed post on string interning and equality comparison performance metrics I have ever seen. Head over to the post here

I loved his conclusion which is the crux of the whole story.

"In conclusion, the String class’s static Equals method is the most efficient way to compare two string literals and the String class’s instance Equals method is the most efficient way to compare two runtime strings. The kicker is that there must be 10^5 (100,000) string comparisons taking place before one method starts becoming more efficient than the next. Until then, use whichever method floats your boat."

Kismat Konnection

<sorry for the total off-topic />

I had a real hard time giving a title to this post.

The post was originally supposed to be about the fire I saw yesterday in a movie theatre Talkie Town (map). However it is not really about the fire, it is more about the people and how they reacted to it.

First things first, the fire.

Folks who know me, know that I go to a movie theatre once in couple of years, I prefer movies visiting me rather than the other way around. In the 4 years I'm in Hyderabad, this was my 3rd visit to a movie. So I can say it was a momentous occasion when me and my wife impromptu decided to see a movie in the 2 screen theatre in Miyapur called Talkie Town. The decision was heavily biased on the fact that my father-in-law was at home looking after our daughter.

Batman  (or rather The Dark Knight) lost to my wife and I was forced to see this bollywood flick Kismat Konnection.

This is where all the fun started. Just after the interval (1.5 hours through, and yes bollywood movies are that long) people saw some light on the theatre ceiling. Soon the light spread and there was a small hole in the ceiling and we could actually see flames. Whatever was above the sound proofed ceiling had caught fire and the heat actually caused the material of the ceiling to burn.

Then the most amazing thing happened, 50% people didn't care. They were gleefully looking up and not even caring to leave the theatre. I'm not talking about some false fire alarm, or smell of smoke, it was a real fire with flames and a burned hole in the ceiling!! The fire alarm didn't sound and no one from the theatre authority seemed to be around. I left and called some folks and they simply went to the terrace. All the noise now prompted some more people to leave but even at that time there were others happily watching as if the whole thing was a part of the movie.

Then even more amazing thing happened, the folks from the theatre came and said that the fire has been doused and re-started the show. The hall was smoky and the AC was off. Me and another guy (yea there were some more sane people around) caught hold of an official and he simply said he has checked and he can guarantee that everything is safe. I asked him that since he couldn't ensure that there is no fire in the first place how can he guarantee against a recurrence, he simply said I can get a refund of my ticket. Which I did and left the hall.

On the way back I had a revelation. From childhood I had seen that people around me had little care about safety in general, but this is the first time I figured out that people didn't even care about their own safety. How can a movie be worth taking the risk of sitting in a fire hazard zone and that too with small children on their lap is something I will never figure out.

A lot of things including the traffic situation and the weird jay walking I see around suddenly makes more sense to me.

Writing exception handlers as separate methods may prove to be a good idea

Let us consider a scenario where you catch some exception and in the exception handler do some costly operation. You can write that code in either of the following ways

Method-1 : Separate method call

public class Program
{
    public static void Main(string[] args)
    {
        try
        {
            using (DataStore ds = new DataStore())
            {
                // ...
            }
        }
        catch (Exception ex)
        {
            ErrorReporter(ex);
        }
    }

    private static void ErrorReporter(Exception ex)
    {
        string path = System.IO.Path.GetTempFileName();
        ErrorDumper ed = new ErrorDumper(path, ex);
        ed.WriteError();

        XmlDocument xmlDoc = new XmlDocument();
        xmlDoc.Load(path);
        RemoteErrorReporter er = new RemoteErrorReporter(xmlDoc);
        er.ReportError();
    }
}

-

Method-2 : Inline

public static void Main(string[] args)
{
    try
    {
        using (DataStore ds = new DataStore())
        {
            // ...
        }
    }
    catch (Exception ex)
    {
        string path = System.IO.Path.GetTempFileName();
        ErrorDumper ed = new ErrorDumper(path, ex);
        ed.WriteError();

        XmlDocument xmlDoc = new XmlDocument();
        xmlDoc.Load(path);
        RemoteErrorReporter er = new RemoteErrorReporter(xmlDoc);
        er.ReportError();
    }
}

-

The simple difference is that in the first case the exception handler is written as a separate method and in the second case it is placed directly inline inside the handler itself.

The question is which is better in terms of performance?

In case you do have significant code and type reference in the handler and you expect the exception to be thrown rarely in an application execution then the Method-1 is going to be more performant.

The reason is that just before executing a method the whole method gets jitted. The jitted code contains stubs to the other method's it will call but it doesn't do a recursive jitting. This means when Main gets called it gets jitted but the method ErrorReporter is still not jitted. So in case the exception is never fired all the code inside ErrorReporter never gets Jitted. This might prove to be significant saving in terms of time and space if the handling code is complex and refers to type not already referenced.

However, if the code is inline then the moment Main gets jitted all the code inside the catch block gets jitted. This is expensive not only because it leads to Jitting of code that is never executed but also because all types referenced in the catch block is also resolved resulting in loading a bunch of dlls after searching though the disk. In our example above System.Xml.dll and the other dll containing remote error reporting gets loaded even though they will never be used. Since disk access, loading assemblies and type resolution are slow, the simple change can prove to give some saving.

Microsoft RoundTable

Our conference rooms have been fitted with this really weird looking device (click to enlarge).

I had no clue what the thing was. Fortunately it's box was still placed in the room along with the manual. It's called the Microsoft RoundTable and it is actually a 360-degree camera (with 5 cameras and 6 microphones). It comes with bundled software that let's all participant be visible to the other side in a live meeting at real time. It shows the white board and the software is intelligent enough to focus on and track the active speaker (using microphone and face recognition) and much much more (lot of MS Research stuff has gone into it). The video below gives you some idea and head on to this post for some review and inside view of the device.

Simply put it's AWSOME

Do namespace using directives affect Assembly Loading?

The simple answer is no, the inquisitive reader can read on :)

Close to 2 year back I had posted about the two styles of coding using directives as follows

Style 1

namespace MyNameSpace
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    // ...
}

-

Style 2

using System;
using System.Collections.Generic;
using System.Text;

namespace MyNameSpace
{
    // ...
}

-

and outlined the benefits of the first style (using directives inside the namespace). This post is not to re-iterate them.

This post to figure out if either of the styles have any bearing in the loading order of assemblies. Obviously at the first look it clearly indicates that is shouldn't, but this has caused some back and forth discussions over the web.

Scot Hanselman posted about a statement on the Microsoft Style cop blog which states

"When using directives are declared outside of a namespace, the .Net Framework will load all assemblies referenced by these using statements at the same time that the referencing assembly is loaded.

However, placing the using statements within a namespace element allows the framework to lazy load the referenced assemblies at runtime. In some cases, if the referencing code is not actually executed, the framework can avoid having to load one or more of the referenced assemblies completely. This follows general best practice rule about lazy loading for performance.

Note, this is subject to change as the .Net Framework evolves, and there are subtle differences between the various versions of the framework."

This just doesn't sound right because using directives have no bearing to assembly loading.

Hanselman did a simple experiment with the following code

using System;  
using System.Xml;  
  
namespace Microsoft.Sample  
{  
   public class Program  
   {  
      public static void Main(string[] args)  
      {  
         Guid g = Guid.NewGuid();  
         Console.WriteLine("Before XML usage");  
         Console.ReadLine();  
         Foo();  
         Console.WriteLine("After XML usage");  
         Console.ReadLine();  
      }  
  
      public static void Foo()  
      {  
         XmlDocument x = new XmlDocument();  
      }  
   }  
}  

-

and then he watched the loading time using process explorer and then he moved the using inside the namespace and did the same. Both loaded the System.Xml.dll after he hit enter on the console clearly indicating that for both the cases they got lazy loaded.

Let me try to give a step by step rundown of how the whole type look up of XmlDocument happens in .NETCF which in turn would throw light on whether using directives have bearing on assembly loading.

1. When Main method is Jitted and ran the System.Xml.dll is not yet loaded
   
2. When method Foo is called the execution engine (referred to as EE) tries to JIT the method. As documented the Jitter only JITs methods that are to be executed.
   
3. The Jitter tries to see if the method Foo is managed (could be native as well due to mixed mode support) and then tries to see if it's already Jitted (by a previous call), since it's not it goes ahead with jitting it
   
4. The jitter validates a bunch of stuff like whether the class on which the method Foo is being called (in this case Microsoft.Sample.Program) is valid, been initialized, stack requirements, etc...
   
5. Then it tries to resolve the local variables of the method. It waits to resolve the local variable type reference till this point so that it is able to save time and memory by not Jitting/loading types that are referenced by methods that are never executed
   
6. Then it tries to resolve the type of the variable which in this case if System.Xml.XmlDocument.
   
7. It sees if it's already in the cache, that is if that type is already loaded
   
8. Since it's not the case it tries to search for the reference based on the type reference information
   
9. This information contains the full type reference including the assembly name, which in this case is System.Xml.dll and also version information,strong name information, etc...
   
10. All of the above information along with other information like the executing application's path is passed to the assembly loader to load the assembly
    
11. The usual assembly search sequence is used to look for the assembly and then it is loaded and the type reference subsequently gets resolved

If you see the above steps there is in no way a dependency of assembly loading on using directive. Hence at least on .NETCF whether you put the using outside or inside the namespace you'd get the referenced assemblies loaded exactly at the time of first reference of a type from that assembly (the step #5 above is the key).

Auto generating Code Review Email for TFS

We use a small command line tool called crmail to auto-generate code review email from shelveset. I find the whole process very helpful and thought I'd share the process and the tool (which has some really cool features).

Features

1. Automatic generation of the email from the shelveset details
2. Hyperlinks are put to TFS webaccess so that you can review code from machines without any tools installed, even without source enlistment. Yes it's true!!! The only thing you need is your office's intranet access
3. You can even use a Windows mobile phone :) and even some non MS browsers. Ok I guess I have sold this enuf
4. This is how the email looks like with all the details pointed out
   
5. Effectively you can see the file diff, history, blame (annotate), shelveset details, associated bugs, everything from your browser and best thing is that all of these takes one click each.
   This is how the fill diff looks in the browser
   

Pre-reqs

1. Team System Web Access (TSWA) 2008 power tool installed on your TFS server
2. Outlook installed on the machine on which the email is generated
3. Enlistment and TFS client installed on the machine on which the email is generated
4. For reviewers there is no pre-req other than a browser and email reader.

Dev process

1. The developer creates a shelveset after he is done with his changes. He ensures he fills up all the details including the reviewers email address ; separated
2. He runs the tool with a simple command
   crmail shelvesetname
3. Email gets generated and opened he fills in additional information and fires send
4. Done!!

Reviewers

Ok they just click on the email links. Since mostly these are managers what more do you expect out of them? Real devs will stick with firing up tfpt command line :)

Configuring the tool

1. Download the binaries from here
2. Unzip. Open the crmail.exe.config file and modify the values in it to point to your tfsserver and your code review distribution list (if you do not have one then make it empty)
3. Checkin to some tools folder in your source control so that everyone in your team has access to it

Support

Self help is the best help :), download the sources from here and enjoy. Buck Hodges post on the direct link URLs would help you in case you want to modify the sources to do more.

How does the .NET CF handle null reference

What happens when we have code as bellow

class B
{
    public virtual void Virt(){
        Console.WriteLine("Base::Virt");
    }
}

class Program
{
    static void Main(string[] args){
        B b = null;
        b.Virt(); // throws System.NullReferenceException
    }
}

Obviously we have a null reference exception being thrown. If you see the IL the call looks like

    L_0000: nop 
    L_0001: ldnull 
    L_0002: stloc.0 
    L_0003: ldloc.0 
    L_0004: callvirt instance void ConsoleApplication1.B::Virt()
    L_0009: nop 
    L_000a: ret 

 

So in effect you'd expect the jitter to generate the following kind of code (in processor instruction)

if (b == null)
   throw new NullReferenceException
else
   b->Virt() // actually call safely using the this pointer

 

However, generating null checks for every call is going to lead to code bloat. So to work around this on some platforms (e.g. .NETCF on WinCE 6.0 and above) it uses the following approach

1. Hook up native access violation exception (WinCE 6.0 supports this) to a method in the execution engine (EE)
   
2. Do not generate any null checking and directly generate calls through references
   
3. In case the reference is null then a native AV (access violation is raised as invalid 0 address is accessed) and the hook method is called
   
4. At this point the EE checks to see if the source of the access violation (native code) is inside Jitted code block. If yes it creates the managed NullRefenceException and propagates it up the call chain.
   
5. If it's outside then obviously it's either CLR itself or some other native component is crashing and it has nothing to do about it..

C# generates virtual calls to non-virtual methods as well

Sometime back I had posted about a case where non-virtual calls are used for virtual methods and promised posting about the reverse scenario. This issue of C# generating callvirt IL instruction even for non-virtual method calls keeps coming back on C# discussion DLs every couple of months. So here it goes :)

Consider the following code

class B
{
    public virtual void Virt(){
        Console.WriteLine("Base::Virt");
    }

    public void Stat(){
        Console.WriteLine("Base::Stat");
    }
}

class D : B
{
    public override void Virt(){
        Console.WriteLine("Derived::Virt");
    }
}

class Program
{
    static void Main(string[] args)
    {
        D d = new D();
        d.Stat(); // should emit the call IL instruction
        d.Virt(); // should emit the callvirt IL instruction
    }
}

The basic scenario is that a base class defines a virtual method and a non-virtual method. A call is made to base using a derived class pointer. The expectation is that the call to the virtual method (B.Virt) will be through the intermediate language (IL) callvirt instruction and that to the non-virtual method (B.Stat) through call IL instruction.

However, this is not true and callvirt is used for both. If we open the disassembly for the Main method using reflector or ILDASM this is what we see

    L_0000: nop 
    L_0001: newobj instance void ConsoleApplication1.D::.ctor()
    L_0006: stloc.0 
    L_0007: ldloc.0 
    L_0008: callvirt instance void ConsoleApplication1.B::Stat()
    L_000d: nop 
    L_000e: ldloc.0 
    L_000f: callvirt instance void ConsoleApplication1.B::Virt()
    L_0014: nop 
    L_0015: ret 

Question is why? There are two reasons that have been brought forward by the CLR team

1. API change.
   The reason is that .NET team wanted a change in an method (API) from non-virtual to virtual to be non-breaking. So in effect since the call is anyway generated as callvirt a caller need not be recompiled in case the callee changes to be a virtual method.
   
   
2. Null checking
   If a call is generated and the method body doesn't access any instance variable then it is possible to even call on null objects successfully. This is currently possible in C++, see a post I made on this here.
   

   With callvirt there's a forced access to this pointer and hence the object on which the method is being called is automatically checked for null.

callvirt does come with additional performance cost but measurement showed that there's no significant performance difference between call with null check vs callvirt. Moreover, since the Jitter has full metadata of the callee, while jitting the callvirt it can generate processor instructions to do static call if it figures out that the callee is indeed non-virtual.

However, the compiler does try to optimize situations where it knows for sure that the target object cannot be null. E.g. for the expression i.ToString(); where i is an int call is used to call the ToString method because Int32 is value type (cannot be null) and sealed.

Guy or a Girl

One interesting aspect of working in Internationally distributed team is that sometime it gets difficult to make common judgements. E.g. when we see a name we inherently figure out whether it's a male or female name and refer to that person as such in email. The issue is that I cannot always make the same judgement in case of names from another country/culture.

In my previous team in a long email thread someone continually referred to Khushboo as "he". Khushboo didn't correct him and it went on for some time until I pointed out that to him in a separate email. Today I was typing an email to someone and suddenly figured out I had no idea whether one of the person I'm referring to is male or female. I took a wild guess and I'm waiting to get corrected.

Baby smash

What is the common thing between every programmer dad/mom? The moment they get onto a new UI platform they write a child proofing application for the keyboard.

Scott Hanselman has just posted his version baby smash (via AmitChat). The funny thing is I've written one in WPF and so did my ex-manager.

Stylecop has been released

Microsoft released the internal tool StyleCop to public under the fancy yet boring name of Microsoft Source Analysis for C#. Even though the name is boring the product is not.

You'll love this tool when it imposes consistent coding style across your team. You'll hate this tool when it imposes the same on you. The result is stunning looking, consistently styled code which your whole team can follow uniformly.

StyleCop has been in use for a long time internally in Microsoft and many teams mandate it's usage. My previous team VSTT used it as well. The only crib I had is that it didn't allow single line getters and setters (and our team didn't agree to disable this rule either).

// StyleCop didn't like this one
public int Foo
{
    get { return Foo; }
}

// StyleCop wanted this instead
public int Foo
{
    get
    {
        return Foo;
    }
}

 

Read more about using StyleCop here. You can set this up to be run as a part of your build process as documented here. Since this is plugged in as a MsBuild project you can use it in as a part of Team Foundation Build process as well.

Let the style wars begin in team meetings :)

Lambda the ultimate

Whatever I said about lambda before is crap. Each time time I use it I feel happy.

I had to fire a timer every so often this is what I can use with lambda...

dataStoreTimer = new Timer(new TimerCallback(
            (obj) => { (obj as AutoResetEvent).Set(); }), pollEvent, 100, 1000);

Sweet!!!

**BTW you can't blame me for having my shortest post just after the longest!!

Cell phone assault

Last two weeks my cell phone got assaulted thrice. First it was someone sending me a virus over bluetooth (a sis file actually). This happened when I was taking a photograph of my daughter with the cell phone camera in a restaurant (Aromas of China, City Center mall in Hyderabad).

The next one was bluetooth based advertisement messages in the Forum Mall in Bangalore. They were actually sending offers of the hour over bluetooth and I got 2 such messages.

The third incident was in the airport when someone was again trying to send me and make me open an trojan app.

I was really surprised with the rapid growth of cell phone based attacks. Worst is few people know of this. My wife had no idea that you can actually send applications over bluetooth and that can infect the phone.

Building Scriptable Applications by hosting JScript

If you have played around with large applications, I'm sure you have been intrigued how they have been build to be extendable. The are multiple options

1. Develop your own extension mechanism where you pick up extension binaries and execute them.
   One managed code example is here, where the application loads dlls (assemblies) from a folder and runs specific types from them. A similar unmanaged approach is allow registration of guids and use COM to load types that implement those interfaces
2. Roll out your own scripting mechanism:
   One managed example is here where on the fly compilation is used. With DLR hosting mechanism coming up this will be very easy going forward
3. Support standard scripting mechanism:
   This involves hosting JScript/VBScript inside the application and exposing a document object model (DOM) to it. So anyone can just write standard JScript to extend the application very much like how JScript in a webpage can extend/program the HTML DOM.

Obviously the 3rd is the best choice if you are developing a native (unmanaged) solution. The advantages are many because of low learning curve (any JScript programmer can write extensions), built in security, low-cost.

In this post I'll try to cover how you go about doing exactly that. I found little online documentation and took help of Kaushik from the JScript team to hack up some code to do this.

The Host Interface

To host JScript you need to implement the IActiveScriptSite. The code below shows how we do that stripping out the details we do not want to discuss here (no fear :) all the code is present in the download pointed at the end of the post). The code below is in the file ashost.h

class IActiveScriptHost : public IUnknown 
{
public:
    // IUnknown
    virtual ULONG __stdcall AddRef(void) = 0;
    virtual ULONG __stdcall Release(void) = 0;
    virtual HRESULT __stdcall QueryInterface(REFIID iid,
                                        void **obj) = 0;

    // IActiveScriptHost
    virtual HRESULT __stdcall Eval(const WCHAR *source, 
                                         VARIANT *result) = 0;
    virtual HRESULT __stdcall Inject(const WCHAR *name, 
                                         IUnknown *unkn) = 0;
};

class ScriptHost : 
    public IActiveScriptHost, 
    public IActiveScriptSite 
{
private:
    LONG _ref;
    IActiveScript *_activeScript;
    IActiveScriptParse *_activeScriptParse;

    ScriptHost(...){}

    virtual ~ScriptHost(){}
public:
    // IUnknown
    virtual ULONG __stdcall AddRef(void);
    virtual ULONG __stdcall Release(void);
    virtual HRESULT __stdcall QueryInterface(REFIID iid, void **obj);

    // IActiveScriptSite
    virtual HRESULT __stdcall GetLCID(LCID *lcid);
    virtual HRESULT __stdcall GetItemInfo(LPCOLESTR name,
        DWORD returnMask, IUnknown **item, ITypeInfo **typeInfo);
    virtual HRESULT __stdcall GetDocVersionString(BSTR *versionString);
    virtual HRESULT __stdcall OnScriptTerminate(const VARIANT *result,
        const EXCEPINFO *exceptionInfo);
    virtual HRESULT __stdcall OnStateChange(SCRIPTSTATE state);
    virtual HRESULT __stdcall OnEnterScript(void);
    virtual HRESULT __stdcall OnLeaveScript(void);
    virtual HRESULT __stdcall OnScriptError(IActiveScriptError *error);

    // IActiveScriptHost
    virtual HRESULT __stdcall Eval(const WCHAR *source,
                                           VARIANT *result);
    virtual HRESULT __stdcall Inject(const WCHAR *name, 
                                           IUnknown *unkn);
public:

    static HRESULT Create(IActiveScriptHost **host)
    {
        ...
    }

};

Here we are defining an interface IActiveScriptHost. ScriptHost implements the IActiveScriptHost and also the required hosting interface IActiveScriptSite. IActiveScriptHost exposes 2 extra methods (in green) that will be used from outside to easily host js scripts.

In addition ScriptHost also implements a factory method Create. This create method does the heavy lifting of using COM querying to get the various interfaces its needs (IActiveScript, IActiveScriptParse) and stores them inside the corresponding pointers.

Instantiating the host

So the client of this host class creates the ScriptHosting instance by using the following (see ScriptHostBase.cpp)

IActiveScriptHost *activeScriptHost = NULL;
HRESULT hr = S_OK;
HRESULT hrInit = S_OK;

hrInit = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
if(FAILED(hr)) throw L"Failed to initialize";

hr = ScriptHost::Create(&activeScriptHost);
if(FAILED(hr)) throw L"Failed to create ScriptHost";

 

With this the script host is available through activeScriptHost pointer and we already have JScript engine hosted in our application

Evaluating Scripts

Post hosting we need to make it do something interesting.This is where the IActiveScriptHost::Eval method comes in.

HRESULT __stdcall ScriptHost::Eval(const WCHAR *source, 
                                   VARIANT *result)
{
    assert(source != NULL);

    if (source == NULL)
        return E_POINTER;

    return _activeScriptParse->ParseScriptText(source, NULL, 
                                  NULL, NULL, 0, 1, 
                                  SCRIPTTEXT_ISEXPRESSION, 
                                  result, NULL);
}

Eval accepts a text of the script, makes it execute using IActiveScriptParse::ParseScriptText and returns the result.

So effectively we can accept input from the console and evaluate it (or read a file and interpret the complete script in it.

while (true) 
{
    wcout << L">> ";
    getline(wcin, input);
    if (quitStr.compare(input) == 0) break;

    if (FAILED(activeScriptHost->Eval(input.c_str(), &result)))
    {
        throw L"Script Error";
    }
    if (result.vt == 3)
        wcout << result.lVal << endl;
}

So all this is fine and at the end you can run the app (which BTW is a console app) and this what you can do.

JScript sample Host
q! to quit

>> Hello = 7
7
>> World = 6
6
>> Hello * World
42
>> q!
Press any key to continue . . .

So you have extended your app to do maths for you or rather run basic scripts which even though exciting but is not of much value.

Extending your app

Once we are past hosting the engine and running scripts inside the application we need to go ahead with actually building the application's DOM and injecting it into the hosting engine so that JScript can extend it.

If you already have a native application which is build on COM (IDispatch) then you have nothing more to do. But lets pretend that we actually have nothing and need to build the DOM.

To build the DOM you need to create IDispatch based DOM tree. There can be more than one roots. In this post I'm not trying to cover how to build IDispatch based COM objects (which you'd do using ATL or some such other means). However, for simplicity we will roll out a hand written implementation which implements an interface as below.

class IDomRoot : public IDispatch 
{
    // IUnknown
    virtual ULONG __stdcall AddRef(void) = 0;
    virtual ULONG __stdcall Release(void) = 0;
    virtual HRESULT __stdcall QueryInterface(REFIID iid, 
                                             void **obj) = 0;

    // IDispatch
    virtual HRESULT __stdcall GetTypeInfoCount( UINT *pctinfo) = 0;
    virtual HRESULT __stdcall GetTypeInfo( UINT iTInfo, LCID lcid,
                                           ITypeInfo **ppTInfo) = 0;
    virtual HRESULT __stdcall GetIDsOfNames( REFIID riid, 
                                      LPOLESTR *rgszNames,
                                      UINT cNames, LCID lcid,  
                                      DISPID *rgDispId) = 0;

    virtual HRESULT __stdcall Invoke( DISPID dispIdMember, REFIID riid, 
                                      LCID lcid, WORD wFlags, 
                                      DISPPARAMS *pDispParams, 
                                      VARIANT *pVarResult, 
                                      EXCEPINFO *pExcepInfo, 
                                      UINT *puArgErr) = 0;

    // IDomRoot
    virtual HRESULT __stdcall Print(BSTR str) = 0;
    virtual HRESULT __stdcall get_Val(LONG* pVal) = 0;
    virtual HRESULT __stdcall put_Val(LONG pVal) = 0;
};

 

At the top we have the standard IUnknown and IDispatch methods and at the end we have our DOM Root's methods (in blue). It implements a Print method that prints a string and a property called Val (with a set and get method for that property).

The class DomRoot implements this method and an additional method named Create which is the factory to create it. Once we are done with creating this we will inject this object inside the JScript scripting engine. So our final script host code looks as follows

IActiveScriptHost *activeScriptHost = NULL;
IDomRoot *domRoot = NULL;
HRESULT hr = S_OK;
HRESULT hrInit = S_OK;

hrInit = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
if(FAILED(hr)) throw L"Failed to initialize";

// Create the host
hr = ScriptHost::Create(&activeScriptHost);
if(FAILED(hr)) throw L"Failed to create ScriptHost";

// create the DOM Root
hr = DomRoot::Create(&domRoot);
if(FAILED(hr)) throw L"Failed to create DomRoot";

// Inject the created DOM Root into the scripting engine
activeScriptHost->Inject(L"DomRoot", (IUnknown*)domRoot);

What happens with the inject is as below

map rootList;
typedef map::iterator MapIter;
typedef pair InjectPair;

HRESULT __stdcall ScriptHost::Inject(const WCHAR *name, 
                                     IUnknown *unkn)
{
    assert(name != NULL);

    if (name == NULL)
        return E_POINTER;

    _activeScript->AddNamedItem(name, SCRIPTITEM_GLOBALMEMBERS | 
                                      SCRIPTITEM_ISVISIBLE );		
    rootList.insert(InjectPair(std::wstring(name), unkn));

    return S_OK;
}

&npsp;

In inject we store the name of the object and the corresponding IUnknown in a map (hash table). Each time the script will encounter a object in its code it calls GetItemInfo with that objects name and we then de-reference into the hash table and return the corresponding IUnknown

HRESULT __stdcall ScriptHost::GetItemInfo(LPCOLESTR name,
                                    DWORD returnMask,
                                    IUnknown **item,
                                    ITypeInfo **typeInfo)
{	
    MapIter iter = rootList.find(name);
    if (iter != rootList.end())
    {
        *item = (*iter).second;
        return S_OK;
    }
    else
        return E_NOTIMPL;
}

After that the script calls into that IDispatch to look for properties and methods and calls into them.

The Whole Flow

By now we have seen a whole bunch of code. Let's see how the whole thing works together. Let's assume we have a extension written in in JScript and it calls DomRoot.Val = 5; this is what happens to get the whole thing to work

1. During initialization we had created the DomRoot object (DomRoot::Create) which implements IDomRoot and injected it in the script engine via AddNamedItem and stored it at our end in a rootList map.
   
2. We call activeScriptHost->Eval(L"DomRoot.Val = 5;", ...) to evaluate the script. Evan calls _activeScriptParse->ParseScriptText.
   
3. When the script parse engine sees the "DomRoot" name it figures out that the name is a valid name added with AddNamedItem and hence it calls its hosts ScriptHost::GetItemInfo("DomRoot");
   
4. The host we have written looks up the same map filled during Inject and returns the IUnknown of it to the scripting engine. So at this point the scripting engine has a handle to our DOM root via an IUnknown to the DomRoot object
   
5. The scripting engine does a QueryInterface on that IUnknown to get the IDispatch interface from it
   
6. Then the engine calls the IDispatch::GetIDsOfNames with the name of the property "Val"
   
7. Our DomRoots implementation of GetIDsOfNames returns the required Dispatch ID of the Val property (which is 2 in our case)
   
8. The script engine calls IDispatch::Invoke with that dispatch id and a flag telling whether it wants the get or the set. In this case its set. Based on this the DomRoot re-directs the call to DomRoot::put_Val
   
9. With this we have a full flow of the host to script back to the DOM

In action

JScript sample Host
q! to quit

>> DomRoot.Val = 5;
5
>> DomRoot.Val = DomRoot.Val * 10
50
>> DomRoot.Val
50
>> DomRoot.Print("The answer is 42");
The answer is 42

 

Source Code

First of all the disclaimer. Let me get it off my chest by saying that the DomRoot code is a super simplified COM object. It commits nothing less than sacrilege. You shouldn't treat it as a sample code. I intentionally didn't do a full implementation so that you can step into it without the muck of IDispatchImpl or ATL coming into your way.

However, you can treat the script hosting part (ashost, ScriptHostBase) as sample code (that is the idea of the whole post :) )

The code organization is as follows

ashost.cpp, ashost.h - The Script host implementation
DomRoot.cpp, DomRoot.h - The DOM Root object injected into the scripting engine
ScriptHostBase.cpp - Driver

Note that in a real life example the driver should load jscript files from a given folder and execute it.

Download from here

Model, View, Controller

These days the whole world is abuzz with the Model, View, Controller (MVC) architecture. This is not something new and is known by computer scientists for close to 30 years. I guess the new found popularity is due to the fact that this has heavy application is web development and lot of main-stream web development platform are putting in support for this. Ruby-on-rails and ASP.NET MVC are classic examples.

Coding horror has a nice post on this topic. I liked the following statement it made

"Skinnability cuts to the very heart of the MVC pattern. If your app isn't "skinnable", that means you've probably gotten your model's chocolate in your view's peanut butter, quite by accident."

I actually use a very similar concept. The moment I see an application's architecture (be it an interview candidate or a friend showing off something) I ask the question "Can you write a console version of this easily?". If the answer is no or it needs a re-design it means that the separation of model, view and controller is not correct. You are going to have a nightmare if you write and maintain that software.

You need to be careful about how you use count in for-loops

Lets consider the following code

 MyCollection myCol = new MyCollection();
 myCol.AddRange(new int[] { 1, 2, 3, 4, 5, });
 for (int i = 0; i < myCol.Count; ++i)
 {
     Console.WriteLine("{0}", i);
 }

 

What most people forgets to consider is the condition check that happens for the for loop (in Red). In this case it is actually a call to a method get_Count. The compiler doesn't optimize this away (even when inlined)!! Since the condition is evaluated after each iteration, there's actually a method call happening each time. For most .NET code it is low-cost because ultimately it's just a small method with count being returned from an instance variable (if inlined the method call overhead also goes away). Something like this is typically written for count.

public int Count
{
    get
    {  
        return this.count;
    }
}

Someone wrote something very similar in C, where in the for-loop he used strlen. Which means that the following code actually is O(n²) because each time you are re-calculating the length...

for(int i = 0; i < strlen(str); ++i) {...}

Why the compiler cannot optimize it is another question. For one it's a method call and it is difficult to predict whether there's going to be any side-effect. So optimizing the call away can have other disastrous effect.

So store the count and re-use it if you know that the count is not going to change...

Alternatives to XML

Though not as much as the Jeff Atwood I don't like overuse of XML as well. In our last project we used XML in a bunch of places where it made sense and also planned to use it in bunch of other places where it didn't. For some strange reason some folks think its actually readable and suggested we use XML to dump the user actions we recorded because it's easy to parse and is human readable/editable. While I'm perfectly fine doing it in XML, but definitely not for that reason.

Anyways, sense prevailed and even though we do store it in XML we dump out automation source code in obviously more readable C#/VB.NET.

Before I completely get sidetracked let me state that this post is not about XML or about JSON but about the fact  that there exists many alternatives to both. Head on to here (via Coding Horror)...

What is the similarity between Windows and Textile Industry

...they both use threads and fibers :)

Most people are aware of processes and threads. Windows offers an even finer granularity over execution. This is called Fiber. To quote MSDN

A fiber is a unit of execution that must be manually scheduled by the application. Fibers run in the context of the threads that schedule them.

The obvious question that should come to managed code developers is whether .NET supports Fibers? The answer is from 2.0, CLR does support fiber mode. This means that there are hosting APIs using which a host can make CLR use Fibers to run its threads. So in effect there's no requirement that a .NET thread be tied to the underlying OS's threads. A classic example is that of SQL Server which hosts .NET in fiber mode because it wants to take care of scheduling directly. Head over to here (scroll down to SQL Server section) for an excellent read about this topic.

There's also the book Customizing the Microsoft .NET Framework Common Language Runtime written by Steven Pratschner which has a chapter on customizing CLR to use Fibers. I have already ordered the book. Once it comes in and I get a chance to read it, I'll post more about this.

Utter Geeky madness

A bunch of geeks (including my previous team mate Srivatsn) from the IronPython team discussing pinky and inventing a new form of encoding.

My workstation layout

Our team just moved to a new building in Microsoft India campus. A lot of people were going around checking out other people's office. I got asked couple of times about my workstation layout and thought I'll do a quick post on that.

Like most people in our team I use a dual monitor setup. Last time I estimated I spend about 12% of my life looking for things (40% of it for my car keys). So even though there are people who use 7 monitors I'm never going to join that gang and bump that number to 30% by adding the time to search for my app window. And Microsoft will definitely not fund those many monitors either :). So for me 2 is enough.

Both monitors I have, are standard HP1965 (19" monitors) hooked on to ATI Radeon cards. One of the monitors (the one on the left) is looped through a KVM switch and I can rotate that among the other 2 machines that I have. The other I have rotated in portrait (vertical) mode and use it primarily for coding. The image below should explain why

This provides a much better code view. In the font size I use (Consolas 9pt) I can see 74 lines of code vs 54 in the landscape mode. So this means 37% more!!! Since I have ATI card I use Catalyst Control Center to rotate the display.

I also prefer dark background and use white/light-color text on it. My eyes feel better with it. I keep both Visual Studio and GVim in dark color mode. You can download my vssettings from here and .vimrc from here.

That kind of rounds up the workstation layout that I use in office. I try my best not to work on the laptop directly.  I TS on to it in case I need to use it for any reason. When I took the picture it was quietly napping on the other side of my office :)

Struggling with email overload

It literally rains email at Microsoft (if you've been to Seattle/Redmond you know why :) )

I've always struggled to keep up with the email in Microsoft. When I joined I was stunned with the downpour. The number of email I got on the first day was more than what I got in a month in Adobe. The situation worsened when I went through team transition last month because for some time I had to listen to the email threads of both teams DLs (distribution list).

I have tried using various techniques to cope before. This included complex labyrinth of folders (I've met folks with 9 level deep folder nesting), rules, search folders, you name it!!

All of them failed until I saw this post from John Lam. This talks about reverse pimping outlook. Even though I didn't go to the extreme he did, I basically got the following done

1. Removed all rules, toolbars, folders
2. Created 3 simple folders Archive, Followup, Automated
3. Created one big fat rule to move all emails from automated DLs (e.g. checkin notices) to the automated folder
4. Copy pasted macros from Johns blog and setup toolbar buttons to launch these macros and also associated short-cuts with these. Go to John's blog for the macros or download from here
5. Effectively all emails from human beings land up in my inbox.
6. When an email comes I read it. After that I have only 3 options
   1. Delete it
   2. Hit Alt+R to archive it (this launches a macro to mark the email as read and moves it to the archive folder)
   3. Hit Alt+U for follow up (this launches a macro to flag the email to be replied by EOD and moves it to the follow up folder)

This ensures that I read all emails that come to me, I never miss an email now. I go on hitting zero emails in the inbox couple of time a day. Couple of times a day I scan the followup folder to ensure that I have replied/taken-action on all emails in the follow-up folder.

Even though the process sounds complex it's working miraculously for me for the last two months. I can finally forget about email overload.

My outlook looks as shown below. It's more cluttered than John's version because I need to see upcoming meetings in the right pane.

Forcing a Garbage Collection is not a good idea

 

Someone asked on a DL about when to force a GC using GC.Collect. This has been answered by many experts before, but I wanted to re-iterate. The simple answer is

"extremely rarely from production code and if used ensure you have consulted the GC folks of your platform".

Lets dissect the response...

Production Code

The "production code" bit is key here. It is always fine to call GC.Collect from test/debug code when you want to ensure your application performs fine when a sudden GC comes up or you want to verify all your objects have been disposed properly or the finalizers behave correctly. All discussion below is relevant only to shipping production code.

Rarely

A lot of folks jumped into the thread giving examples of where they have done/seen GC.Collect being used successfully. I tried understanding each of the scenarios and explaining why in my opinion it is not required and doesn't qualify to make it to the rare scenario. I have copy pasted some of these scenarios with my response below (with some modifications).

1. For example, your process has a class which wraps a native handle and implements Dispose pattern. And the handle will used in exclusive mode. The client of this class forgets to call Dispose/Close to release the native handle (they rely on Finalizer), then other process (suppose the native handle is inter-process resource) have to wait until next GC or even full GC to run Finalizer, since when Finalizer will run is not expected – other process will suffer from waiting such exclusive sharing resource…
   This is a program bug. You shouldn’t be covering a dispose pattern misuse with a GC call. You are essentially shipping buggy code or in case you provide the framework then allowing users to write buggy code. This should be fixed by ensuring that the clients call the dispose and not by forcing GC. I would suggest adding an Assert in the finalizer in your debug bits to ensure that you fail in the test scenario. In case of Fx write the right code and let performance issues surface so that users also writes the right code
   
2. Robotics might be another example—you might want time-certain sampling and processing of data.
   .NET is not a Real time system. If you assume or try to simulate Real Time operations on it then I have only sympathy to offer :). Is the next suggestion to call all methods in advance so that they are already jitted?
   
3. Another case I can think of is the program is either ill-designed or designed specially to have a lot of Finalizers (they wrap a lot of native resources in the design?). Objects with Finalizer cannot be collected in generation 0, at least generation 1, and have great chance to go to generation 2…
   This is not correct. The dispose pattern is there exactly for this reason. Any reason why you are not using dispose pattern and using GC suppress in the dispose method?
   
4. Well, one “real world” scenario that I know of is in a source control file diff creation utility.  It loops through processing each file in the pack, and loads that entire file into memory in order to do so it calls GC.Collect when it’s finished with each file, so that the GC can reclaim the large strings that are allocated.
   Why cannot it just not do anything and is there a perf measurement to indicate otherwise? GC has per-run overhead. So incase nothing is done it may so happen that for a short diff creation the GC is never run or atleast run for every 10 files handled leading to less number of runs and hence better perf. For a batch system where there is no user interaction happening in the middle what is the issue if there is a system decided GC in the middle of the next file?
   
5. A rare case in my mind is you allocate a lot of large objects > 85k bytes, and such size objects will be treated as generation 2 objects. You do not want to wait for next full GC to run (normally GC clears generation 0 or generation 1), you want to compact managed heap as soon as possible.
   Is it paranoia or some real reason? If it holds native resources then you are covered by dispose patterns and if you are considering memory pressure then isn’t GC there to figure out when to do it for you?

In effect most usage are redundant.

Question is then what qualifies as a rare scenario where you want to do a GC.Collect. This has been explained by Rico Mariani (here) and Patrick Dussud (here).

‘In a nutshell, don’t call it, unless your code is unloading large amounts of data at well-understood, non-repeating points (like at the end of a level in a game), where you need to discard large amounts of data that will no longer be used.”

Its almost always when you know for sure a GC run is coming ahead (which you completely understand and maybe confirmed with the GC guys of your framework) and you want to control the exact point when you want it to happen. E.g.in case of a game level end you have burned out all the data and you know that you can discard them and if you don’t GC will start after 6 frames of rendering in your next level and you are better off doing it now as the system is idle and you’d drop a frame of two if it happened in the middle of the next frame.

And obviously you call GC.Collect if you found an issue reported/discussed in the forums and you have figured out a GC bug which you want to work around.

I would highly recommend seeing this video where Patrick Dussud the father of .NET GC explains why apparent GC issues may actually be side-effect of other things (e.g finalizes stuck trying to delete behind the scene STA COM objects).

What is the problem with calling GC.Collect

So why are folks against calling GC.Collect? There are multiple reasons

1. There's an inherent assumption that the user knows more about when the GC is run. This cannot be true because according to CLR spec there is no standard time. See here. Since GC is plugged into the execution engine it knows best of the system state and knows when to fire. With Silver Light and other cross-plat technologies being mainstream it will become harder and harder to predict where your app is run. There's already 3 separate GCs the desktop, server and compact framework. Silver light will bring in more and your assumptions can be totally wrong.
   
2. GC has some cost (rather large):
   GC is run by first marking all the objects and then cleaning them. So whether garbage or not the objects will be touched and it takes awful amount of time to do that. I've seen folks measure the time to do GC.Collect and figure out the time taken. This is not correct because GC.Collect fires the collection and returns immediately. Later GC goes about freezing all the threads. So GC time is way more than what collect takes and you need to monitor performance counter to figure out what is going on,
   
3. GC could be self tuning:
   The desktop GC for example tunes itself based on historical data. Lets assume that a large collection just happened which cleaned up 100mb of data. Incidentally exactly after that a forced GC happened which resulted in no data to be cleaned up. GC learns that collection is not helping and next time when a real collection is to be fired (low memory condition) it simply backs off based on the historical data. However, if the forced GC didn't occur it'd have remembered that 100mb got cleared and would've jumped in right away.

Both 2 and 3 are GC implementation specific (differs across desktop and Compact GC) stressing the first point which is most assumption are implementation details of the GC and may/will change jeopardizing the attempt to try out-guess the GC when to run.

Ensuring only one instance of an application can be run and then breaking that approach

A question posted on a DL was as follows

"I have an application that allows only one instance on the desktop, how do I force it to open multiple instance for some testing purposes".

Disclaimer: Note that this is kinda hacky and only relevant if you are a tester trying to break something. This shouldn't be treated as a way to achieve anything productive.

Since the user didn't explain which approach is used to ensure only one instance is allowed I'll try listing down the methods I know off and how I can break them.

Using named Mutex:
This is the more resilient approach taken by applications like Windows Media player. Here the application tries to create a named Mutex with sufficiently complicated name to avoid collision. E.g. namespace.app.guid and then try to acquire the mutex. In case it fails to acquire it then an instance is already running and it closes itself. If it acquired the mutex it means it is the first instance and it continues normally. An approach is outlined here.

Even though this seems to be the most robust solution this can be made to fail as well. If an user has sufficient permission he can do a CloseHandle on a Mutex handle created by some other application as well. For this you don't even need to write code. Do the following

1. Download and install Process Explorer.
2. Launch it (might need elevation on Vista)
3. Have an application that uses this approach like my application named SingleInstanceApp.exe. Try launching a second instance and it will fail to open it.
4. Click on the app finder toolbar button in Process Explorer and drag it to SingleInstanceApp window. This will get this application selected in process explorer.
5. Select View->Lower Pane View ->Handles or simply Ctrl+H
6. This will show all the handles created by SingleInstanceApp.
7. Scroll down looking for the mutexes. At the end the screen looks something as follows
8. Right click on the mutex and choose Close Handle.
9. Now try creating the second instance and it will open fine.

Enumerate the names of applications running and see if the app is already running:
This uses a combination of Process.GetCurrentProcess and Process.GetProcessByName(), see an implementation here.

This is easy to break. To do a DOS (Deinal of Service or DOS) just  create another application with the name that the app expects and per-launch it. To open two instances copy the application to another name and open that first and then the original application.

Enumerate Windows

This involves iterating through all the open windows in the system using EnumWindows and then seeing if a Window with a given name or classname is already present. Since the system doesn't guarantee uniqueness in either of the text or class name this can be broken in a similar approach to the first one. However, there are some complications to it as it is difficult to change window text from outside. However, a combination of code injection and SetWindowText win32 API should work.

Trivia: How does CLR create a OutOfMemoryException

 

When the system goes out of memory a OutOfMemoryException is thrown. Similarly for stack overflow a StackOverFlorException is thrown. Now typically an exception is thrown as follows (or the corresponding native way)

throw new System.OutOfMemoryException();

But when the system is already out of memory there's a little chance that creating an exception object will succeed. So how does the CLR create these exception objects?

The answer is trivial and as expect, at the very application start it creates all these exception objects and stores them in a static list. In case these exceptions are to be thrown then it is fetched from this list and thrown.

The consequence of handling Stack overflow is even bigger because once the stack has overflown calling even a single method in that thread is equally dangerous and can cause further corruption. That is the material for another post :)

When does the .NET Compact Framework Garbage Collector run

 

Other than the exact when part this post applies equally for the desktop portion.

Disclaimer: This post is mainly indicative. When the GC runs is an implementation detail and shouldn't be relied on. This is not part of any contract or specification and may (most probably will) change.

The ECMA specification for Garbage Collection is intentionally vague about when an object will be collected (or freed up). The memory management cycle mentioned in the spec is as follows

1. When the object is created, memory is allocated for it, the constructor is run, and the object is considered live.

2. If no part of the object can be accessed by any possible continuation of execution, other than the running of finalizers, the object is considered no longer in use and it becomes eligible for finalization. [Note: Implementations might choose to analyze code to determine which references to an object can be used in the future. For instance, if a local variable that is in scope is the only existing reference to an object, but that local variable is never referred to in any possible continuation of execution from the current execution point in the procedure, an implementation might (but is not required to) treat the object as no longer in use. end note]

3. Once the object is eligible for finalization, at some unspecified later time the finalizer (§17.12) (if any) for the object is run. Unless overridden by explicit calls, the finalizer for the object is run once only.

4. Once the finalizer for an object is run, if that object, or any part of it, cannot be accessed by any possible continuation of execution, including the running of finalizers, the object is considered inaccessible and the object becomes eligible for collection.

5. Finally, at some time after the object becomes eligible for collection, the garbage collector frees the memory associated with that object.

As you can see the specification doesn't even need an implementation to do code analysis to figure out garbage. It can simply use scoping rules (used anyway by the compiler to detect valid variable usage) for garbage detection. The specification also doesn't specify eventually when the objects are collected. The only need is that it is finalized and freed unspecified time later than the time when it goes out of use. This convenient open statement lets each GC implementers to choose whatever they deem fit for the purpose. Since even thread is not specified a concurrent GC or a non-concurrent GC can be used.

However, everyone wants to know exactly when their platform's GC is run. Here goes the non-exhaustive list for the .NET Compact Framework's Garbage Collector

1. Out of memory condition:
   When the system fails to allocate or re-allocate memory a full GC is run to free up as much as possible and then allocation is re-attempted once more before giving up
2. After some significant allocation:
   If one megabyte of memory is allocated since the last garbage collection then GC is fired.
3. Failure of allocating some native resources:
   Internally .NET uses various native resources. Some native resource allocation can fail due to memory issues and GC is run before re-attempting
4. Profiler:
   Profiler APIs build into the framework can force a GC
5. Forced GC:
   System.GC class is provided in the BCL to force a collection
   

Obviously there can be small differences across various platforms on which .NET CF is implemented. However, the differences are small enough to ignore for this discussion.

Even though this list seems small it works pretty well across disparate systems like XBox and Windows Mobile. In the next post I'll try to get into why "production user code should never do a forced GC". I know that statement is a bit controversial (at least it got so in an internal thread).

Which end of the egg do you crack. Putting it differently, what is your Endianess

Few people seem to know that the word Endianess comes from Gullivers travels. In Gulliver's travel where there were two types of people, the Lilliputs who cracked the small side of their soft boiled eggs and the Blefuscu who used the big side (Big-Endian). Since I'm well networked these days (over Orkut/Facebook/LinkedIn) I make a conscious decision to be Big-Endian while cracking an egg as its the preferred network endianess.

I do not want to delve into the holy endianess war especially because most modern processors allow hardware/software methods to switch it (reminds me of some politicians though :))

However, I do use bit/byte questions as the acid test for fly/no-fly interviews as suggested by this guy. One of them involves asking about the whole endianess business and a code snippet to find out the endianess of the current system. I'm usually looking for something as below

short a = 1;
    if (*((char *)&a) == 1)
        printf("Little Endian\n");
    else
        printf("Big Endian\n");

The idea is not to look for an exact code but to ensure that there's no complete ignorance of this area...

Multi-threaded developer

Once upon a time a kick-ass developer I knew told me that a good developer needs to be multi-threaded and run the following threads

1. The main worker thread: This is the one used to code up 3-tier applications for your employer. This pays for the rent and for that fancy big car.
   
2. The core thread: This is the one that's used to read up data-structures, OS and other fundamental CS stuff which helps you to join into discussion when folks are discussing threaded BST. This also helps you to crack an interview in case you need a new job.
   
3. The cool thread: This is the one you use to read up about Silver Light Mobile, ASP.NET MVC Framework, JSON, etc. This keeps you up to date and let's you hang around with other geeks like Scot Hanselman and Don Box

The same person also told me that real programmers do not blog, he asked me "Do you know about Dave Cutler or Linus Torvald's blog?". So I guess we can safely ignore him :)

Time for a change

I've been working for some time in Visual Studio Team System (first on the TFS server and then on the testing infrastructure). I was looking for a change and then I happened to talk with the GPM of the .NET COMPACT FRAMEWORK team. The following explains in brief what followed...

... and so I landed a job in the .NETCF Garbage Collector and other parts of the execution engine. I'm sure I'll be super busy as the team is heads down into getting Silver Light onto mobile devices and going cross-plat (I guess most folks have heard about SL on Symbian/Nokia).

What this means is that I can (or rather I have to) muck more with .NET (both desktop and CF) without any guilt and can claim that blogging about them is actually a part of my job. So what this means for this blog is that there'll be more hardcore stuff about .NET and an additional Tag called NETCF when I post specifically about it.

In case you are in India and want to work in a Kick-ass team delivering one of the most key technologies for the industry then drop me a line at abhinaba.basu @ you know where. Bragging rights will come for free when you point to all those cool WinMobile/S60/XBox/Zune devices and claim how you coded .NET for it....

Wish me luck...

Coolest reception

 

Rental car chauffeurs waiting for some software company employees. No marks for guessing the name of the company :)

International Mother Language Day is back and being celebrated with a lunar eclipse

For starters it doesn't include C++ or C#, even though your mom codes in it :) ...

A lot of people speaking English natively forget the importance of mother language due to its predominance. They take their language for granted. However, each year a bunch of languages become extinct, the latest being Eyak, which got extinct exactly a month ago with the death of Marie Smith Jones the last native Eyak speaking person.

To me this day (21st Feb) is even more special because on this day in 1952 people in Bangladesh laid down their lives while demanding the right to use their own (and mine) mother language, Bangla.

I believe that if we don't actively try to preserve our mother language they will slowly become extinct. One of the most important things to preserve a language is to ensure that they are better covered by technology. Until XP complex script handling was not enabled by default. This resulted in Bangla and other Indic language to be rendered completely wrong on XP. This was a serious deterrent to use Bangla on Windows. I used to have Bangla signature in my email and got countless replies indicating the spelling is wrong. I always replied back to them about how to turn the complex script handling on. Things are changing rapidly, Vista has this on by default and with better keyboard and font support I'm sure using Bangla on Computers will become really easy.

Linked on the msdn forum for all the wrong reasons

Someone saw a message in their Team Foundation Server history which was "Edited by God" and asked how is this possible on the msdn forum. Someone actual read my blog (or some other's but I tend to believe mine 'cause I'm linked in that page)  which shows exactly how to do that. So now I have a poor-joke (PJ) of mine being featured in msdn :)

For the last couple of days a vast majority of the movie that I'm seeing is somehow landing in Africa. Since I didn't choose any of them (Amit did), it wasn't a conscious decision. Example include The Interpreter, Blood Diamond, Duma.

In all of these movies African music is used in the background, and I've began to like them a lot. I have no idea about the tribe/nationality of these but the sound is just enchanting. Any idea where I can get some good original African music on the web? Buying is also an option if Indian stores/sites have them (which I heavily doubt).

Update: A comprehensive list from someone who obviously knows a lot more about this, link