If you ever are in a situation where you need to identify production code issues or perf. issues, etc. (and who has not been there), then I would recommend checking out AppSight Black Box - a “kick-ass” product that I had an opportunity to play with sometime last year. They have solutions covering both .NET and J2EE and is very very impressive - I just want to know how the heck do they do it. You have to see it in person to believe it - and I have grilled these guys extensively using some real code, so if the demo looks to good to be true, well it is true! Here is the excerpt from their site which really does not do them justice:

Identify’s patented Black Box is a lightweight software module, which operates like a Black Box flight recorder on a plane. Deployed on servers and/or clients, AppSight Black Boxes record application execution at multiple, synchronized levels, based on a dynamic, user-defined recording profile. Easy to deploy, the Black Box requires no changes to source code or executables.

User:
Client-side: Video-style capture of user actions and screen events
Server-side: User sessions, ASP.NET Web pages and services accessed by each user

System:
Recording of .NET Web services calls, Remoting calls, database calls, assembly invocations, COM/COM+ calls, application and system configuration, application performance, .NET Framework resource utilization, and more

Code:
.NET Tracing calls; full trace of function calls and variables in unmanaged code

Black Boxes can be run continuously or when problems occur. The data recorded by the Black Boxes is stored in a centralized repository or saved as a highly compressed file, which can be sent by email or attached to a trouble ticket.

Using the AppSight Console, all teams engaged in problem resolution replay Black Box recordings and drill down based on skill set and responsibility to identify the root cause of problems.

PS: No, I am not getting a kickback from them to drooooool over this… :)

[Listening to: Rabba - Musafir [Club Mix - CD1] - Musafir (07:50)]