dropthings Wiki & Documentation Rss Feed

Updated Wiki: Home

oazabir — Tue, 06 Oct 2009 14:37:05 GMT

Project Description
Ajax Web Portal built on Linq, Workflow Foundation and ASP.NET AJAX. Code is in Visual Studio 2008 using .NET 3.0 and .NET 3.5.

Production site

http://dropthings.omaralzabir.com
For latest news and updates, visit my blog
http://msmvps.com/blogs/omar/

Source Code

Latest source code and releases:
http://code.google.com/p/dropthings/

Click here for the code that matches with the book

Book about this project

This book explains how this project has been built step by step. It also explains many advance AJAX concepts, development and production challenges for building and maintaining a high volume production website.

Get it from Amazon:
http://www.amazon.com/Building-Web-2-0-Portal-ASP-NET/dp/0596510500

Technologies

ASP.NET 2.0
jQuery
ASP.NET AJAX (.NET 3.5)
Silverlight
Linq to Sql
Linq to Xml
Workflow Foundation (.NET 3.0)
Visual Studio 2008 and SQL Server 2005

Features

Configure widgets for specific user roles.
Configure different default page setup for different roles - Managers get some widgets, Employees get different widgets and so on.
Ability to define page setup for anonymous users and different page setup for logged in users.
Multiple Tabs.
Customizable Widget Gallery.
Different column setup.
Build your own widgets using Silverlight, ASP.NET or plain Javascripts.
Host widgets on any page of your website
Easy to integrate Dropthings to your own website e.g. http://myoffice.bt.com is built on top of Dropthings.
Business Layer is entirely Workflow Foundation driven - easy to plug-in your own business logic as Activities.

Join Us
If you are a seasoned ASP.NET Developer and have ASP.NET AJAX skill, come and join us to make Dropthings a feature rich Ajax Web Portal. Please email me at "OmarALZabir" at gmail dot com.

What is an AJAX Portal
A Portal refers to a page that allows users to customize their own homepage by dragging and dropping widgets onto the page. This approach gives users complete control over what content they see on their page, where they want to see it, and how they want to interact with it.

A widget is a discrete piece on a Web page that performs a particular function and comes with its own UI and set of features. Examples of widgets include a to-do-list, an address book, a contact list, an RSS feed, or even a clock, calendar, playlist, stock ticker, weather report, traffic report, dictionary, game, or almost anything you can imagine that can be packaged up and dropped on a Web page. In a corporate environment, widgets can connect to internal systems, such as an Expense Tracker widget that interacts directly with the internal Accounting System. If you are familiar with Sharepoint Portal, then you already know about Widgets. They are called Web parts in Sharepoint’s term and also in ASP.NET 2.0.

Portals are powerful RSS aggregation platform. You can put as many RSS widgets as you like on your page and get fresh content delivered to you as soon as it is published. Some Portal like Pageflakes archives RSS for a long time and thus you can go back in time and read older posts, save posts, and forward interesting articles to your friends.

An Ajax-powered portal is specifically a portal that uses Ajax technologies to create richer experiences for its users. It is one step ahead of previous generation portals like My Yahoo or MSN.com, because it gives you state-of-the-art UI that behaves more like a Windows client application -- with widgets, animations, popups, client side data grids, and other effects not usually found on a non-Ajax Web portal .

How to run the project

Download the latest source code from Google Code site http://code.google.com/p/dropthings/
Follow the Readme.txt

How is ASP.NET AJAX used in this project?
It is an N-tier application, with a user interface (UI) layer, a business layer, and a data access layer. I have used ASP.NET AJAX to implement the UI layer of the portal application which includes the homepage and the widgets’ UI. ASP.NET AJAX provides the framework for loading widgets onto the home page, updating widgets without doing any postbacks (via UpdatePanel), and changing page layout by dragging and dropping widgets on the page. It also provides a rich collection of Control Extenders, that add cool effects like fade in/fade out, smooth transitions, and client side animations . You can add to the rich clientside experience by providing auto-completion behavior on text boxes, asynchronous data loading via webservice calls, and client side paging, sorting and many more.

How is .NET 3.5 used in this project
The business layer of the application is built with the Workflow Foundation in .NET 3.0 . Major operations like a first-time user visit, a subsequent user visit, adding a new widget, and creating a new page are all orchestrated using workflow . The workflows contain all the business rules and activities needed to complete each operation. For example, the "New User Visit" workflow creates the user account, populates the user profile with default values, creates some default pages, populates them with specific widgets, etc. Such compound operations are very easy to build with Workflows , which enables you to break the complete workflow operation into smaller chunks named Activities. Each Activity does a very small amount of work. It talks to the data access layer and performs the task. The data access layer is built with .NET 3.5 , utilizing LINQ to SQL .
The web project and the widgets make good use of .NET 3.5 by utilizing lambda expressions , LINQ to SQL, and LINQ to XML. You will use Linq queries to work with collections and database rows. Widgets make good use of Linq to Xml in order to consume XML from external data sources.

Warning: Dropthings.com is a very simple, open-source example of what can be done with AJAX and Microsoft technologies. It is intended for educational purposes only. Dropthings.com has absolutely nothing to do with http://www.pageflakes.com. But this project does a good job to show you how all the new hot technologies work together in a working web application that's production ready.

My Blog

---

Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 News Feed

7 tips for for loading Javascript rich Web 2.0-like sites significantly faster

Friday, September 25, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Introduction
When you create rich Ajax application, you use external JavaScript frameworks and you have your own homemade code that drives your application. The problem with well known JavaScript framework is, they offer rich set of features which are not always necessary in its entirety. You may end up using only 30% of jQuery but you still download the full jQuery framework. So, you are downloading 70% unnecessary scripts. Similarly, you might have written your own javascripts which are not always used. There might be features which are not used when the site loads for the first time, resulting in unnecessary download during initial load. Initial loading time is crucial – it can make or break your website. We did some analysis and found that every 500ms we added to initial loading, we lost approx 30% traffic who never wait for the whole page to load and just close browser or go away. So, saving initial loading time, even by couple of hundred milliseconds, is crucial for survival of a startup, especially if it’s a Rich AJAX website.
You must have noticed Microsoft’s new tool Doloto which helps solve the following problem:
Modern Web 2.0 applications, such as GMail, Live Maps, Facebook and many others, use a combination of Dynamic HTML, JavaScript and other Web browser technologies commonly referred as AJAX to push page generation and content manipulation to the client web browser. This improves the responsiveness of these network-bound applications, but the shift of application execution from a back-end server to the client also often dramatically increases the amount of code that must first be downloaded to the browser. This creates an unfortunate Catch-22: to create responsive distributed Web 2.0 applications developers move code to the client, but for an application to be responsive, the code must first be transferred there, which takes time.
Microsoft Research looked at this problem and published this research paper in 2008, where they showed how much improvement can be achieved on initial loading if there was a way to split the javascripts frameworks into two parts – one primary part which is absolutely essential for initial rendering of the page and one auxiliary part which is not essential for initial load and can be downloaded later or on-demand when user does some action. They looked at my earlier startup Pageflakes and reported:
2.2.2 Dynamic Loading: Pageflakes
A contrast to Bunny Hunt is the Pageflakes application, an
industrial-strength mashup page providing portal-like functionality.
While the download size for Pageflakes is over 1 MB, its initial
execution time appears to be quite fast. Examining network activity
reveals that Pageflakes downloads only a small stub of code
with the initial page, and loads the rest of its code dynamically in
the background. As illustrated by Pageflakes, developers today can
use dynamic code loading to improve their web...

Visit my Blog for more details.

Windows 7 64bit works!

Saturday, September 19, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Windows 7 64bit finally works! This is the first 64bit OS I could really use in my daily acitvities. I tried Vista 64bit, it was unreliable. It would show blue screen right when I am about to make a presentation to the CEO. Until Microsoft released SP1, Vista 64 bit was not usable at all. Then came Windows 7 beta. I immediately tried the 64bit version of Windows 7 beta. It was even worse than Vista. It would crash every now and then – waking up from standby, trying to do livemeeting share, switching screens, plugging in external USB drives and what not. So, I patiently waited for the final version to come out before I get on installing it on all my laptops. Happy to say, the final version works perfectly on HP tx2000 Tablet PC, DELL Vostro 1500, DELL Inspiron 1520. Once you do a full windows update and install some drivers here and there, it all works perfectly. And let me say, Windows 7 is beautiful. I found back the joy of working on computers again!
Working on 64bit Operating System is challenging. You don’t always find the right printer driver. Your cool external USB speakers won’t work – even if it is made by Microsoft. And above all, there’s that C:\Windows\Winsxs folder which keeps increasing forever. By the time I was done with Vista 64bit (two years approx in business), my Winsxs folder was staggering 26 GB eating up every bit out of my C: partition. I had no choice but to format and start over. It seems like this folder keeps copy of every single DLL version it ever sees. The more windows update I do, the larger it gets. Now on a fresh new Windows 7 installation, after installing VS 2008, Office Applications, Windows Live applications and some handy tools, the Winsxs folder is 5.62 GB. Let’s see how it keeps growing over the year. A useful information for 64bit wannabes, make sure your C partition is at least 60 GB. I just installed Windows 7 64bit 3 days back and it has already taken 31 GB space.

Since I am doing a totally useless post, let me sprinkle some productivity tips on it before you lose interest reading my blog.
I realized I do a lot of context swiching. I get over 200 mails per day, so I pretty much switch focus from Visual Studio/Browser to Outlook once every minute, which is big cencentration killer. So, I tried the above setup on my 25” screen and it works great!
The left half of the screen is visual studio and the right half screen shows Outlook and my todolist. As you see, I can see the emails coming up on Outlook without ever switching. The Visual Studio screen width is the right size to read code without horizontally scrolling. The right bottom half of the screen shows my toodlist so that I am always doing the right task from my todolist and not wondering around heedless. If I browse, I bring up the browser on top of the Visual Studio and keep the right half same so that while browsing I am not missing important mails and I still have an eye on my next actions.
I have been using Toodledo for a year. I...

Visit my Blog for more details.

AspectF fluent way to put Aspects into your code for separation of concern

Saturday, September 19, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Aspects are common features that you write every now and then in different parts of your project. it can be some specific way of handling exceptions in your code, or logging method calls, or timing execution of methods, or retrying some methods and so on. If you are not doing it using any Aspect Oriented Programming framework, then you are repeating a lot of similar code throughout the project, which is making your code hard to maintain. For ex, say you have a business layer where methods need to be logged, errors need to be handled in a certain way, execution needs to be timed, database operations need to be retried and so on. So, you write code like this: public bool InsertCustomer(string firstName, string lastName, int age,
Dictionary<string, string> attributes)
{
if (string.IsNullOrEmpty(firstName))
throw new ApplicationException("first name cannot be empty");

if (string.IsNullOrEmpty(lastName))
throw new ApplicationException("last name cannot be empty");

if (age < 0)
throw new ApplicationException("Age must be non-zero");

if (null == attributes)
throw new ApplicationException("Attributes must not be null");

// Log customer inserts and time the execution
Logger.Writer.WriteLine("Inserting customer data...");
DateTime start = DateTime.Now;

try
{
CustomerData data = new CustomerData();
bool result = data.Insert(firstName, lastName, age, attributes);
if (result == true)
{
Logger.Writer.Write("Successfully inserted customer data in "
+ (DateTime.Now-start).TotalSeconds + " seconds");
}
return result;
}
catch (Exception x)
{
// Try once more, may be it was a network blip or some temporary downtime
try
{
CustomerData data = new CustomerData();
if (result == true)
{
Logger.Writer.Write("Successfully inserted customer data in "
+ (DateTime.Now-start).TotalSeconds + " seconds");
}
return result;
}
catch
{
// Failed on retry, safe to assume permanent failure.

// Log the exceptions produced
Exception current = x;
int indent = 0;
while (current != null)
{
string message = new string(Enumerable.Repeat('\t', indent).ToArray())
+ current.Message;
Debug.WriteLine(message);
Logger.Writer.WriteLine(message);
current = current.InnerException;
indent++;
}
Debug.WriteLine(x.StackTrace);
Logger.Writer.WriteLine(x.StackTrace);

return false;
}
}

}

Here you see the two lines of real code, which inserts the...

Visit my Blog for more details.

MSDN Day @ Dhaka

Wednesday, June 17, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Venue.
IDB Auditorium
E/8-A Rokeya Sharani,
Sher-e-Bangla Nagar,
Agargaon, Dhaka 1207
Saturday June 20th 9:00 AM to 6:00 PM
Microsoft Community in Bangladesh proudly presents Microsoft Day @ Dhaka. This is a special day dedicated to all Microsoft technology professionals and students in Bangladesh. We will be having the best Microsoft community technologists from Bangladesh - Microsoft Most Valuable Professionals (MVPs) delivering sessions at the event. This technology marathon is a great opportunity to learn from the best and network with each other.
Both Microsoft developers and networking professionals would find the event worth attending.
The event will also feature a demo bonanza with Windows 7 and an extensive Question and Answer panel with the Bangladesh MVPs to answer your queries.
http://msdnbangladesh.net/content/msday.aspx
Please register soon! Limited sits. I have a session there as well.

AUDITORIUM – Dev Track
9:00 - 9:30: Opening Speech - Feroz Mahmood
9:30 - 10:30: Development in ASP.NET [LINQ, Web Forms, Dynamic Data] - Tanzim Saqib
10:30 - 11:15: My First ASP.NET MVC App - Mehfuz Hossain
11:15 - 11:45 : Unit Testing in MVC and Demo of dotnetshoutouts.com - Kazi Manzur Rashid
11:45- 12:30: Developing in Silverlight - Faisal Hossain Khan
12:30 - 1:30: Lunch
1:30 - 2:00 : Introduction to Sharepoint/ MOSS - Jannatul Ferdous
2:00 - 2:45: Production Challenges of ASP.NET Websites - Omar Al Zabir
2:45 - 3:15: Windows Azure - Ashic
3:15 - 3:45: Tea Break
3:45 - 4:30: Overview of Visual Studio Team System 2010 - Mohammad Ashraful Alam
4:30 - 5:30: Features of Windows 7 - IE8 and Windows Live Features - Omi Azad
BREAK OUT – IT Pro Track
9:00 - 9:30: Opening Speech - Feroz Mahmood [IN AUDITORIUM]
9:30 – 10:30: Exchange Server 2010
10:30 - 11:30: Windows Server 2008 - Virtualisation & HyperV
11:30 - 12:30: Talking Windows Server 2008 and R2 [Windows Client & Windows Server 2008 NAP – Better Together] - Anwar Hossain (Technical Specialist, MS Bangladesh)
12:30 - 1:30: Lunch
1:30 - 2:15: Session on MS Project & EPM : M. Manzurur Rahman (CEO, ICT Alliance)
2:15 - 3:00: Office 2007
3:00 - 3:30: Tea Break
3:30 - 4:30: Introduction to SQL Server 2008
4:30 - 5:30: Features of Windows 7 - IE8 and Windows Live Features - Omi Azad [IN AUDITORIUM]

Visit my Blog for more details.

ASP.NET AJAX testing made easy using Visual Studio 2008 Web Test

Monday, May 25, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Visual Studio 2008 comes with rich Web Testing support, but it’s not rich enough to test highly dynamic AJAX websites where the page content is generated dynamically from database and the same page output changes very frequently based on some external data source e.g. RSS feed. Although you can use the Web Test Record feature to record some browser actions by running a real browser and then play it back. But if the page that you are testing changes everytime you visit the page, then your recorded tests no longer work as expected. The problem with recorded Web Test is that it stores the generated ASP.NET Control ID, Form field names inside the test. If the page is no longer producing the same ASP.NET Control ID or same Form fields, then the recorded test no longer works. A very simple example is in VS Web Test, you can say “click the button with ID ctrl00_UpdatePanel003_SubmitButton002”, but you cannot say “click the 2nd Submit button inside the third UpdatePanel”. Another key limitation is in Web Tests, you cannot address Controls using the Server side Control ID like “SubmitButton”. You have to always use the generated Client ID which is something weird like “ctrl_00_SomeControl001_SubmitButton”. Moreover, if you are making AJAX calls where certain call returns some JSON or updates some UpdatePanel and then based on the server returned response, you want to make further AJAX calls or post the refreshed UpdatePanel, then recorded tests don’t work properly. You *do* have the option to write the tests hand coded and write code to handle such scenario but it’s pretty difficult to write hand coded tests when you are using UpdatePanels because you have to keep track of the page viewstates, form hidden variables etc across async post backs. So, I have built a library that makes it significantly easier to test dynamic AJAX websites and UpdatePanel rich web pages. There are several ExtractionRule and ValidationRule available in the library which makes testing Cookies, Response Headers, JSON output, discovering all UpdatePanel in a page, finding controls in the response body, finding controls inside some UpdatePanel all very easy.
First, let me give you an example of what can be tested using this library. My open source project Dropthings produces a Web 2.0 Start Page where the page is composed of widgets.

Each widget is composed of two UpdatePanel. There’s a header area in each widget which is one UpdatePanel and the body area is another UpdatePanel. Each widget is rendered from database using the unique ID of the widget row, which is an INT IDENTITY. Every page has unique widgets, with unique ASP.NET Control ID. As a result, there’s no way you can record a test and play it back because none of the ASP.NET Control IDs are ever same for the same page on different visits. This is where my library comes to the rescue.
See the web test I did:

This test simulates an anonymous user visit. When anonymous user visits Dropthings for the first...

Visit my Blog for more details.

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

Wednesday, April 08, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Dropthings – my open source Web 2.0 Ajax Portal has gone through a technology overhauling. Previously it was built using ASP.NET AJAX, a little bit of Workflow Foundation and Linq to SQL. Now Dropthings boasts full jQuery front-end combined with ASP.NET AJAX UpdatePanel, Silverlight widget, full Workflow Foundation implementation on the business layer, 100% Linq to SQL Compiled Queries on the data access layer, Dependency Injection and Inversion of Control (IoC) using Microsoft Enterprise Library 4.1 and Unity. It also has a ASP.NET AJAX Web Test framework that makes it real easy to write Web Tests that simulates real user actions on AJAX web pages. This article will walk you through the challenges in getting these new technologies to work in an ASP.NET website and how performance, scalability, extensibility and maintainability has significantly improved by the new technologies. Dropthings has been licensed for commercial use by prominent companies including BT Business, Intel, Microsoft IS, Denmark Government portal for Citizens; Startups like Limead and many more. So, this is serious stuff! There’s a very cool open source implementation of Dropthings framework available at National University of Singapore portal.
Visit: http://dropthings.omaralzabir.com

I have published a new article on this on CodeProject:
http://www.codeproject.com/KB/ajax/Web20Portal.aspx Get the source code
Latest source code is hosted at Google code:
http://code.google.com/p/dropthings
There’s a CodePlex site for documentation and issue tracking:
http://www.codeplex.com/dropthings
You will need Visual Studio 2008 Team Suite with Service Pack 1 and Silverlight 2 SDK in order to run all the projects. If you have only Visual Studio 2008 Professional, then you will have to remove the Dropthings.Test project. New features introduced
Dropthings new release has the following features: Template users – you can define a user who’s pages and widgets are used as a template for new users. Whatever you put in that template user’s pages, it will be copied for every new user. Thus this is an easier way to define the default pages and widgets for new users. Similarly you can do the same for a registered user. The template users can be defined in the web.config. Widget-to-Widget communication – Widgets can send message to each other. Widgets can subscribe to an Event Broker and exchange messages using a Pub-Sub pattern. WidgetZone – you can create any number of zones in any shape on the page. You can have widgets laid in horizontal layout, you can have zones on different places on the page and so on. With this zone model, you are no longer limited to the Page-Column model where you could only have N vertical columns. Role based widgets – now widgets are mapped to roles so that you can allow different users to see different widget list using ManageWidgetPersmission.aspx. Role based page setup – you can define page setup for different roles. For...

Visit my Blog for more details.

Memory Leak with delegates and workflow foundation

Saturday, March 14, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Recently after Load Testing my open source project Dropthings, I encountered a lot of memory leak. I found lots of Workflow Instances and Linq Entities were left in memory and never collected. After profiling the web application using .NET Memory Profiler, it showed the real picture:

It shows you that instances of the several types are being created but not being removed. You see the “New” column has positive value, but the “Remove” column has 0. That means new instances are being created, but not removed. Basically the way you do Memory Profiling is, you take two snapshots. Say you take one snapshot when you first visit your website. Then you do some action on the website that results in allocation of objects. Then you take another snapshot. When you compare both snapshots, you can see how many instances of classes were created between these two snapshots and how many were removed. If they are not equal, then you have leak. Generally in web application many objects are created on every page hit and the end of the request, all those objects are supposed to be released. If they are not released, then we have a problem. But that’s the scenario for desktop applications because in a desktop application, objects can remain in memory until app is closed. But you should know best from the code which objects were supposed to go out of scope and get released.
For beginners, leak means objects are being allocated but not being freed because someone is holding reference to the objects. When objects leak, they remain in memory forever, until the process (or app domain) is closed. So, if you have a leaky website, your website is continuously taking up memory until it runs out of memory on the web server and thus crash. So, memory leak is a bad – it prevents you from running your product for long duration and requires frequent restart of app pool.
So, the above screenshot shows Workflow and Linq related classes are not being removed, and thus leaking. This means somewhere workflow instances are not being released and thus all workflow related objects are remaining. You can see the number is same 48 for all workflow related objects. This is a good indication that, almost every instance of workflow is leaked because there were total 48 workflows created and ran. Moreover it indicates we have a leak from a top Workflow instance level, not in some specific Activity or somewhere deep in the code.
As the workflows use Linq stuff, they held reference to the Linq stuffs and thus the Linq stuffs leaked as well. Sometimes you might be looking for why A is leaking. But you actually end up finding that since B was holding reference to A and B was leaking and thus A was leaking as well. This is sometimes tricky to figure out and you spend a lot of time looking at the wrong direction.
Now let me show you the buggy code: ManualWorkflowSchedulerService manualScheduler =
workflowRuntime.GetService<ManualWorkflowSchedulerService>();

WorkflowInstance...

Visit my Blog for more details.

Optimize ASP.NET Membership Stored Procedures for greater speed and scalability

Friday, March 13, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Last year at Pageflakes, when we were getting millions of hits per day, we were having query timeout due to lock timeout and Transaction Deadlock errors. These locks were produced from aspnet_Users and aspnet_Membership tables. Since both of these tables are very high read (almost every request causes a read on these tables) and high write (every anonymous visit creates a row on aspnet_Users), there were just way too many locks created on these tables per second. SQL Counters showed thousands of locks per second being created. Moreover, we had queries that would select thousands of rows from these tables frequently and thus produced more locks for longer period, forcing other queries to timeout and thus throw errors on the website.

If you have read my last blog post, you know why such locks happen. Basically every table when it grows up to hold millions of records and becomes popular goes through this trouble. It’s just a part of scalability problem that is common to database. But we rarely take prevention about it in our early design.

The solution is simple, you should either have WITH (NOLOCK) or SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED before SELECT queries. Either of this will do. They tell SQL Server not to hold any lock on the table while it is reading the table. If some row is locked while the read is happening, it will just ignore that row. When you are reading a table thousand times per second, without these options, you are issuing lock on many places around the table thousand times per second. It not only makes read from table slower, but also so many lock prevents insert, update, delete from happening timely and thus queries timeout. If you have queries like “show the currently online users from last one hour based on LastActivityDate field”, that is going to issue such a wide lock that even other harmless select queries will timeout. And did I tell you that there’s no index on LastActivityDate on aspnet_Users table?

Now don’t blame yourself for not putting either of these options on your every stored proc and every dynamically generated SQL from the very first day. ASP.NET developers made the same mistake. You won’t see either of these used in any of the stored procs used by ASP.NET Membership. For example, the following stored proc gets called whenever you access Profile object:
ALTER PROCEDURE [dbo].[aspnet_Profile_GetProperties]
@ApplicationName nvarchar(256),
@UserName nvarchar(256),
@CurrentTimeUtc datetime
AS
BEGIN

DECLARE @ApplicationId uniqueidentifier
SELECT @ApplicationId = NULL
SELECT @ApplicationId = ApplicationId FROM
dbo.aspnet_Applications WHERE LOWER(@ApplicationName) = LoweredApplicationName
IF (@ApplicationId IS NULL)
RETURN

DECLARE @UserId uniqueidentifier
DECLARE @LastActivityDate datetime
SELECT @UserId = NULL

SELECT @UserId = UserId, @LastActivityDate = LastActivityDate
FROM...

Visit my Blog for more details.

Linq to SQL solve Transaction deadlock and Query timeout problem using uncommitted reads

Saturday, March 07, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

When your database tables start accumulating thousands of rows and many users start working on the same table concurrently, SELECT queries on the tables start producing lock contentions and transaction deadlocks. This is a common problem in any high volume website. As soon as you start getting several concurrent users hitting your website that results in SELECT queries on some large table like aspnet_users table that are also being updated very frequently, you end up having one of these errors:
Transaction (Process ID ##) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Or,
Timeout Expired. The Timeout Period Elapsed Prior To Completion Of The Operation Or The Server Is Not Responding.
The solution to these problems are – use proper index on the table and use transaction isolation level Read Uncommitted or WITH (NOLOCK) in your SELECT queries. So, if you had a query like this: SELECT * FORM aspnet_users
where ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

You should end up having any of the above errors under high load. There are two ways to solve this:

SET TRANSACTION LEVEL READ UNCOMMITTED;
SELECT * FROM aspnet_Users
WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

Or use the WITH (NOLOCK):

SELECT * FROM aspnet_Users WITH (NOLOCK)
WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser'

The reason for the errors are that since aspnet_users is a high read and high write table, during read, the table is partially locked and during write, it is also locked. So, when the locks overlap on each other from several queries and especially when there’s a query that’s trying to read a large number of rows and thus locking large number of rows, some of the queries either timeout or produce deadlocks.

Linq to Sql does not produce queries with the WITH (NOLOCK) option nor does it use READ UNCOMMITTED. So, if you are using Linq to SQL queries, you are going to end up with any of these problems on production pretty soon when your site becomes highly popular.

For example, here’s a very simple query:

using (var db = new DropthingsDataContext())
{
var user = db.aspnet_Users.First();
var pages = user.Pages.ToList();
}

DropthingsDataContext is a DataContext built from Dropthings database.

When you attach SQL Profiler, you get this:

You see none of the queries have READ UNCOMMITTED or WITH (NOLOCK).

The fix is to do this:

using (var db = new DropthingsDataContext2())
{
db.Connection.Open();
db.ExecuteCommand("SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;");

var user = db.aspnet_Users.First();
var pages = user.Pages.ToList();
}

This will result in the following profiler output

As you see, both queries execute within the same connection and the isolation level is set before the queries execute. So, both queries enjoy the isolation level.

Now there’s a...

Visit my Blog for more details.

Strongly typed workflow input and output arguments

Saturday, December 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

When you run a Workflow using Workflow Foundation, you pass arguments to the workflow in a Dictionary form where the type of Dictionary is Dictionary<string, object>. This means you miss the strong typing features of .NET languages. You have to know what arguments the workflow expects by looking at the Workflow public properties. Moreover, there’s no way to make arguments required. You pass parameter, expect it to run, if it throws exception, you pass more arguments, hope it works now. Similarly, if you are running workflow synchronously using ManualWorkflowSchedulerService, you expect return arguments from the Workflow immediately, but there again, you have to rely on the Dictionary key and value pair. No strong typing there as well.
In order to solve this, so that you could pass Workflow arguments as strongly typed classes, you can establish a format that every Workflow has only two arguments named "Request” and “Response” and none other. Whatever needs to be passed to the Workflow and expected out of it, must be passed via Request and must be expected via Response properties. Now the type of these arguments can be workflow specific, it can be any class with one or more parameters. This way, you could write code like this:

The advantages of these strongly typed approach are: Compile time validation of input parameters passed to workflow. No risk of passing unexpected object in Dictionary’s object type value. Enforce required values by creating Request objects with non-default constructor. Establish a fixed contract for Workflow input and output via the strongly typed Request and Response classes or interfaces. Validate input arguments for the Workflow directly from the Request class, without going through the overhead of running a workflow.
If we follow this approach, we create workflows with only two DependencyProperty, one for Request and one for Response. Showing you an example from my open source project Dropthings, which uses Workflow for the entire Business Layer. Below you see the Workflow that executes when a new user visits Dropthings.com, creates a new user and setups all the pages and widgets for the user. It has only two Dependency property – Request and Response.

The Request parameters is of type IUserVisitWorkflowRequest. So, you can pass any class as Request argument that implements the interface.

Here I have used fancy inheritance to create Request object hierarchy. You don’t need to do that. Just remember, you can pass any class. You don’t even need to use interface for Request parameter. It can be a class directly. I use all these interfaces in order to facilitate Dependency Inversion.
Similarly, the Response object is also a class.

The Response returns quite some properties. So, it’s kinda handy to wrap them all in one property.
So, there you have it, strongly typed Workflow arguments. You can attach properties of the Request object to any activity directly form the...

Visit my Blog for more details.

99.99% available ASP.NET and SQL Server SaaS Production Architecture

Wednesday, December 10, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

You have a hot ASP.NET+SQL Server product, growing at thousand users per day and you have hit the limit of your own garage hosting capability. Now that you have enough VC money in your pocket, you are planning to go out and host on some real hosting facility, maybe a colocation or managed hosting. So, you are thinking, how to design a physical architecture that will ensure performance, scalability, security and availability of your product? How can you achieve four-nine (99.99%) availability? How do you securely let your development team connect to production servers? How do you choose the right hardware for web and database server? Should you use Storage Area Network (SAN) or just local disks on RAID? How do you securely connect your office computers to production environment?

Here I will answer all these queries. Let me first show you a diagram that I made for Pageflakes where we ensured we get four-nine availability. Since Pageflakes is a Level 3 SaaS, it’s absolutely important that we build a high performance, highly available product that can be used from anywhere in the world 24/7 and end-user gets quick access to their content with complete personalization and customization of content and can share it with others and to the world. So, you can take this production architecture as a very good candidate for Level 3 SaaS:

Here’s a CodeProject article that explains all the ideas:

99.99% available ASP.NET and SQL Server SaaS Production Architecture

Hope you like it. Appreciate your vote.

Visit my Blog for more details.

Linq to SQL: Delete an entity using Primary Key only

Thursday, October 30, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Linq to Sql does not come with a function like .Delete(ID) which allows you to delete an entity using it’s primary key. You have to first get the object that you want to delete and then call .DeleteOnSubmit(obj) to queue it for delete. Then you have to call DataContext.SubmitChanges() to play the delete queries on database. So, how to delete object without getting them from database and avoid database roundtrip?

You can call this function using DeleteByPK<Employee, int>(10, dataContext);

First type is the entity type and second one is the type of the primary key. If your object’s primary key is a Guid field, specify Guid instead of int.

How it works:

It figures out the table name and the primary key field name from the entity
Then it uses the table name and primary key field name to build a DELETE query

Figuring out the table name and primary key field name is a bit hard. There’s some reflection involved. The GetTableDef<TSource>() returns the table name and primary key field name for an entity.

Every Linq Entity class is decorated with a Table attribute that has the table name:

Then the primary key field is decorated with a Column attribute with IsPrimaryKey = true.

So, using reflection we can figure out the table name and the primary key property and the field name.

Here’s the code that does it:

Before you scream “Reflection is SLOW!!!!” the definition is cached. So, reflection is used only once per appDomain per entity. Subsequent call is just a dictionary lookup away, which is as fast as it can get.

You can also delete a collection of object without ever getting any one of them. The the following function to delete a whole bunch of objects:

The code is available here:

http://code.msdn.microsoft.com/DeleteEntitiesLinq

Visit my Blog for more details.

How to convince developers and management to use automated unit test for AJAX websites

Monday, October 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Everyone agrees that unit testing is a good thing, we should all write unit tests. We read articles and blogs to keep us up-to-date on what’s going on in the unit test world so that we can sound cool talking to peers at lunch. But when we really sit down and try to write unit tests ourselves – “Naaah, this is waste of time, let’s ask my QA to test it; that’s much more reliable and guaranteed way to test this. What’s the point testing these functions when there are so many other functions that we should unit test first?” Had such moment yourself or with someone else? Read on.
I had a conversation with our development lead Mike (using a highly generic name since my last post caused some trouble), who runs “the show” in our engineering team. As usual there was reservation in introducing unit test to regular development schedule. Mike also had valid points about lack of powerful tools for doing unit test on AJAX websites. He also had confusion on ‘what’ and ‘how’ to unit test our code so that we aren’t just testing database failures but real user actions that executes both business and rendering logics. So, the discussion has a lot of useful information, that will help you take the right decision when you want to sell unit test to your ASP.NET and/or AJAX development team and finally to higher management so that you can buy enough time for the effort.
Friday, Jan 2007 – hallway
Omar: Hey Mike, we need to start doing unit testing at least on our web services. We are wasting way too much time on manual QA. Since we are an AJAX shop, unit testing all our web services should give us pretty well coverage.
Mike: Sure, that sounds fun. I will do some feasibility check and see how can we chip this in into our next sprint.
Friday, Feb 2007 – washroom
Omar: Hey Mike, let’s start doing unit tests. I haven’t seen any tests last month. Can we start from this sprint?
Mike: Sure, we can surely start from this sprint. Let me find out which tool is the right one for us.
Friday, March 2007 – meeting room
Omar: Hey Mike, haven’t seen any unit tests in the solution so far. Let’s seriously start writing unit tests. Did you make any plan how you want to start unit testing the webservices?
Mike: Yeah, I did some digging around and found some tools. But most of them are for non-AJAX sites where you can programmatically hit a URL or programmatically do HTTP POST on a URL. You can also record button clicks and form posts from the browser. There’s Visual Studio’s Web Test, which does pretty good job recording regular ASP.NET site, but poor on AJAX sites. Moreover, you need to buy Team Suite edition to get that Web Test feature. Besides, recording tests and playing them back really does not help us because all those tests contain hard coded data. We can’t repeat a particular step many times with random data, at least not using any off-the-shelf tools. We need to test things carefully and systematically using random data set and sometimes use...

Visit my Blog for more details.

Solving common problems with Compiled Queries in Linq to Sql for high demand ASP.NET websites

Monday, October 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

If you are using Linq to SQL, instead of writing regular Linq Queries, you should be using Compiled Queries. if you are building an ASP.NET web application that’s going to get thousands of hits per hour, the execution overhead of Linq queries is going to consume too much CPU and make your site slow. There’s a runtime cost associated with each and every Linq Query you write. The queries are parsed and converted to a nice SQL Statement on *every* hit. It’s not done at compile time because there’s no way to figure out what you might be sending as the parameters in the queries during runtime. So, if you have common Linq to Sql statements like the following one throughout your growing web application, you are soon going to have scalability nightmares:

var query = from widget in dc.Widgets
where widget.ID == id && widget.PageID == pageId
select widget;

var widget = query.SingleOrDefault();

There’s a nice blog post by JD Conley that shows how evil Linq to Sql queries are:

You see how many times SqlVisitor.Visit is called to convert a Linq Query to its SQL representation? The runtime cost to convert a Linq query to its SQL Command representation is just way too high.

Rico Mariani has a very informative performance comparison of regular Linq queries vs Compiled Linq queries performance:

Compiled Query wins on every case.

So, now you know about the benefits of compiled queries. If you are building ASP.NET web application that is going to get high traffic and you have a lot of Linq to Sql queries throughout your project, you have to go for compiled queries. Compiled Queries are built for this specific scenario.

In this article, I will show you some steps to convert regular Linq to Sql queries to their Compiled representation and how to avoid the dreaded exception “Compiled queries across DataContexts with different LoadOptions not supported.”

Here are some step by step instruction on converting a Linq to Sql query to its compiled form:

First we need to find out all the external decision factors in a query. It mostly means parameters in the WHERE clause. Say, we are trying to get a user from aspnet_users table using Username and Application ID:

Here, we have two external decision factor – one is the Username and another is the Application ID. So, first think this way, if you were to wrap this query in a function that will just return this query as it is, what would you do? You would create a function that takes the DataContext (dc named here), then two parameters named userName and applicationID, right?

So, be it. We create one function that returns just this query:

Next step is to replace this function with a Func<> representation that returns the query. This is the hard part. If you haven’t dealt with Func<> and Lambda expression before, then I suggest you read this and this and then continue.

So, here’s the delegate...

Visit my Blog for more details.

Tips and tricks to rescue overdue projects

Monday, October 20, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

One of my friends, who runs his own offshore development shop, was having nightmare situation with one of his customers. He's way overdue on a release, the customer is screaming everyday, he's paying his team from his own pocket, customer is sending an ever increasing list of changes and so on. Here's how we discussed some ideas to get out of such a situation and make sure it does not repeat in future:
Kabir: Hey, can you help me? My customer is making us work for free for extra two months to fix bugs from our last delivery. We did what he said. But after he saw the output, he came up with hundred changes, which he somehow presents as bugs or missing features and make them look like they are all our fault and making us work for last two months for free. He is sending new changes every week. We have no idea when we will complete the iteration.
Omar: I see. Did you get a signed list of requirements from customer before you started the development?
Kabir: Of course, I did. He sent us a word document explaining what he wants and we sent him a task breakup with hour estimates and total duration of three months. Now after three months when we showed him the product, he said, it's no where close to what he had expected. Then he sent a gigantic list of things to change.
Omar: All of those are bugs?
Kabir: Of course not. Most of them are new features.
Omar: Then why don't you say those are new features? You have the original word document to prove. Just ask him to show where in the word document did he said X needs to be done?
Kabir: Well..., he's tricky. He somehow makes things look like it is obvious that X needs to be done and he's not going to accept a requirement as done until X is done. For example, he said there must be a complete login form in the homepage. So, we did a typical login form with user name, password and OK, Cancel button. Now he says where's the email verification thing? We said, you did not ask for it. He said, "this is obvious, every login form has a forgot password and email verification; I said *complete* login form, not half-baked login form". So, you see, we can't really argue to keep our image. Then, we did the login form exactly how he said. Now he says, where the client side validations of proper email address, username length, password confirmation? We said, you never asked for it! He says, "come on, every single website nowadays has AJAX enabled client side validation, do I have to tell you every single thing? Aren't you guys smart enough to figure this out? You are already doing this for the third time, can't you do it really well this time?"
Omar: OK, stop. I see what's your problem. Some customer will always try to make you work more for less money. They will try to squeeze out every bit of development they can for their bucks. So, you have to be extra careful on how much you commit to them and make sure they cannot chip in more...

Visit my Blog for more details.

An Agile Developer's workflow when SCRUM is used

Saturday, October 11, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

If you are planning to start SCRUM at your company, you might need to train developers and QA to get into the mindset of an Agile developer. SCRUM is only successful when the developers and QA get into the habit of following the principles of SCRUM by heart. So, sometimes you need to offer training or do trial sprints to give some room to your developers how to get used to the working fashion of SCRUM. Giving them a handy workflow diagram that shows how they should work helps soothe the steep learning curve required for non-super star developers. I made such a workflow while I was teaching SCRUM at my friend's company. The following diagram was printed and hung over the desk of each and every developer to help them grasp the culture of SCRUM quickly:

We use Flyspray for issue tracking, so you will see the mention of it frequently.

You will see the step to "Update Sprint backlog with remaining hours" is missing. This is done kinda verbally and scrum master (sometimes same person who is the product owner) keeps track of it.

Hope you find this useful.

Visit my Blog for more details.

ASP.NET website Continuous Integration+Deployment using CruiseControl.NET, Subversion, MSBuild and Robocopy

Monday, October 06, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

You can setup continuous integration and automated deployment for your web application using CruiseControl.NET, Subversion, MSBuild and Robocopy. I will show you how you can automatically build the entire solution, email build report to developers and QA, deploy latest code in IIS all using CruiseControl.NET every N minutes.

First get the following:

CruiseControl.NET
Subversion (install the command line tools and add the Subversion bin path to PATH environment variable)
Robocopy (Windows Vista/2008 has it built-in, here's the link for Windows 2003)
Install .NET Framework. You need it for MSBuild.

You will learn how I have configured Continuous Integration and Deployment for my open source AJAX Portal project www.Dropthings.com. The code is hosted at CodePlex. When some developer makes a commit, CruiseControl downloads the latest code, builds the entire solution, emails build report and then deploys the latest web site to IIS 6.0.

After installing CruiseControl.NET, go to Programs -> Cruise Control -> CruiseControl.NET Config.

Now keep copying and pasting the following XML blocks and make sure you understand each block and make necessary changes:

1: <cruisecontrol>
2: <project name="Dropthings" queue="DropthingsQueue" queuePriority="1">
3: 
7: <workingDirectory>d:\cc\dropthings\code\trunk\</workingDirectory>
8: 
9: <artifactDirectory>d:\cc\dropthings\artifact\</artifactDirectory>
10: <category>Dropthings</category>
11: 
12: <webURL>http://localhost/ccnet/</webURL>
13: <modificationDelaySeconds>60</modificationDelaySeconds>
14: <labeller type="defaultlabeller">
15: <prefix>0.1.</prefix>
16: <incrementOnFailure>true</incrementOnFailure>
17: <labelFormat>000</labelFormat>
18: </labeller>
19: <state type="state" directory="State" />

First change the working directory. It needs to be the path of the folder where you will have the solution downloaded. I generally create folder structure like this:

D:\CC - Root for all CC.NET enabled projects

\ProjectName - Root project folder

\Code - Code folder where code is downloaded from subversion

\Artifact - CC.NET generates a lot of stuff. All goes here.

Next comes the Subversion integration block:

1: <sourcecontrol type="svn">
2: ...

Visit my Blog for more details.

Using multiple broadband connections without using any special router or software

Sunday, October 05, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

I have two broadband connections. One cheap connection, which I mostly use for browsing and downloading. Another very expensive connection that I use for voice chat, remote desktop connection etc. Now, using these two connections at the same time required two computers before. But I figured out a way to use both connections at the same time using the same computer. Here's how:
Connect the cheap internet connection that is used mostly for non-critical purpose like downloading, browsing to a wireless router.
Connect the expensive connection that is used for network latency sensitive work like Voice Conference, Remote Desktop directly via LAN.
When you want to establish a critical connection like starting voice conf app (Skype) or remote desktop client, momentarily disconnect the wireless. This will make your LAN connection the only available internet. So, all the new connections will be established over the LAN. Now you can start Skype and initiate a voice conference or use Remote Desktop client and connect to a computer. The connection will be established over LAN.
Now turn on wireless. Wireless will now become the first preference for Windows to go to internet. So, now you can start Outlook, browser etc and they will be using the wireless internet connection. During this time, Skype and Terminal Client is still connected over the LAN connection. As they use persisted connection, they keep using the LAN connection and do not switch to the wireless.
This way you get to use two broadband connections simultaneously.

Here you see I have data transfer going on through two different connection. The bottom one is the LAN which is maintaining a continuous voice data stream. The upper one is the wireless connection that sometimes consumes bandwidth when I browse.

Using Sysinternal's TCPView, I can see some connection is going through LAN and some through Belkin router. The selected ones - the terminal client and the MSN Messenger is using LAN where the Internet Explorer and Outlook is working over Wireless connection.

Visit my Blog for more details.

Best practices for creating websites in IIS 6.0

Saturday, October 04, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Every time I create an IIS website, I do some steps, which I consider as best practice for creating any IIS website for better performance, maintainability, and scalability. Here' re the things I do:
Create a separate application pool for each web application
I always create separate app pool for each web app because I can select different schedule for app pool recycle. Some heavy traffic websites have long recycle schedule where low traffic websites have short recycle schedule to save memory. Moreover, I can choose different number of processes served by the app pool. Applications that are made for web garden mode can benefit from multiple process where applications that use in-process session, in memory cache needs to have single process serving the app pool. Hosting all my application under the DefaultAppPool does not give me the flexibility to control these per site.
The more app pool you create, the more ASP.NET threads you make available to your application. Each w3wp.exe has it's own thread pool. So, if some application is congesting particular w3wp.exe process, other applications can run happily on their separate w3wp.exe instance, running under separate app pool. Each app pool hosts its own w3wp.exe instance.
So, my rule of thumb: Always create new app pool for new web applications and name the app pool based on the site's domain name or some internal name that makes sense. For example, if you are creating a new website alzabir.com, name the app pool alzabir.com to easily identify it.
Another best practice: Disable the DefaultAppPool so that you don't mistakenly keep adding sites to DefaultAppPool.

First you create a new application pool. Then you create a new Website or Virtual Directory, go to Properties -> Home Directory tab -> Select the new app pool.

Customize Website properties for performance, scalability and maintainability
First you map the right host headers to your website. In order to do this, go to WebSite tab and click on "Advanced" button. Add mapping for both domain.com and www.domain.com. Most of the time, people forget to map the domain.com. Thus many visitors skip typing the www prefix and get no page served.

Next turn on some log entries:

These are very handy for analysis. If you want to measure your bandwidth consumption for specific sites, you need the Bytes Sent. If you want to measure the execution time of different pages and find out the slow running pages, you need Time Taken. If you want to measure unique and returning visitors, you need the Cookie. If you need to know who is sending you most traffic - search engines or some websites, you need the Referer. Once these entries are turned on, you can use variety of Log Analysis tools to do the analysis. For example, open source AWStats.
But if you are using Google Analytics or something else, you should have these turned off, especially the Cookie and Referer because they take quite some space on the log. If...

Visit my Blog for more details.

Create ASP.NET MVC Controllers under Namespace and specific URL

Saturday, October 04, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

When you have a lot of controllers, you need to organize them under Namespaces. Also, it is better to put controllers under subfolders in order to organize them properly and have some meaningful URL for them like /Admin/User where Admin is the subfolder and User is the controller. For example, you might have a lot of controllers that are APIs exposed by your web app, not regular pages. So, you might want to put them under /API/ folder. You also want to make sure no one can access those controllers from the root url. For example, no one must call /User/GetUserList instead they must call /API/User/GetUserList
ASP.NET MVC default routing and Controller Factory is very greedy, it ignores the subfolders inside the "Controllers" folder. There's a DefaultControllerFactory class in ASP.NET MVC which traverses all controllers under the "Controller" folder and creates a cache using just the class name as the key. So, it ignores any namespace or any subfolder where you have put the controller. So, I created a derivative of the default controller factory and created a new factory that checks if the requested controller belongs to any specific namespace and whether that controller must be inside a specific subfolder. You can map /Admin folder to respond to all controllers that are under then YourWebApp.Admin namespace. Similarly, you can map /API folder to respond to all controllers that are under the YourWebApp.API namespace. None of these controllers can be accessed outside the specified URL. Here's the factory that does it:

This controller checks the namespace of the controller being returned by ASP.NET MVC's default implementation and ensures the requested URL is the right url where controllers from the namespace can be accessed. So, this means if the controller matches MvcWebAPI.API.UserController, it ensures the URL being requested must be /MvcWebAPI/API/User. It will return null if the URL was something else like /MvcWebAPI/User or /MvcWebAPI/SomeotherFolder/User.
Here's how you use it form Global.asax:

You create a mapping for a Namespace and the subfolder it must belong to. Then you register the new Controller Factory as the default controller factory.
Now the second catch is, the default Route for the {controller}/{action}/{id} won't work for you. You need to create a specific router that starts with the subfolder name. For example, API/{controller}/{action}

Here' two things to notice: The use of PathStartWith routing constraint, which I will explain soon The last parameter which tells the route to include the API namespace for this route. Otherwise it can't find the controllers in the API namespace
So, the PathStartsWith routing constraint ensures this route gets hit only when the requested URL is under the /API/ folder. For any other URL, it returns false and thus the routing handler skips this route.

It just does a comparison on the AbsolutePath of the current request URL to...

Visit my Blog for more details.

Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 News Feed

Updated Wiki: Home

oazabir — Sat, 13 Jun 2009 17:22:33 GMT

Project Description
Ajax Web Portal built on Linq, Workflow Foundation and ASP.NET AJAX. Code is in Visual Studio 2008 using .NET 3.0 and .NET 3.5.

Production site

http://dropthings.omaralzabir.com
For latest news and updates, visit my blog
http://msmvps.com/blogs/omar/

Source Code

http://code.google.com/p/dropthings/

Book about this project

This book explains how this project has been built step by step. It also explains many advance AJAX concepts, development and production challenges for building and maintaining a high volume production website.

Get it from Amazon:
http://www.amazon.com/Building-Web-2-0-Portal-ASP-NET/dp/0596510500

Technologies

ASP.NET 2.0
jQuery
ASP.NET AJAX (.NET 3.5)
Silverlight
Linq to Sql
Linq to Xml
Workflow Foundation (.NET 3.0)
Visual Studio 2008 and SQL Server 2005

Features

Configure widgets for specific user roles.
Configure different default page setup for different roles - Managers get some widgets, Employees get different widgets and so on.
Ability to define page setup for anonymous users and different page setup for logged in users.
Multiple Tabs.
Customizable Widget Gallery.
Different column setup.
Build your own widgets using Silverlight, ASP.NET or plain Javascripts.
Host widgets on any page of your website
Easy to integrate Dropthings to your own website e.g. http://myoffice.bt.com is built on top of Dropthings.
Business Layer is entirely Workflow Foundation driven - easy to plug-in your own business logic as Activities.

Download the latest source code from Google Code site http://code.google.com/p/dropthings/
Follow the Readme.txt

My Blog

---

Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 News Feed

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

Thursday, April 09, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Memory Leak with delegates and workflow foundation

Saturday, March 14, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Optimize ASP.NET Membership Stored Procedures for greater speed and scalability

Friday, March 13, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Linq to SQL solve Transaction deadlock and Query timeout problem using uncommitted reads

Saturday, March 07, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Strongly typed workflow input and output arguments

Saturday, December 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

99.99% available ASP.NET and SQL Server SaaS Production Architecture

Wednesday, December 10, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Linq to SQL: Delete an entity using Primary Key only

Thursday, October 30, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

How to convince developers and management to use automated unit test for AJAX websites

Monday, October 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Solving common problems with Compiled Queries in Linq to Sql for high demand ASP.NET websites

Monday, October 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Tips and tricks to rescue overdue projects

Tuesday, October 21, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

An Agile Developer's workflow when SCRUM is used

Saturday, October 11, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

ASP.NET website Continuous Integration+Deployment using CruiseControl.NET, Subversion, MSBuild and Robocopy

Monday, October 06, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Using multiple broadband connections without using any special router or software

Sunday, October 05, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Best practices for creating websites in IIS 6.0

Saturday, October 04, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Create ASP.NET MVC Controllers under Namespace and specific URL

Saturday, October 04, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Create REST API using ASP.NET MVC that speaks both Json and plain Xml

Friday, October 03, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

ASP.NET MVC Controllers can directly return objects and collections, without rendering a view, which makes it quite appealing for creating REST like API. The nice extensionless Url provided by MVC makes it handy to build REST services, which means you can create APIs with smart Url like "something.com/API/User/GetUserList"
There are some challenges to solve in order to expose REST API: Based on who is calling your API, you need to be able to speak both Json and plain old Xml (POX). If the call comes from an AJAX front-end, you need to return objects serialized as Json. If it's coming from some other client, say a PHP website, you need to return plain Xml. Similarly you need to be able to understand REST, Json and plain Xml calls. Someone can hit you using REST url, someone can post a Json payload or someone can post Xml payload.
I have created an ObjectResult class which takes an object and generates Xml or Json output automatically looking at the Content-Type header of HttpRequest. AJAX calls send Content-Type=application/json. So, it generates Json as response in that case, but when Content-Type is something else, it does simple Xml Serialzation.

Here's the ObjectResult that you can use from Controllers to return objects and it takes care of proper serialization method. Above shows the Json serialization, which is quite simple. XmlSerialization is a bit complex though:
Things to note here: You have to force UTF8 encoding. Otherwise it produces UTF16 output. XML Declaration is skipped because that's not quite necessary. Wastes bandwidth. If you need it, turn it on. I have turned on indenting for better readability. You can turn it off to save bandwidth.
Some of you might be boiling inside looking at my obscure coding style. I love this style! I am spoiled by jQuery. I wish there was a cQuery. I actually started writing one, but it never saw day light just like my hundred other open source attempts.
Now back to Object Serialization, we got the serialization done. Now you can return objects from Controller easily:

You can use the test web project to call these methods and see the result:

So far you have seen simple object and list serialization. A best practice is to return a common result object that has some status, message and then the real payload. It's handy when you only need to return some error but no object or list. I use a common Result object that has three properties - ErrorCode (0 by default means success), Message (a string data type) and Data which is the real object.

When you want to return only a result with error message, you can do this:

This produces a result like this:

No payload here. So, the return format is always consistent. Those who are consuming service can write a common Xml or Json parsing code to consume both success and failure response. Those who are building API for their website, I humbly request you to return consistent response for both success and...

Visit my Blog for more details.

HTTP handler to combine multiple files, cache and deliver compressed output for faster page load

Friday, August 29, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

It's a good practice to use many small Javascript and CSS files instead of one large Javascript/CSS file for better code maintainability, but bad in terms of website performance. Although you should write your Javascript code in small files and break large CSS files into small chunks but when browser requests those javascript and css files, it makes one Http request per file. Every Http Request results in a network roundtrip form your browser to the server and the delay in reaching the server and coming back to the browser is called latency. So, if you have four javascripts and three css files loaded by a page, you are wasting time in seven network roundtrips. Within USA, latency is average 70ms. So, you waste 7x70 = 490ms, about half a second of delay. Outside USA, average latency is around 200ms. So, that means 1400ms of waiting. Browser cannot show the page properly until Css and Javascripts are fully loaded. So, the more latency you have, the slower page loads.
Here's a graph that shows how each request latency adds up and introduces significant delay in page loading:

You can reduce the wait time by using a CDN. Read my previous blog post about using CDN. However, a better solution is to deliver multiple files over one request using an HttpHandler that combines several files and delivers as one output. So, instead of putting many <script> or <link> tag, you just put one <script> and one <link> tag, and point them to the HttpHandler. You tell the handler which files to combine and it delivers those files in one response. This saves browser from making many requests and eliminates the latency.

Here you can see how much improvement you get if you can combine multiple javascripts and css into one.
In a typical web page, you will see many javascripts referenced:<script type="text/javascript" src="/Content/JScript/jquery.js">
</script>
<script type="text/javascript" src="/Content/JScript/jDate.js">
</script>
<script type="text/javascript" src="/Content/JScript/jQuery.Core.js">
</script>
<script type="text/javascript" src="/Content/JScript/jQuery.Delegate.js">
</script>
<script type="text/javascript" src="/Content/JScript/jQuery.Validation.js">
</script>

Instead of these individual <script> tags, you can use only one <script> tag to serve the whole set of scripts using an Http Handler:<script type="text/javascript"
src="HttpCombiner.ashx?s=jQueryScripts&t=text/javascript&v=1" >
</script>

The Http Handler reads the file names defined in a configuration and combines all those files and delivers as one response. It delivers the response as gzip compressed to save bandwidth. Moreover, it generates proper cache header to cache the response in browser cache, so that, browser does not request it...

Visit my Blog for more details.

Loading static content in ASP.NET pages from different domain for faster parallel download

Saturday, August 02, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Generally we put static content (images, css, js) of our website inside the same web project. Thus they get downloaded from the same domain like www.dropthings.com. There are three problems in this approach: They occupy connections on the same domain www.dropthings.com and thus other important calls like Web service call do not get a chance to happen earlier as browser can only make two simultaneous connections per domain. If you are using ASP.NET Forms Authentication, then you have that gigantic Forms Authentication cookie being sent with every single request on www.dropthings.com. This cookie gets sent for all images, CSS and JS files, which has no use for the cookie. Thus it wastes upload bandwidth and makes every request slower. Upload bandwidth is very limited for users compared to download bandwidth. Generally users with 1Mbps download speed has around 128kbps upload speed. So, adding another 100 bytes on the request for the unnecessary cookie results in delay in sending the request and thus increases your site load time and the site feels slow to respond. It creates enormous IIS Logs as it records the cookies for each static content request. Moreover, if you are using Google Analytics to track hits to your site, it issues four big cookies that gets sent for each and every image, css and js files resulting in slower requests and even larger IIS log entries.
Let's see the first problem, browser's two connection limit. See what happens when content download using two HTTP requests in parallel:

This figure shows only two files are downloaded in parallel. All the hits are going to the same domain e.g. www.dropthings.com. As you see, only two call can execute at the same time. Moreover, due to browser's way of handling script tags, once a script is being downloaded, browser does not download anything else until the script has downloaded and executed.
Now, if we can download the images from different domain, which allows browser to open another two simultaneous connections, then the page loads a lot faster:

You see, the total page downloads 40% faster. Here only the images are downloaded from a different domain e.g. "s.dropthings.com", thus the calls for the script, CSS and webservices still go to main domain e.g. www.dropthings.com
The second problem for loading static content from same domain is the gigantic forms authentication cookie or any other cookie being registered on the main domain e.g. www subdomain. Here's how Pageflake's website's request looks like with the forms authentication cookie and Google Analytics cookies:

You see a lot of data being sent on the request header which has no use for any static content. Thus it wastes bandwidth, makes request reach server slower and produces large IIS logs.
You can solve this problem by loading static contents from different domain as we have done it at Pageflakes by loading static contents from a different domain e.g. flakepage.com. As the...

Visit my Blog for more details.

Open Source ASP.NET 3.5 AJAX Portal - new and improved

Monday, July 14, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Last week I released a new version of Dropthings, my open source AJAX portal, that shows many fancy Web 2.0 features and showcases extensive use of ASP.NET 3.5, Workflow Foundation, C# 3.0 new language features, custom ASP.NET AJAX extenders, many performance and scalability techniques. I have written a book on these topics as well.

The new version implements the following performance and scalability improvement techniques:

10 ASP.NET Performance and Scalability Secrets
Fast ASP.NET page rendering by deferred script loading
Load large amount of Javascripts in batch and thus load AJAX sites a lot faster
Fast Streaming AJAX proxy that solves double downloading problems and continuously streams content from external domain
Making best use of cache for high performance website
On-demand UI loading on AJAX websites

Here's how the new version looks:

Hope you like the new design and the performance and scalability techniques that can significantly boost your ASP.NET website's quality. I highly recommend these techniques for ASP.NET websites. These are easy to implement and makes a world of difference in speed and smoothness for ASP.NET websites.

I am thinking about making an ASP.NET MVC version of this portal using jQuery. Do you think it will be a hot area to explore?

Share this post :

Visit my Blog for more details.

Deploy ASP.NET MVC on IIS 6, solve 404, compression and performance problems

Monday, June 30, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

There are several problems with ASP.NET MVC application when deployed on IIS 6.0: Extensionless URLs give 404 unless some URL Rewrite module is used or wildcard mapping is enabled IIS 6.0 built-in compression does not work for dynamic requests. As a result, ASP.NET pages are served uncompressed resulting in poor site load speed. Mapping wildcard extension to ASP.NET introduces the following problems: Slow performance as all static files get handled by ASP.NET and ASP.NET reads the file from file system on every call Expires headers doesn't work for static content as IIS does not serve them anymore. Learn about benefits of expires header from here. ASP.NET serves a fixed expires header that makes content expire in a day. Cache-Control header does not produce max-age properly and thus caching does not work as expected. Learn about caching best practices from here. After deploying on a domain as the root site, the homepage produces HTTP 404. Problem 1: Visiting your website's homepage gives 404 when hosted on a domain
You have done the wildcard mapping, mapped .mvc extention to ASP.NET ISAPI handler, written the route mapping for Default.aspx or default.aspx (lowercase), but still when you visit your homepage after deployment, you get:

You will find people banging their heads on the wall here: http://forums.asp.net/t/1237051.aspx http://forums.asp.net/t/1253599.aspx http://forums.asp.net/p/1239943/2294813.aspx
Solution is to capture hits going to "/" and then rewrite it to Default.aspx:

You can apply this approach to any URL that ASP.NET MVC is not handling for you and it should handle. Just see the URL reported on the 404 error page and then rewrite it to a proper URL. Problem 2: IIS 6 compression is no longer working after wildcard mapping
When you enable wildcard mapping, IIS 6 compression no longer works for extensionless URL because IIS 6 does not see any extension which is defined in IIS Metabase. You can learn about IIS 6 compression feature and how to configure it properly from my earlier post.
Solution is to use an HttpModule to do the compression for dynamic requests. Problem 3: ASP.NET ISAPI does not cache Static Files
When ASP.NET's DefaultHttpHandler serves static files, it does not cache the files in-memory or in ASP.NET cache. As a result, every hit to static file results in a File read. Below is the decompiled code in DefaultHttpHandler when it handles a static file. As you see here, it makes a file read on every hit and it only set the expiration to one day in future. Moreover, it generates ETag for each file based on file's modified date. For best caching efficiency, we need to get rid of that ETag, produce an expiry date on far future (like 30 days), and produce Cache-Control header which offers better control over caching.

So, we need to write a custom static file handler that will cache small files like images, Javascripts, CSS, HTML and so on in ASP.NET cache and serve the...

Visit my Blog for more details.

Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 News Feed

Updated Wiki: Home

oazabir — Thu, 09 Apr 2009 14:51:08 GMT

Project Description
Ajax Web Portal built on Linq, Workflow Foundation and ASP.NET AJAX. Code is in Visual Studio 2008 using .NET 3.0 and .NET 3.5.
Production site
http://dropthings.omaralzabir.com
For latest news and updates, visit my blog
http://msmvps.com/blogs/omar/
Source Code
http://code.google.com/p/dropthings/

Book about this project

This book explains how this project has been built step by step. It also explains many advance AJAX concepts, development and production challenges for building and maintaining a high volume production website.

Get it from Amazon:
http://www.amazon.com/Building-Web-2-0-Portal-ASP-NET/dp/0596510500

Technologies

ASP.NET 2.0
jQuery
ASP.NET AJAX (.NET 3.5)
Silverlight
Linq to Sql
Linq to Xml
Workflow Foundation (.NET 3.0)
Visual Studio 2008 and SQL Server 2005

Features

Configure widgets for specific user roles.
Configure different default page setup for different roles - Managers get some widgets, Employees get different widgets and so on.
Ability to define page setup for anonymous users and different page setup for logged in users.
Multiple Tabs.
Customizable Widget Gallery.
Different column setup.
Build your own widgets using Silverlight, ASP.NET or plain Javascripts.
Host widgets on any page of your website
Easy to integrate Dropthings to your own website e.g. http://myoffice.bt.com is built on top of Dropthings.
Business Layer is entirely Workflow Foundation driven - easy to plug-in your own business logic as Activities.

Download the latest source code from Google Code site http://code.google.com/p/dropthings/
Follow the Readme.txt

My Blog

---

Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 News Feed

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

Wednesday, April 08, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Dropthings – my open source Web 2.0 Ajax Portal has gone through a technology overhauling. Previously it was built using ASP.NET AJAX, a little bit of Workflow Foundation and Linq to SQL. Now Dropthings boasts full jQuery front-end combined with ASP.NET AJAX UpdatePanel, Silverlight widget, full Workflow Foundation implementation on the business layer, 100% Linq to SQL Compiled Queries on the data access layer, Dependency Injection and Inversion of Control (IoC) using Microsoft Enterprise Library 4.1 and Unity. It also has a ASP.NET AJAX Web Test framework that makes it real easy to write Web Tests that simulates real user actions on AJAX web pages. This article will walk you through the challenges in getting these new technologies to work in an ASP.NET website and how performance, scalability, extensibility and maintainability has significantly improved by the new technologies. Dropthings has been licensed for commercial use by prominent companies including BT Business, Intel, Microsoft IS, Denmark Government portal for Citizens; Startups like Limead and many more. So, this is serious stuff! There’s a very cool open source implementation of Dropthings framework available at National University of Singapore portal. Visit: http://dropthings.omaralzabir.com I have published a new article on this on CodeProject: http://www.codeproject.com/KB/ajax/Web20Portal.aspx Get the source code Latest source code is hosted at Google code: http://code.google.com/p/dropthings There’s a CodePlex site for documentation and issue tracking: http://www.codeplex.com/dropthings You will need Visual Studio 2008 Team Suite with Service Pack 1 and Silverlight 2 SDK in order to run all the projects. If you have only Visual Studio 2008 Professional, then you will have to remove the Dropthings.Test project. New features introduced Dropthings new release has the following features: Template users – you can define a user who’s pages and widgets are used as a template for new users. Whatever you put in that template user’s pages, it will be copied for every new user. Thus this is an easier way to define the default pages and widgets for new users. Similarly you can do the same for a registered user. The template users can be defined in the web.config. Widget-to-Widget communication – Widgets can send message to each other. Widgets can subscribe to an Event Broker and exchange messages using a Pub-Sub pattern. WidgetZone – you can create any number of zones in any shape on the page. You can have widgets laid in horizontal layout, you can have zones on different places on the page and so on. With this zone model, you are no longer limited to the Page-Column model where you could only have N vertical columns. Role based widgets – now widgets are mapped to roles so that you can allow different users to see different widget list using ManageWidgetPersmission.aspx. Role based page setup – you can define page setup for different roles. For...

Visit my Blog for more details.

Memory Leak with delegates and workflow foundation

Saturday, March 14, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Recently after Load Testing my open source project Dropthings, I encountered a lot of memory leak. I found lots of Workflow Instances and Linq Entities were left in memory and never collected. After profiling the web application using .NET Memory Profiler, it showed the real picture: It shows you that instances of the several types are being created but not being removed. You see the “New” column has positive value, but the “Remove” column has 0. That means new instances are being created, but not removed. Basically the way you do Memory Profiling is, you take two snapshots. Say you take one snapshot when you first visit your website. Then you do some action on the website that results in allocation of objects. Then you take another snapshot. When you compare both snapshots, you can see how many instances of classes were created between these two snapshots and how many were removed. If they are not equal, then you have leak. Generally in web application many objects are created on every page hit and the end of the request, all those objects are supposed to be released. If they are not released, then we have a problem. But that’s the scenario for desktop applications because in a desktop application, objects can remain in memory until app is closed. But you should know best from the code which objects were supposed to go out of scope and get released. For beginners, leak means objects are being allocated but not being freed because someone is holding reference to the objects. When objects leak, they remain in memory forever, until the process (or app domain) is closed. So, if you have a leaky website, your website is continuously taking up memory until it runs out of memory on the web server and thus crash. So, memory leak is a bad – it prevents you from running your product for long duration and requires frequent restart of app pool. So, the above screenshot shows Workflow and Linq related classes are not being removed, and thus leaking. This means somewhere workflow instances are not being released and thus all workflow related objects are remaining. You can see the number is same 48 for all workflow related objects. This is a good indication that, almost every instance of workflow is leaked because there were total 48 workflows created and ran. Moreover it indicates we have a leak from a top Workflow instance level, not in some specific Activity or somewhere deep in the code. As the workflows use Linq stuff, they held reference to the Linq stuffs and thus the Linq stuffs leaked as well. Sometimes you might be looking for why A is leaking. But you actually end up finding that since B was holding reference to A and B was leaking and thus A was leaking as well. This is sometimes tricky to figure out and you spend a lot of time looking at the wrong direction. Now let me show you the buggy code: ManualWorkflowSchedulerService manualScheduler = workflowRuntime.GetService<ManualWorkflowSchedulerService>(); WorkflowInstance...

Visit my Blog for more details.

Optimize ASP.NET Membership Stored Procedures for greater speed and scalability

Friday, March 13, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Last year at Pageflakes, when we were getting millions of hits per day, we were having query timeout due to lock timeout and Transaction Deadlock errors. These locks were produced from aspnet_Users and aspnet_Membership tables. Since both of these tables are very high read (almost every request causes a read on these tables) and high write (every anonymous visit creates a row on aspnet_Users), there were just way too many locks created on these tables per second. SQL Counters showed thousands of locks per second being created. Moreover, we had queries that would select thousands of rows from these tables frequently and thus produced more locks for longer period, forcing other queries to timeout and thus throw errors on the website. If you have read my last blog post, you know why such locks happen. Basically every table when it grows up to hold millions of records and becomes popular goes through this trouble. It’s just a part of scalability problem that is common to database. But we rarely take prevention about it in our early design. The solution is simple, you should either have WITH (NOLOCK) or SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED before SELECT queries. Either of this will do. They tell SQL Server not to hold any lock on the table while it is reading the table. If some row is locked while the read is happening, it will just ignore that row. When you are reading a table thousand times per second, without these options, you are issuing lock on many places around the table thousand times per second. It not only makes read from table slower, but also so many lock prevents insert, update, delete from happening timely and thus queries timeout. If you have queries like “show the currently online users from last one hour based on LastActivityDate field”, that is going to issue such a wide lock that even other harmless select queries will timeout. And did I tell you that there’s no index on LastActivityDate on aspnet_Users table? Now don’t blame yourself for not putting either of these options on your every stored proc and every dynamically generated SQL from the very first day. ASP.NET developers made the same mistake. You won’t see either of these used in any of the stored procs used by ASP.NET Membership. For example, the following stored proc gets called whenever you access Profile object: ALTER PROCEDURE [dbo].[aspnet_Profile_GetProperties] @ApplicationName nvarchar(256), @UserName nvarchar(256), @CurrentTimeUtc datetime AS BEGIN DECLARE @ApplicationId uniqueidentifier SELECT @ApplicationId = NULL SELECT @ApplicationId = ApplicationId FROM dbo.aspnet_Applications WHERE LOWER(@ApplicationName) = LoweredApplicationName IF (@ApplicationId IS NULL) RETURN DECLARE @UserId uniqueidentifier DECLARE @LastActivityDate datetime SELECT @UserId = NULL SELECT @UserId = UserId, @LastActivityDate = LastActivityDate FROM...

Visit my Blog for more details.

Linq to SQL solve Transaction deadlock and Query timeout problem using uncommitted reads

Saturday, March 07, 2009 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

When your database tables start accumulating thousands of rows and many users start working on the same table concurrently, SELECT queries on the tables start producing lock contentions and transaction deadlocks. This is a common problem in any high volume website. As soon as you start getting several concurrent users hitting your website that results in SELECT queries on some large table like aspnet_users table that are also being updated very frequently, you end up having one of these errors: Transaction (Process ID ##) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Or, Timeout Expired. The Timeout Period Elapsed Prior To Completion Of The Operation Or The Server Is Not Responding. The solution to these problems are – use proper index on the table and use transaction isolation level Read Uncommitted or WITH (NOLOCK) in your SELECT queries. So, if you had a query like this: SELECT * FORM aspnet_users where ApplicationID =’xxx’ AND LoweredUserName = 'someuser' You should end up having any of the above errors under high load. There are two ways to solve this: SET TRANSACTION LEVEL READ UNCOMMITTED; SELECT * FROM aspnet_Users WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser' Or use the WITH (NOLOCK): SELECT * FROM aspnet_Users WITH (NOLOCK) WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser' The reason for the errors are that since aspnet_users is a high read and high write table, during read, the table is partially locked and during write, it is also locked. So, when the locks overlap on each other from several queries and especially when there’s a query that’s trying to read a large number of rows and thus locking large number of rows, some of the queries either timeout or produce deadlocks. Linq to Sql does not produce queries with the WITH (NOLOCK) option nor does it use READ UNCOMMITTED. So, if you are using Linq to SQL queries, you are going to end up with any of these problems on production pretty soon when your site becomes highly popular. For example, here’s a very simple query: using (var db = new DropthingsDataContext()) { var user = db.aspnet_Users.First(); var pages = user.Pages.ToList(); } DropthingsDataContext is a DataContext built from Dropthings database. When you attach SQL Profiler, you get this: You see none of the queries have READ UNCOMMITTED or WITH (NOLOCK). The fix is to do this: using (var db = new DropthingsDataContext2()) { db.Connection.Open(); db.ExecuteCommand("SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;"); var user = db.aspnet_Users.First(); var pages = user.Pages.ToList(); } This will result in the following profiler output As you see, both queries execute within the same connection and the isolation level is set before the queries execute. So, both queries enjoy the isolation level. Now there’s a...

Visit my Blog for more details.

Strongly typed workflow input and output arguments

Saturday, December 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

When you run a Workflow using Workflow Foundation, you pass arguments to the workflow in a Dictionary form where the type of Dictionary is Dictionary<string, object>. This means you miss the strong typing features of .NET languages. You have to know what arguments the workflow expects by looking at the Workflow public properties. Moreover, there’s no way to make arguments required. You pass parameter, expect it to run, if it throws exception, you pass more arguments, hope it works now. Similarly, if you are running workflow synchronously using ManualWorkflowSchedulerService, you expect return arguments from the Workflow immediately, but there again, you have to rely on the Dictionary key and value pair. No strong typing there as well. In order to solve this, so that you could pass Workflow arguments as strongly typed classes, you can establish a format that every Workflow has only two arguments named "Request” and “Response” and none other. Whatever needs to be passed to the Workflow and expected out of it, must be passed via Request and must be expected via Response properties. Now the type of these arguments can be workflow specific, it can be any class with one or more parameters. This way, you could write code like this: The advantages of these strongly typed approach are: Compile time validation of input parameters passed to workflow. No risk of passing unexpected object in Dictionary’s object type value. Enforce required values by creating Request objects with non-default constructor. Establish a fixed contract for Workflow input and output via the strongly typed Request and Response classes or interfaces. Validate input arguments for the Workflow directly from the Request class, without going through the overhead of running a workflow. If we follow this approach, we create workflows with only two DependencyProperty, one for Request and one for Response. Showing you an example from my open source project Dropthings, which uses Workflow for the entire Business Layer. Below you see the Workflow that executes when a new user visits Dropthings.com, creates a new user and setups all the pages and widgets for the user. It has only two Dependency property – Request and Response. The Request parameters is of type IUserVisitWorkflowRequest. So, you can pass any class as Request argument that implements the interface. Here I have used fancy inheritance to create Request object hierarchy. You don’t need to do that. Just remember, you can pass any class. You don’t even need to use interface for Request parameter. It can be a class directly. I use all these interfaces in order to facilitate Dependency Inversion. Similarly, the Response object is also a class. The Response returns quite some properties. So, it’s kinda handy to wrap them all in one property. So, there you have it, strongly typed Workflow arguments. You can attach properties of the Request object to any activity directly form the...

Visit my Blog for more details.

99.99% available ASP.NET and SQL Server SaaS Production Architecture

Wednesday, December 10, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

You have a hot ASP.NET+SQL Server product, growing at thousand users per day and you have hit the limit of your own garage hosting capability. Now that you have enough VC money in your pocket, you are planning to go out and host on some real hosting facility, maybe a colocation or managed hosting. So, you are thinking, how to design a physical architecture that will ensure performance, scalability, security and availability of your product? How can you achieve four-nine (99.99%) availability? How do you securely let your development team connect to production servers? How do you choose the right hardware for web and database server? Should you use Storage Area Network (SAN) or just local disks on RAID? How do you securely connect your office computers to production environment? Here I will answer all these queries. Let me first show you a diagram that I made for Pageflakes where we ensured we get four-nine availability. Since Pageflakes is a Level 3 SaaS, it’s absolutely important that we build a high performance, highly available product that can be used from anywhere in the world 24/7 and end-user gets quick access to their content with complete personalization and customization of content and can share it with others and to the world. So, you can take this production architecture as a very good candidate for Level 3 SaaS: Here’s a CodeProject article that explains all the ideas: 99.99% available ASP.NET and SQL Server SaaS Production Architecture Hope you like it. Appreciate your vote.

Visit my Blog for more details.

Linq to SQL: Delete an entity using Primary Key only

Thursday, October 30, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Linq to Sql does not come with a function like .Delete(ID) which allows you to delete an entity using it’s primary key. You have to first get the object that you want to delete and then call .DeleteOnSubmit(obj) to queue it for delete. Then you have to call DataContext.SubmitChanges() to play the delete queries on database. So, how to delete object without getting them from database and avoid database roundtrip? You can call this function using DeleteByPK<Employee, int>(10, dataContext); First type is the entity type and second one is the type of the primary key. If your object’s primary key is a Guid field, specify Guid instead of int. How it works: It figures out the table name and the primary key field name from the entity Then it uses the table name and primary key field name to build a DELETE query Figuring out the table name and primary key field name is a bit hard. There’s some reflection involved. The GetTableDef<TSource>() returns the table name and primary key field name for an entity. Every Linq Entity class is decorated with a Table attribute that has the table name: Then the primary key field is decorated with a Column attribute with IsPrimaryKey = true. So, using reflection we can figure out the table name and the primary key property and the field name. Here’s the code that does it: Before you scream “Reflection is SLOW!!!!” the definition is cached. So, reflection is used only once per appDomain per entity. Subsequent call is just a dictionary lookup away, which is as fast as it can get. You can also delete a collection of object without ever getting any one of them. The the following function to delete a whole bunch of objects: The code is available here: http://code.msdn.microsoft.com/DeleteEntitiesLinq

Visit my Blog for more details.

How to convince developers and management to use automated unit test for AJAX websites

Monday, October 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Everyone agrees that unit testing is a good thing, we should all write unit tests. We read articles and blogs to keep us up-to-date on what’s going on in the unit test world so that we can sound cool talking to peers at lunch. But when we really sit down and try to write unit tests ourselves – “Naaah, this is waste of time, let’s ask my QA to test it; that’s much more reliable and guaranteed way to test this. What’s the point testing these functions when there are so many other functions that we should unit test first?” Had such moment yourself or with someone else? Read on. I had a conversation with our development lead Mike (using a highly generic name since my last post caused some trouble), who runs “the show” in our engineering team. As usual there was reservation in introducing unit test to regular development schedule. Mike also had valid points about lack of powerful tools for doing unit test on AJAX websites. He also had confusion on ‘what’ and ‘how’ to unit test our code so that we aren’t just testing database failures but real user actions that executes both business and rendering logics. So, the discussion has a lot of useful information, that will help you take the right decision when you want to sell unit test to your ASP.NET and/or AJAX development team and finally to higher management so that you can buy enough time for the effort. Friday, Jan 2007 – hallway Omar: Hey Mike, we need to start doing unit testing at least on our web services. We are wasting way too much time on manual QA. Since we are an AJAX shop, unit testing all our web services should give us pretty well coverage. Mike: Sure, that sounds fun. I will do some feasibility check and see how can we chip this in into our next sprint. Friday, Feb 2007 – washroom Omar: Hey Mike, let’s start doing unit tests. I haven’t seen any tests last month. Can we start from this sprint? Mike: Sure, we can surely start from this sprint. Let me find out which tool is the right one for us. Friday, March 2007 – meeting room Omar: Hey Mike, haven’t seen any unit tests in the solution so far. Let’s seriously start writing unit tests. Did you make any plan how you want to start unit testing the webservices? Mike: Yeah, I did some digging around and found some tools. But most of them are for non-AJAX sites where you can programmatically hit a URL or programmatically do HTTP POST on a URL. You can also record button clicks and form posts from the browser. There’s Visual Studio’s Web Test, which does pretty good job recording regular ASP.NET site, but poor on AJAX sites. Moreover, you need to buy Team Suite edition to get that Web Test feature. Besides, recording tests and playing them back really does not help us because all those tests contain hard coded data. We can’t repeat a particular step many times with random data, at least not using any off-the-shelf tools. We need to test things carefully and systematically using random data set and sometimes use...

Visit my Blog for more details.

Solving common problems with Compiled Queries in Linq to Sql for high demand ASP.NET websites

Monday, October 27, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

If you are using Linq to SQL, instead of writing regular Linq Queries, you should be using Compiled Queries. if you are building an ASP.NET web application that’s going to get thousands of hits per hour, the execution overhead of Linq queries is going to consume too much CPU and make your site slow. There’s a runtime cost associated with each and every Linq Query you write. The queries are parsed and converted to a nice SQL Statement on *every* hit. It’s not done at compile time because there’s no way to figure out what you might be sending as the parameters in the queries during runtime. So, if you have common Linq to Sql statements like the following one throughout your growing web application, you are soon going to have scalability nightmares: var query = from widget in dc.Widgets where widget.ID == id && widget.PageID == pageId select widget; var widget = query.SingleOrDefault(); There’s a nice blog post by JD Conley that shows how evil Linq to Sql queries are: You see how many times SqlVisitor.Visit is called to convert a Linq Query to its SQL representation? The runtime cost to convert a Linq query to its SQL Command representation is just way too high. Rico Mariani has a very informative performance comparison of regular Linq queries vs Compiled Linq queries performance: Compiled Query wins on every case. So, now you know about the benefits of compiled queries. If you are building ASP.NET web application that is going to get high traffic and you have a lot of Linq to Sql queries throughout your project, you have to go for compiled queries. Compiled Queries are built for this specific scenario. In this article, I will show you some steps to convert regular Linq to Sql queries to their Compiled representation and how to avoid the dreaded exception “Compiled queries across DataContexts with different LoadOptions not supported.” Here are some step by step instruction on converting a Linq to Sql query to its compiled form: First we need to find out all the external decision factors in a query. It mostly means parameters in the WHERE clause. Say, we are trying to get a user from aspnet_users table using Username and Application ID: Here, we have two external decision factor – one is the Username and another is the Application ID. So, first think this way, if you were to wrap this query in a function that will just return this query as it is, what would you do? You would create a function that takes the DataContext (dc named here), then two parameters named userName and applicationID, right? So, be it. We create one function that returns just this query: Next step is to replace this function with a Func<> representation that returns the query. This is the hard part. If you haven’t dealt with Func<> and Lambda expression before, then I suggest you read this and this and then continue. So, here’s the delegate...

Visit my Blog for more details.

Tips and tricks to rescue overdue projects

Monday, October 20, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

One of my friends, who runs his own offshore development shop, was having nightmare situation with one of his customers. He's way overdue on a release, the customer is screaming everyday, he's paying his team from his own pocket, customer is sending an ever increasing list of changes and so on. Here's how we discussed some ideas to get out of such a situation and make sure it does not repeat in future: Kabir: Hey, can you help me? My customer is making us work for free for extra two months to fix bugs from our last delivery. We did what he said. But after he saw the output, he came up with hundred changes, which he somehow presents as bugs or missing features and make them look like they are all our fault and making us work for last two months for free. He is sending new changes every week. We have no idea when we will complete the iteration. Omar: I see. Did you get a signed list of requirements from customer before you started the development? Kabir: Of course, I did. He sent us a word document explaining what he wants and we sent him a task breakup with hour estimates and total duration of three months. Now after three months when we showed him the product, he said, it's no where close to what he had expected. Then he sent a gigantic list of things to change. Omar: All of those are bugs? Kabir: Of course not. Most of them are new features. Omar: Then why don't you say those are new features? You have the original word document to prove. Just ask him to show where in the word document did he said X needs to be done? Kabir: Well..., he's tricky. He somehow makes things look like it is obvious that X needs to be done and he's not going to accept a requirement as done until X is done. For example, he said there must be a complete login form in the homepage. So, we did a typical login form with user name, password and OK, Cancel button. Now he says where's the email verification thing? We said, you did not ask for it. He said, "this is obvious, every login form has a forgot password and email verification; I said *complete* login form, not half-baked login form". So, you see, we can't really argue to keep our image. Then, we did the login form exactly how he said. Now he says, where the client side validations of proper email address, username length, password confirmation? We said, you never asked for it! He says, "come on, every single website nowadays has AJAX enabled client side validation, do I have to tell you every single thing? Aren't you guys smart enough to figure this out? You are already doing this for the third time, can't you do it really well this time?" Omar: OK, stop. I see what's your problem. Some customer will always try to make you work more for less money. They will try to squeeze out every bit of development they can for their bucks. So, you have to be extra careful on how much you commit to them and make sure they cannot chip in more...

Visit my Blog for more details.

An Agile Developer's workflow when SCRUM is used

Saturday, October 11, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

If you are planning to start SCRUM at your company, you might need to train developers and QA to get into the mindset of an Agile developer. SCRUM is only successful when the developers and QA get into the habit of following the principles of SCRUM by heart. So, sometimes you need to offer training or do trial sprints to give some room to your developers how to get used to the working fashion of SCRUM. Giving them a handy workflow diagram that shows how they should work helps soothe the steep learning curve required for non-super star developers. I made such a workflow while I was teaching SCRUM at my friend's company. The following diagram was printed and hung over the desk of each and every developer to help them grasp the culture of SCRUM quickly: We use Flyspray for issue tracking, so you will see the mention of it frequently. You will see the step to "Update Sprint backlog with remaining hours" is missing. This is done kinda verbally and scrum master (sometimes same person who is the product owner) keeps track of it. Hope you find this useful.

Visit my Blog for more details.

ASP.NET website Continuous Integration+Deployment using CruiseControl.NET, Subversion, MSBuild and Robocopy

Monday, October 06, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

You can setup continuous integration and automated deployment for your web application using CruiseControl.NET, Subversion, MSBuild and Robocopy. I will show you how you can automatically build the entire solution, email build report to developers and QA, deploy latest code in IIS all using CruiseControl.NET every N minutes. First get the following: CruiseControl.NET Subversion (install the command line tools and add the Subversion bin path to PATH environment variable) Robocopy (Windows Vista/2008 has it built-in, here's the link for Windows 2003) Install .NET Framework. You need it for MSBuild. You will learn how I have configured Continuous Integration and Deployment for my open source AJAX Portal project www.Dropthings.com. The code is hosted at CodePlex. When some developer makes a commit, CruiseControl downloads the latest code, builds the entire solution, emails build report and then deploys the latest web site to IIS 6.0. After installing CruiseControl.NET, go to Programs -> Cruise Control -> CruiseControl.NET Config. Now keep copying and pasting the following XML blocks and make sure you understand each block and make necessary changes: 1: <cruisecontrol> 2: <project name="Dropthings" queue="DropthingsQueue" queuePriority="1"> 3:  7: <workingDirectory>d:\cc\dropthings\code\trunk\</workingDirectory> 8:  9: <artifactDirectory>d:\cc\dropthings\artifact\</artifactDirectory> 10: <category>Dropthings</category> 11:  12: <webURL>http://localhost/ccnet/</webURL> 13: <modificationDelaySeconds>60</modificationDelaySeconds> 14: <labeller type="defaultlabeller"> 15: <prefix>0.1.</prefix> 16: <incrementOnFailure>true</incrementOnFailure> 17: <labelFormat>000</labelFormat> 18: </labeller> 19: <state type="state" directory="State" /> First change the working directory. It needs to be the path of the folder where you will have the solution downloaded. I generally create folder structure like this: D:\CC - Root for all CC.NET enabled projects \ProjectName - Root project folder \Code - Code folder where code is downloaded from subversion \Artifact - CC.NET generates a lot of stuff. All goes here. Next comes the Subversion integration block: 1: <sourcecontrol type="svn"> 2: ...

Visit my Blog for more details.

Using multiple broadband connections without using any special router or software

Sunday, October 05, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

I have two broadband connections. One cheap connection, which I mostly use for browsing and downloading. Another very expensive connection that I use for voice chat, remote desktop connection etc. Now, using these two connections at the same time required two computers before. But I figured out a way to use both connections at the same time using the same computer. Here's how: Connect the cheap internet connection that is used mostly for non-critical purpose like downloading, browsing to a wireless router. Connect the expensive connection that is used for network latency sensitive work like Voice Conference, Remote Desktop directly via LAN. When you want to establish a critical connection like starting voice conf app (Skype) or remote desktop client, momentarily disconnect the wireless. This will make your LAN connection the only available internet. So, all the new connections will be established over the LAN. Now you can start Skype and initiate a voice conference or use Remote Desktop client and connect to a computer. The connection will be established over LAN. Now turn on wireless. Wireless will now become the first preference for Windows to go to internet. So, now you can start Outlook, browser etc and they will be using the wireless internet connection. During this time, Skype and Terminal Client is still connected over the LAN connection. As they use persisted connection, they keep using the LAN connection and do not switch to the wireless. This way you get to use two broadband connections simultaneously. Here you see I have data transfer going on through two different connection. The bottom one is the LAN which is maintaining a continuous voice data stream. The upper one is the wireless connection that sometimes consumes bandwidth when I browse. Using Sysinternal's TCPView, I can see some connection is going through LAN and some through Belkin router. The selected ones - the terminal client and the MSN Messenger is using LAN where the Internet Explorer and Outlook is working over Wireless connection.

Visit my Blog for more details.

Best practices for creating websites in IIS 6.0

Saturday, October 04, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Every time I create an IIS website, I do some steps, which I consider as best practice for creating any IIS website for better performance, maintainability, and scalability. Here' re the things I do: Create a separate application pool for each web application I always create separate app pool for each web app because I can select different schedule for app pool recycle. Some heavy traffic websites have long recycle schedule where low traffic websites have short recycle schedule to save memory. Moreover, I can choose different number of processes served by the app pool. Applications that are made for web garden mode can benefit from multiple process where applications that use in-process session, in memory cache needs to have single process serving the app pool. Hosting all my application under the DefaultAppPool does not give me the flexibility to control these per site. The more app pool you create, the more ASP.NET threads you make available to your application. Each w3wp.exe has it's own thread pool. So, if some application is congesting particular w3wp.exe process, other applications can run happily on their separate w3wp.exe instance, running under separate app pool. Each app pool hosts its own w3wp.exe instance. So, my rule of thumb: Always create new app pool for new web applications and name the app pool based on the site's domain name or some internal name that makes sense. For example, if you are creating a new website alzabir.com, name the app pool alzabir.com to easily identify it. Another best practice: Disable the DefaultAppPool so that you don't mistakenly keep adding sites to DefaultAppPool. First you create a new application pool. Then you create a new Website or Virtual Directory, go to Properties -> Home Directory tab -> Select the new app pool. Customize Website properties for performance, scalability and maintainability First you map the right host headers to your website. In order to do this, go to WebSite tab and click on "Advanced" button. Add mapping for both domain.com and www.domain.com. Most of the time, people forget to map the domain.com. Thus many visitors skip typing the www prefix and get no page served. Next turn on some log entries: These are very handy for analysis. If you want to measure your bandwidth consumption for specific sites, you need the Bytes Sent. If you want to measure the execution time of different pages and find out the slow running pages, you need Time Taken. If you want to measure unique and returning visitors, you need the Cookie. If you need to know who is sending you most traffic - search engines or some websites, you need the Referer. Once these entries are turned on, you can use variety of Log Analysis tools to do the analysis. For example, open source AWStats. But if you are using Google Analytics or something else, you should have these turned off, especially the Cookie and Referer because they take quite some space on the log. If...

Visit my Blog for more details.

Create ASP.NET MVC Controllers under Namespace and specific URL

Saturday, October 04, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

When you have a lot of controllers, you need to organize them under Namespaces. Also, it is better to put controllers under subfolders in order to organize them properly and have some meaningful URL for them like /Admin/User where Admin is the subfolder and User is the controller. For example, you might have a lot of controllers that are APIs exposed by your web app, not regular pages. So, you might want to put them under /API/ folder. You also want to make sure no one can access those controllers from the root url. For example, no one must call /User/GetUserList instead they must call /API/User/GetUserList ASP.NET MVC default routing and Controller Factory is very greedy, it ignores the subfolders inside the "Controllers" folder. There's a DefaultControllerFactory class in ASP.NET MVC which traverses all controllers under the "Controller" folder and creates a cache using just the class name as the key. So, it ignores any namespace or any subfolder where you have put the controller. So, I created a derivative of the default controller factory and created a new factory that checks if the requested controller belongs to any specific namespace and whether that controller must be inside a specific subfolder. You can map /Admin folder to respond to all controllers that are under then YourWebApp.Admin namespace. Similarly, you can map /API folder to respond to all controllers that are under the YourWebApp.API namespace. None of these controllers can be accessed outside the specified URL. Here's the factory that does it: This controller checks the namespace of the controller being returned by ASP.NET MVC's default implementation and ensures the requested URL is the right url where controllers from the namespace can be accessed. So, this means if the controller matches MvcWebAPI.API.UserController, it ensures the URL being requested must be /MvcWebAPI/API/User. It will return null if the URL was something else like /MvcWebAPI/User or /MvcWebAPI/SomeotherFolder/User. Here's how you use it form Global.asax: You create a mapping for a Namespace and the subfolder it must belong to. Then you register the new Controller Factory as the default controller factory. Now the second catch is, the default Route for the {controller}/{action}/{id} won't work for you. You need to create a specific router that starts with the subfolder name. For example, API/{controller}/{action} Here' two things to notice: The use of PathStartWith routing constraint, which I will explain soon The last parameter which tells the route to include the API namespace for this route. Otherwise it can't find the controllers in the API namespace So, the PathStartsWith routing constraint ensures this route gets hit only when the requested URL is under the /API/ folder. For any other URL, it returns false and thus the routing handler skips this route. It just does a comparison on the AbsolutePath of the current request URL to...

Visit my Blog for more details.

Create REST API using ASP.NET MVC that speaks both Json and plain Xml

Friday, October 03, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

ASP.NET MVC Controllers can directly return objects and collections, without rendering a view, which makes it quite appealing for creating REST like API. The nice extensionless Url provided by MVC makes it handy to build REST services, which means you can create APIs with smart Url like "something.com/API/User/GetUserList" There are some challenges to solve in order to expose REST API: Based on who is calling your API, you need to be able to speak both Json and plain old Xml (POX). If the call comes from an AJAX front-end, you need to return objects serialized as Json. If it's coming from some other client, say a PHP website, you need to return plain Xml. Similarly you need to be able to understand REST, Json and plain Xml calls. Someone can hit you using REST url, someone can post a Json payload or someone can post Xml payload. I have created an ObjectResult class which takes an object and generates Xml or Json output automatically looking at the Content-Type header of HttpRequest. AJAX calls send Content-Type=application/json. So, it generates Json as response in that case, but when Content-Type is something else, it does simple Xml Serialzation. Here's the ObjectResult that you can use from Controllers to return objects and it takes care of proper serialization method. Above shows the Json serialization, which is quite simple. XmlSerialization is a bit complex though: Things to note here: You have to force UTF8 encoding. Otherwise it produces UTF16 output. XML Declaration is skipped because that's not quite necessary. Wastes bandwidth. If you need it, turn it on. I have turned on indenting for better readability. You can turn it off to save bandwidth. Some of you might be boiling inside looking at my obscure coding style. I love this style! I am spoiled by jQuery. I wish there was a cQuery. I actually started writing one, but it never saw day light just like my hundred other open source attempts. Now back to Object Serialization, we got the serialization done. Now you can return objects from Controller easily: You can use the test web project to call these methods and see the result: So far you have seen simple object and list serialization. A best practice is to return a common result object that has some status, message and then the real payload. It's handy when you only need to return some error but no object or list. I use a common Result object that has three properties - ErrorCode (0 by default means success), Message (a string data type) and Data which is the real object. When you want to return only a result with error message, you can do this: This produces a result like this: No payload here. So, the return format is always consistent. Those who are consuming service can write a common Xml or Json parsing code to consume both success and failure response. Those who are building API for their website, I humbly request you to return consistent response for both success and...

Visit my Blog for more details.

HTTP handler to combine multiple files, cache and deliver compressed output for faster page load

Thursday, August 28, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

It's a good practice to use many small Javascript and CSS files instead of one large Javascript/CSS file for better code maintainability, but bad in terms of website performance. Although you should write your Javascript code in small files and break large CSS files into small chunks but when browser requests those javascript and css files, it makes one Http request per file. Every Http Request results in a network roundtrip form your browser to the server and the delay in reaching the server and coming back to the browser is called latency. So, if you have four javascripts and three css files loaded by a page, you are wasting time in seven network roundtrips. Within USA, latency is average 70ms. So, you waste 7x70 = 490ms, about half a second of delay. Outside USA, average latency is around 200ms. So, that means 1400ms of waiting. Browser cannot show the page properly until Css and Javascripts are fully loaded. So, the more latency you have, the slower page loads. Here's a graph that shows how each request latency adds up and introduces significant delay in page loading: You can reduce the wait time by using a CDN. Read my previous blog post about using CDN. However, a better solution is to deliver multiple files over one request using an HttpHandler that combines several files and delivers as one output. So, instead of putting many <script> or <link> tag, you just put one <script> and one <link> tag, and point them to the HttpHandler. You tell the handler which files to combine and it delivers those files in one response. This saves browser from making many requests and eliminates the latency. Here you can see how much improvement you get if you can combine multiple javascripts and css into one. In a typical web page, you will see many javascripts referenced:<script type="text/javascript" src="/Content/JScript/jquery.js"> </script> <script type="text/javascript" src="/Content/JScript/jDate.js"> </script> <script type="text/javascript" src="/Content/JScript/jQuery.Core.js"> </script> <script type="text/javascript" src="/Content/JScript/jQuery.Delegate.js"> </script> <script type="text/javascript" src="/Content/JScript/jQuery.Validation.js"> </script> Instead of these individual <script> tags, you can use only one <script> tag to serve the whole set of scripts using an Http Handler:<script type="text/javascript" src="HttpCombiner.ashx?s=jQueryScripts&t=text/javascript&v=1" > </script> The Http Handler reads the file names defined in a configuration and combines all those files and delivers as one response. It delivers the response as gzip compressed to save bandwidth. Moreover, it generates proper cache header to cache the response in browser cache, so that, browser does not request it...

Visit my Blog for more details.

Loading static content in ASP.NET pages from different domain for faster parallel download

Friday, August 01, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Generally we put static content (images, css, js) of our website inside the same web project. Thus they get downloaded from the same domain like www.dropthings.com. There are three problems in this approach: They occupy connections on the same domain www.dropthings.com and thus other important calls like Web service call do not get a chance to happen earlier as browser can only make two simultaneous connections per domain. If you are using ASP.NET Forms Authentication, then you have that gigantic Forms Authentication cookie being sent with every single request on www.dropthings.com. This cookie gets sent for all images, CSS and JS files, which has no use for the cookie. Thus it wastes upload bandwidth and makes every request slower. Upload bandwidth is very limited for users compared to download bandwidth. Generally users with 1Mbps download speed has around 128kbps upload speed. So, adding another 100 bytes on the request for the unnecessary cookie results in delay in sending the request and thus increases your site load time and the site feels slow to respond. It creates enormous IIS Logs as it records the cookies for each static content request. Moreover, if you are using Google Analytics to track hits to your site, it issues four big cookies that gets sent for each and every image, css and js files resulting in slower requests and even larger IIS log entries. Let's see the first problem, browser's two connection limit. See what happens when content download using two HTTP requests in parallel: This figure shows only two files are downloaded in parallel. All the hits are going to the same domain e.g. www.dropthings.com. As you see, only two call can execute at the same time. Moreover, due to browser's way of handling script tags, once a script is being downloaded, browser does not download anything else until the script has downloaded and executed. Now, if we can download the images from different domain, which allows browser to open another two simultaneous connections, then the page loads a lot faster: You see, the total page downloads 40% faster. Here only the images are downloaded from a different domain e.g. "s.dropthings.com", thus the calls for the script, CSS and webservices still go to main domain e.g. www.dropthings.com The second problem for loading static content from same domain is the gigantic forms authentication cookie or any other cookie being registered on the main domain e.g. www subdomain. Here's how Pageflake's website's request looks like with the forms authentication cookie and Google Analytics cookies: You see a lot of data being sent on the request header which has no use for any static content. Thus it wastes bandwidth, makes request reach server slower and produces large IIS logs. You can solve this problem by loading static contents from different domain as we have done it at Pageflakes by loading static contents from a different domain e.g. flakepage.com. As the...

Visit my Blog for more details.

Open Source ASP.NET 3.5 AJAX Portal - new and improved

Monday, July 14, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

Last week I released a new version of Dropthings, my open source AJAX portal, that shows many fancy Web 2.0 features and showcases extensive use of ASP.NET 3.5, Workflow Foundation, C# 3.0 new language features, custom ASP.NET AJAX extenders, many performance and scalability techniques. I have written a book on these topics as well. The new version implements the following performance and scalability improvement techniques: 10 ASP.NET Performance and Scalability Secrets Fast ASP.NET page rendering by deferred script loading Load large amount of Javascripts in batch and thus load AJAX sites a lot faster Fast Streaming AJAX proxy that solves double downloading problems and continuously streams content from external domain Making best use of cache for high performance website On-demand UI loading on AJAX websites Here's how the new version looks: Hope you like the new design and the performance and scalability techniques that can significantly boost your ASP.NET website's quality. I highly recommend these techniques for ASP.NET websites. These are easy to implement and makes a world of difference in speed and smoothness for ASP.NET websites. I am thinking about making an ASP.NET MVC version of this portal using jQuery. Do you think it will be a hot area to explore? Share this post :

Visit my Blog for more details.

Deploy ASP.NET MVC on IIS 6, solve 404, compression and performance problems

Monday, June 30, 2008 | From Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

There are several problems with ASP.NET MVC application when deployed on IIS 6.0: Extensionless URLs give 404 unless some URL Rewrite module is used or wildcard mapping is enabled IIS 6.0 built-in compression does not work for dynamic requests. As a result, ASP.NET pages are served uncompressed resulting in poor site load speed. Mapping wildcard extension to ASP.NET introduces the following problems: Slow performance as all static files get handled by ASP.NET and ASP.NET reads the file from file system on every call Expires headers doesn't work for static content as IIS does not serve them anymore. Learn about benefits of expires header from here. ASP.NET serves a fixed expires header that makes content expire in a day. Cache-Control header does not produce max-age properly and thus caching does not work as expected. Learn about caching best practices from here. After deploying on a domain as the root site, the homepage produces HTTP 404. Problem 1: Visiting your website's homepage gives 404 when hosted on a domain You have done the wildcard mapping, mapped .mvc extention to ASP.NET ISAPI handler, written the route mapping for Default.aspx or default.aspx (lowercase), but still when you visit your homepage after deployment, you get: You will find people banging their heads on the wall here: http://forums.asp.net/t/1237051.aspx http://forums.asp.net/t/1253599.aspx http://forums.asp.net/p/1239943/2294813.aspx Solution is to capture hits going to "/" and then rewrite it to Default.aspx: You can apply this approach to any URL that ASP.NET MVC is not handling for you and it should handle. Just see the URL reported on the 404 error page and then rewrite it to a proper URL. Problem 2: IIS 6 compression is no longer working after wildcard mapping When you enable wildcard mapping, IIS 6 compression no longer works for extensionless URL because IIS 6 does not see any extension which is defined in IIS Metabase. You can learn about IIS 6 compression feature and how to configure it properly from my earlier post. Solution is to use an HttpModule to do the compression for dynamic requests. Problem 3: ASP.NET ISAPI does not cache Static Files When ASP.NET's DefaultHttpHandler serves static files, it does not cache the files in-memory or in ASP.NET cache. As a result, every hit to static file results in a File read. Below is the decompiled code in DefaultHttpHandler when it handles a static file. As you see here, it makes a file read on every hit and it only set the expiration to one day in future. Moreover, it generates ETag for each file based on file's modified date. For best caching efficiency, we need to get rid of that ETag, produce an expiry date on far future (like 30 days), and produce Cache-Control header which offers better control over caching. So, we need to write a custom static file handler that will cache small files like images, Javascripts, CSS, HTML and so on in ASP.NET cache and serve the...

Visit my Blog for more details.

Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5 News Feed