Tuesday, 26 July 2011

Why is ASP.net on mono like a new pet that hasn't been house trained?

...because it leaks all over the place.

We've been working for a while now on an MVC (initially MVC2, now MVC3) application running on mono 2.10.2 with FluentNHibernate 1.2 sitting on NHibernate 3.1, built as a web app but also available via a kiosked browser on the server. This effectively means we've got a web browser pointed at the server 24/7. In addition, it's a fairly heavily AJAX-ed application, pinging for data every second or so, which means we've got a browser sending requests to the application every second, 24/7. Still, not exactly heavy usage.

We've had a very, very hard time getting this working stably, and all of our troubles seem to be coming from the mono stack (possibly nhibernate as well, but even with that problem, it looks more likely that mono is the root cause). If you're putting together a web application using mono, you should probably look out for the following:


Memory leaks when running with sessions enabled

This seems to be something to do with CacheItems being created at a phenomenal rate without (apparently) ever being garbage collected. Even if you see no reason whatsoever why your application would be generating CacheItems, something under the hood will be there merrily churning them out. This will, eventually, bring your system to its knees before the mono process is killed. Switching garbage collectors to sgen (newer, better, see below...) doesn't help. This is almost certainly why mono web applications mysteriously stop working under heavy load according to the mono project.

Workarounds:

 You can, as suggested in the above link, restart your mono process every so often to release the memory. Not an ideal solution, obviously. Alternatively, if you don't need sessions you can run sessionless, which seems to stop this leak from happening (so it seems the CacheItems are being generated by something in the session code...). Or you could figure out what the fricking bug is and fix it.


An apparent race hazard when starting up a web app

If you choose to accept that you'll need to restart mono every so often, the next thing to watch out for is an intermittent exception when starting up. This occasionally manifests as an object not set to an instance of an object exception, but most commonly will appear as duplicate key exception thrown by the RoleManager - even when you aren't using roles for anything. To be honest, this being a race hazard is a total guess, I have no idea what the problem is, but something which is so intermittent just smacks of a race hazard. Since we didn't need sessions enabled, we simply switched to sessionless mode and stopped pursuing this one. Interestingly, since addressing all of the other problems listed in this post we've not seen any of these problems. So perhaps again something to do with the session code in mono...?

Workarounds:

 No idea. This seems to have just gone away as we've worked around all the rest of the weirdnesses. Best of luck to you if you're trying to solve this one. Sorry I can't help more.

Running the MVC sessionless is broken in mono

You should be able to run sessionlessly simply by adding the tag
< sessionstate mode="Off" > 
inside the system.web node in your web.config file. Providing you're not actually using TempData for anything, this should be fine. However, you'll get the following exception:

System.ArgumentNullException: Argument cannot be null.
Parameter name: httpSessionState
  at System.Web.HttpSessionStateWrapper..ctor (System.Web.SessionState.HttpSessionState httpSessionState) [0x00000] in <filename unknown>:0 
  at System.Web.HttpContextWrapper.get_Session () [0x00000] in <filename unknown>:0 
  at System.Web.Mvc.SessionStateTempDataProvider.LoadTempData (System.Web.Mvc.ControllerContext controllerContext) [0x00000] in <filename unknown>:0 
...

Workarounds:

According to the mighty Stack Overflow this has been fixed in .net. Alas not in mono. We worked around this by creating a dummy temp data provider
public class DummyTempDataProvider : ITempDataProvider
  {
   IDictionary ITempDataProvider.LoadTempData (ControllerContext controllerContext)
   {
    return null;
   }
   
   
   void ITempDataProvider.SaveTempData (ControllerContext controllerContext, IDictionary values)
   {
   }
  }

And then making sure all controllers overrode the CreateTempDataProvider function (since all our controllers inherit from a common base, we just stuck it in there).
protected override ITempDataProvider CreateTempDataProvider ()
  {
   //return base.CreateTempDataProvider();
   return new DummyTempDataProvider();
  }


Using the nHibernate WebSessionContext leaks memory

Our application uses nHibernate for persistence. Unfortunately, for some reason we've not been able to identify, if you use the WebSessionContext (which is what you should be using for a web application), you'll find that you are, once again, leaking memory. 

Workarounds:

 
We found that the ThreadStaticSessionContext has no such trouble and - on a hunch - tested out a hybrid session context we had lying around from an earlier project. This works fine, no leaks. The only real difference between this and the WebSessionContext seems to be that our HybridSessionContext uses the Items hashtable in the HttpContext directly to store the session whereas the WebSessionContext creates a new hashtable inside the items hashtable which uses the ISessionFactory as the key to the Session. Why this should cause a leak is a mystery; it may have nothing to do with the leak. But it's the only obvious difference.
internal class HybridSessionContext : CurrentSessionContext
 {
  [ThreadStatic]
  private static ISession threadSession = null;
  private const string webSession   = "YourNamespace.YourProject.WebContext";
  
  
  public HybridSessionContext(ISessionFactoryImplementor factory)
  {
  }
  
  
  protected override ISession Session
  {
   get 
   { 
    return (HttpContext.Current != null) ? HttpContext.Current.Items[webSession] as ISession : threadSession;  
   }
   
   set 
   { 
    if (HttpContext.Current != null)  HttpContext.Current.Items[webSession]  = value;
    else         threadSession        = value; 
   }
  }
 }


The boehm garbage collector causes further leaks

After all that, you may still be seeing leaks. We've found that the boehm garbage collector that mono uses by default is rather easily confused, especially, for some reason, when it comes to web applications. 

Workarounds:

Try switching to mono-sgen, the generational garbage collector (see the link above). We found this moved us from a still inexplicably-leaking application to one behaving normally... while testing on xsp. Because...



mod_mono causes even more leaks

After all of the above were resolved, and everything looked peachy testing locally through XSP, we pushed the system onto a target machine and ran through mod_mono instead.... and got yet more memory leaks.

Workarounds:

Run through xsp and use mod_proxy to serve it via apache. And you'll get back to not having leaks.

To summarise

Currently, I'd say that ASP.net on mono is a bit of a mess. You can work round the problems to some extent, but we've spent a ridiculous amount of time dealing with them. Unfortunately the mono mailing list on this subject seems to have gone very quiet (Dan in the list is the other part of the "we" that I refer to in this post), and we're sufficiently out of time that we simply cannot continue to bughunt for the underlying causes of these issues without at least some support from folks who know their way around the guts of mono better than we do.

6 comments:

Patrick Epstein said...

Great Post, Thanks!

Mau said...

After all this time, how did you manage to solve those mono issues?
I am planning to move one asp mvc application from windows 2008 to ubuntu server. I would like to know if as of today, it does make sense to even spend time on that experiment or not.
Thanks for the post by the way.

Dan said...

We stuck with mono but moved to a different web framework:
Blog post about the move

Throctukes said...

I'd say it's not worth the time if you just want to move an mvc application from windows to linux with minimal hassle and changes. But I'd also say that if this exercise has taught us anything it's that asp.net and mvc both sort of suck. If you can justify the time, I'd heartily recommend trying out something more lightweight such as ServiceStack. Be aware, though, that ServiceStack is about to change its licensing model from fully free and open source to commercially licensed for non-open-source projects. May not be an issue for you. There are other options, too, depending upon what you need. We're currently looking into Nancy which looks nice if you want something very lean.

Mau said...

Thanks for your comments. I am actually developing using ServiceStack. It makes sense to spend like two weeks doing the migration to Mono because that would save around $200 USD / month for hosting (taking into account I would get away with it with a 512Mb ubuntu server instead of the current 2Gb windows 2008 server).
I didn't know SS was going to be released as a commercial product, that could be a problem in the near future. Cheers.

Throctukes said...

It was announced a couple of weeks ago. The bright side, though, is that it means Demis is now working on ServiceStack full-time.