Asynchronous HTTP GETs being slow

Note to self:

If asynchronous HTTPS GETs from the same browser (in multiple tabs, say, or from an ajax site) seem slower than they should be - for example, if request roundtrip times get multiplied up by the number of tabs open - then it's very likely a bottleneck being caused by the browser cache. Firefox and Chrome at least (not tested any others) seem to only allow cache writing through a critical section or something. So the entire browser has a single, locked cache. And seems to lock it for the *entire* duration of the GET request. Which means aysnchronous GETs end up forming an orderly but very slow queue.

Turn off caching in the http request object and the problem goes away (though, of course, you don't get to cache anything any more).

Came as a surprise to me. Couldn't find an explanation for the behaviour when it cropped up - I assumed for ages our server was to blame. It wasn't.

Ill-Considered Smartphone Griping

My current phone, a Samsung Galaxy S, is the first smartphone and by far the most expensive phone that I've owned. And sometimes it hacks me off far more than any past phone. Let me start by saying that smartphones are amazing devices. They're so far beyond phones it's not funny. And that's sort of the problem; there's so much potential there, and the phone forces your expectations high by being so neat in so many way.

I'm likely to invest in a backup phone soon which is just a phone purely so I have a more rugged, less expensive phone with a less fleeting battery life. I just can't get comfortable heading out on the bike for a day with my SGS tucked in the back of my jersey. Aside from anything, if I didn't remember to fully charge it before setting out then who knows if it'll still be working when I need it late on after disaster has struck. But I think that's fair enough, and not really a criticism of the SGS - or smartphones in general. Complaining that a smartphone doesn't work well as a long battery life, cheap, bombproof emergency backup phone is like complaining that bread makes a lousy sandwich filling; simultaneously right and dumb.

Some of what annoys me about the SGS is probably device-specific. Some of it is probably OS-specific. And some of it isn't the phones fault at all. But I still blame it, because it's the point at which the annoyance enters my life. So where to start...?

I suppose at the beginning. The SGS, as delivered, was shit. When I first got the SGS, it rapidly became apparent that various bits of it just didn't work. Samsung had cunningly designed a device whose GPS performance was so flaky as to be useless. Use it to navigate and you'd find your apparent location hopping around between neighbouring streets. Recording routes showed similar problems. An entire feature of the phone was working so badly, it would have been a better experience if it hadn't been implemented at all. Several months later - somewhere between six months and a year - and the GPS was finally fixed enough to use. Another massive problem with the phone was the file system: In their wisdom, Samsung decided to use a half-baked in-house mess called RFS (essentially FAT with an attempt at journalling tacked on). It's utter shit, and very rapidly access times plummet to the point that the phone lags whenever accessing storage. The community cooked up various lagfixes fairly quickly, but the experience out of the box was significantly worse than one would hope for. On top of all that the official support software for the phone, the route used for official updates, was so bug-ridden that it took me hours (including hacking the registry) to get it to work at all. And, of course, I had to boot a Windows VM to use it anyway since there's no support for linux hosts. Again, the community stepped in and have provided a variety of functional alternatives.

There were other issues too, but those were the headlines. And here's the thing.

I cannot - absolutely cannot - understand how a flagship device was released in such a state, or how it was left for so long without being officially fixed. I can only assume that no one in Samsung ever bothered to test any of this stuff before release. It's inexcusable, in my opinion, for such fundamental and blindingly obvious problems to be present especially in an ostensibly high-end device. I have NO desire to go through a similar process of cobbling my phone into functionality over a period of months again. I don't mind hacking it to tweak it, but in my opinion the above isn't tweaking. It's getting the damn thing working in the first place.

Moving on. There are things about the phone that even now hack me off. The one that annoys me most regularly, oddly, isn't strictly the phones fault at all but the BBCs. When listening to catch-up radio on my phone.... LET ME TURN THE FRICKING SCREEN OFF. The Beeb, in their wisdom, implemented (and by "implemented" I mean "hacked together and pretended they'd thought it through") iplayer for Android in flash. Which (a) means my wifes phone can't use it *at all* and (b) means it's shit. I mean, it works, but it's hideously inefficient and also keeps the screen turned on whilever it's running. Even if you're just listening to the radio. Power drain much? It pisses me off *so* much that Android is stuck with this shit when Apple devices aren't. I know the Apple ecosystem more established, but it's the BBC for fucks sake. They should be providing the same quality of service to all the major smartphone platforms. Anyway. More annoyances: When the battery runs low, the phone turns the screen on to tell you. And pops up a popup. Which won't dismiss until you press the appropriate button. And keeps the screen on until you do. When the battery is low. WELL PLAYED SIR. Also, even without the shonky file system, for no reason I can figure out the device briefly becomes laggy and unresponsive. Then recovers. Given I've owned less powerful PCs (fairly recently) than this device, I expect more than groaning as it tries to render a webpage because it's doing whatever housekeeping in the background.

And moving towards a more general point... I think largely what annoys me with the SGS, aside from the specifics, is any time I notice it pretending to be a phone not a computer. I mean, honestly, it's a pocket computer. It should act like one. And this is where I think lots - if not all - smartphones are going wrong (for the likes of me). Whenever I want to do something to the phone but I can't because I'm not allowed... that hacks me off. I'm used to computers that I *own* (like I own my phone!). That I can do anything to because they're fricking MINE. Now, yes, you can root an android phone to get this fundamental right of ownership back. But why should I have to? It's MY DEVICE.

Updating the SGS seems far more difficult than it should be. Maybe apt has spoiled me, but what I expect from my device is, basically, that the OS and all the software on it updates with no faff. Every Android update has been a hassle, with some of them randomly stopping various applications from working (and you'll only find out if you try to run them). And there's always eother the threat or promise of data loss. Unacceptable! Sort it the fuck out. ESPECIALLY if you're going to lock down the system for the average user so it's hard/impossible to fix stuff if it does go wrong. In fact, why do I have to stick with Android? Why can't I pay for the OS of my choice? It's just a little computer, after all.

One of the big things I was interested in doing before getting an SGS was writing stuff for it. I figured once I've actually got a device I can have a think about what I want to do with it. Yet since getting the SGS I've written *nothing* for it. There are a few reasons for this, one good, one bad, one indifferent. The good reason is that there's an awful lot of stuff already out there. I didn't need to write many of the apps I wanted. That's nice. The indifferent reason is that I *hate* writing the same thing twice. And almost everything I've wanted to do that I CAN do (see below) has been better served by just writing a web app. Then I can use it from anywhere, on any phone. The bad reason really annoys me, and although I knew this when I was getting the SGS it's only become more annoying as I've used it. And this is the big one. Fix this, and I'll buy your phone:

GIVE ME USB HOST SUPPORT, AND LET ME WRITE STUFF THAT USES IT.

They're COMPUTERS. Think of what we could do if we could use them to control USB devices! IT WOULD BE AWESOME. There are *so many things* we could do. And I keep thinking of them, and having to dismiss them because mobile phone manufacturers are knobs.

I'm fairly sure there was other stuff I wanted to grouse about, but I thought a quick stream of consciousness rant that actually got written was better than a well considered, complete but nonexistent essay.

Why is ASP.net on mono like a new pet that hasn't been house trained?

...because it leaks all over the place.

We've been working for a while now on an MVC (initially MVC2, now MVC3) application running on mono 2.10.2 with FluentNHibernate 1.2 sitting on NHibernate 3.1, built as a web app but also available via a kiosked browser on the server. This effectively means we've got a web browser pointed at the server 24/7. In addition, it's a fairly heavily AJAX-ed application, pinging for data every second or so, which means we've got a browser sending requests to the application every second, 24/7. Still, not exactly heavy usage.

We've had a very, very hard time getting this working stably, and all of our troubles seem to be coming from the mono stack (possibly nhibernate as well, but even with that problem, it looks more likely that mono is the root cause). If you're putting together a web application using mono, you should probably look out for the following:

Memory leaks when running with sessions enabled

This seems to be something to do with CacheItems being created at a phenomenal rate without (apparently) ever being garbage collected. Even if you see no reason whatsoever why your application would be generating CacheItems, something under the hood will be there merrily churning them out. This will, eventually, bring your system to its knees before the mono process is killed. Switching garbage collectors to sgen (newer, better, see below...) doesn't help. This is almost certainly why mono web applications mysteriously stop working under heavy load according to the mono project.

Workarounds:

 You can, as suggested in the above link, restart your mono process every so often to release the memory. Not an ideal solution, obviously. Alternatively, if you don't need sessions you can run sessionless, which seems to stop this leak from happening (so it seems the CacheItems are being generated by something in the session code...). Or you could figure out what the fricking bug is and fix it.

An apparent race hazard when starting up a web app

If you choose to accept that you'll need to restart mono every so often, the next thing to watch out for is an intermittent exception when starting up. This occasionally manifests as an object not set to an instance of an object exception, but most commonly will appear as duplicate key exception thrown by the RoleManager - even when you aren't using roles for anything. To be honest, this being a race hazard is a total guess, I have no idea what the problem is, but something which is so intermittent just smacks of a race hazard. Since we didn't need sessions enabled, we simply switched to sessionless mode and stopped pursuing this one. Interestingly, since addressing all of the other problems listed in this post we've not seen any of these problems. So perhaps again something to do with the session code in mono...?

Workarounds:

 No idea. This seems to have just gone away as we've worked around all the rest of the weirdnesses. Best of luck to you if you're trying to solve this one. Sorry I can't help more.

Running the MVC sessionless is broken in mono

You should be able to run sessionlessly simply by adding the tag

< sessionstate mode="Off" > 

inside the system.web node in your web.config file. Providing you're not actually using TempData for anything, this should be fine. However, you'll get the following exception:

System.ArgumentNullException: Argument cannot be null.
Parameter name: httpSessionState
  at System.Web.HttpSessionStateWrapper..ctor (System.Web.SessionState.HttpSessionState httpSessionState) [0x00000] in <filename unknown>:0 
  at System.Web.HttpContextWrapper.get_Session () [0x00000] in <filename unknown>:0 
  at System.Web.Mvc.SessionStateTempDataProvider.LoadTempData (System.Web.Mvc.ControllerContext controllerContext) [0x00000] in <filename unknown>:0 
...

Workarounds:

According to the mighty Stack Overflow this has been fixed in .net. Alas not in mono. We worked around this by creating a dummy temp data provider

public class DummyTempDataProvider : ITempDataProvider
  {
   IDictionary ITempDataProvider.LoadTempData (ControllerContext controllerContext)
   {
    return null;
   }
   
   
   void ITempDataProvider.SaveTempData (ControllerContext controllerContext, IDictionary values)
   {
   }
  }

And then making sure all controllers overrode the CreateTempDataProvider function (since all our controllers inherit from a common base, we just stuck it in there).

protected override ITempDataProvider CreateTempDataProvider ()
  {
   //return base.CreateTempDataProvider();
   return new DummyTempDataProvider();
  }

Using the nHibernate WebSessionContext leaks memory

Our application uses nHibernate for persistence. Unfortunately, for some reason we've not been able to identify, if you use the WebSessionContext (which is what you should be using for a web application), you'll find that you are, once again, leaking memory. 

Workarounds:

 We found that the ThreadStaticSessionContext has no such trouble and - on a hunch - tested out a hybrid session context we had lying around from an earlier project. This works fine, no leaks. The only real difference between this and the WebSessionContext seems to be that our HybridSessionContext uses the Items hashtable in the HttpContext directly to store the session whereas the WebSessionContext creates a new hashtable inside the items hashtable which uses the ISessionFactory as the key to the Session. Why this should cause a leak is a mystery; it may have nothing to do with the leak. But it's the only obvious difference.

internal class HybridSessionContext : CurrentSessionContext
 {
  [ThreadStatic]
  private static ISession threadSession = null;
  private const string webSession   = "YourNamespace.YourProject.WebContext";
  
  
  public HybridSessionContext(ISessionFactoryImplementor factory)
  {
  }
  
  
  protected override ISession Session
  {
   get 
   { 
    return (HttpContext.Current != null) ? HttpContext.Current.Items[webSession] as ISession : threadSession;  
   }
   
   set 
   { 
    if (HttpContext.Current != null)  HttpContext.Current.Items[webSession]  = value;
    else         threadSession        = value; 
   }
  }
 }

The boehm garbage collector causes further leaks

After all that, you may still be seeing leaks. We've found that the boehm garbage collector that mono uses by default is rather easily confused, especially, for some reason, when it comes to web applications. 

Workarounds:

Try switching to mono-sgen, the generational garbage collector (see the link above). We found this moved us from a still inexplicably-leaking application to one behaving normally... while testing on xsp. Because...

mod_mono causes even more leaks

After all of the above were resolved, and everything looked peachy testing locally through XSP, we pushed the system onto a target machine and ran through mod_mono instead.... and got yet more memory leaks.

Workarounds:

Run through xsp and use mod_proxy to serve it via apache. And you'll get back to not having leaks.

To summarise

Currently, I'd say that ASP.net on mono is a bit of a mess. You can work round the problems to some extent, but we've spent a ridiculous amount of time dealing with them. Unfortunately the mono mailing list on this subject seems to have gone very quiet (Dan in the list is the other part of the "we" that I refer to in this post), and we're sufficiently out of time that we simply cannot continue to bughunt for the underlying causes of these issues without at least some support from folks who know their way around the guts of mono better than we do.