I planned to keep a bit of a log of my two-wheeled adventures. The problem is, I've left it so long since I was going to start this that I've got a bit of a backlog to get through. So we're going to have to do a touch of pretending we're back in time for this...
28th June - The Chilterns Challenge
Very very late in the day - just a couple of weeks before - I got a message from Ed saying he and Beth would be visiting that weekend. So how about doing this years Chilterns Challenge? Since I'd hoped to do it anyway, and was already training for a 100 mile ride from York later in the year, I leaped at the chance. The route was similar to last years and I'd really enjoyed that.
I say really enjoyed.
It was a nice route, but it damn near killed me getting up those hills... Admittedly, I'd been doing almost no exercise that time round, and was riding a heavy rustbucket of a bike, but still. It was tough. So I had a little trepidation (to say the least) about how I'd find it this year. All the more because the weather forecast as the day approached kept tending more towards "you are going to melt". I can't remember what temperature was threatened, but it was alleged to be feeling well in excess of 30 degrees.
The ednbeth arrived late on Thursday night in advance of the ride. We decided to leave the bikes in the car and sort out taking them round to the garage the next morning. Which we did - along with my shiny new secondhand Kona Caldera which they'd brought down from Glentress for me (yup, I got the bug... although where I'm going to find mountains round here is anyones guess - more on that in subsequent posts, I suspect). Unfortunately, Ed decided to ride Beths bike round to the garage while simultaneously pushing his. And fell off, tacoing his front wheel, bending his bars, mangling his grip tape and nearly severing the cables on that side. Fortunately, he had a couple of days in hand to Do Fixing and by the time Saturday night came, his bike had been bludgeoned back into something approaching a rideable state.
The night before the ride, we cooked Epic Pasta and resolved to get up before six, in a bid to beat the worst of the afternoon heat. Shockingly, we did so. After a couple of false starts (forgetting such vital tools as a pump the first time round), we were off.
Arriving at Henley, we found the tiny car park at the start absolutely rammed with cars, but through some sneaky parking-on-the-driveway we managed to bag a plot and begin assembling our glorious steeds. A brief queue for registration, and a mere quarter of an hour or so of faffing around later we were off!
...for all of about three minutes. And then we realised we'd forgotten the sunscreen. Ed, bless his enthusiastic little socks, decided he was the person to ride back to the car, grab the cream, and catch us back up. Beth and I were to pootle along until he caught up. And pootle we did. After a little while, we were beginning to get concerned that he'd maybe missed the first turn. We were just about to turn back and go searching when we saw him in the distance, bombing up the road. It turns out that our pootle was a bit more like a brisk zoom, and poor Ed was now wrecked already from trying to catch up with us. Oops. Still, he had some respite while we gooped up. But not much. For we'd managed to stop at the start of the first big climb of the day... now, I remembered this with much trepidation from last year. The two big hills on the route had very nearly finished me off, and I'd been worrying that this year would be no different despite the training. I was both surprised and delighted to scoot up the first one pretty easily - so much so that I found the energy to wonder why the fellow in front of me was insisting on wobbling all over the road rather than just dropping gears and spinning more, like I had. It's not like he was going any faster than me, and I was having a much easier time of it... As an aside, I should mention that this is something I really don't get about a lot of road cyclists. They seem to actively want to make life harder for themselves by not having a triple chain set. Now, I sort of see that for the top-end elite cyclists that not having that tiny amount of extra weight might make a difference. And they're elite, so they don't need a granny ring. But, honestly, the majority of the folks I see out aren't elite. They're just in pain. I suppose that part of the answer might be that it's not fashionable to have all the gears at the moment, so it's hard to find bikes with them. Anyway.
Zooming down from the first hill, the road was beginning to feel a little sketchy, and since I wasn't sure how far it was to the junction I started to slow down a bit. Ed zoomed past me (probably with a "why are you slowing down?"), and then realised - just as I did - that we were about to hit the T junction at the end of the road. Fortunately, our brakes were up to the job.
Excitement over, we settled in to a merry pace, chatting as we went. Seemingly in no time at all we were at the first food stop. We zoomed straight past, deciding we didn't need to stop, only to stop a mile or so down the road (I think Beth ran out of water and needed a transfer, but I'm not sure now). We timed that perfectly so we were just setting off again as we passed the event photographer, who somehow managed to completely miss us.
Now we were hitting the BIG climb of the ride. Last year it was at the start of the route, and I remember it being pure purgatory and seeming to go on forever. I started to realise as I overtook a few people on the way up it that either my new bike was awesome compared to the old one, or I was far, far fitter this year. I suspect mostly the former. Once we were all up that hill, we were fast approaching the route split. We'd all signed up for 100km, but for some reason the organisers had Ed and Beth down for 100 miles. Destiny maybe? We decided not, and opted for the short route whilst confidently proclaiming we'd do the 100 miles next year (which, incidentally, Ed and I had confidently proclaimed last year too. We were now onto the swoopy downhill fun-run I remembered so fondly from my previous attempt at this ride, and swoopy swift fun it was. We ate up the miles to the next food stop, where we paused for a re-gooping and to swipe some supplies. There were zip-vit energy gels being given out, and I bagged a few in case I needed them later. Curious as to their effect, though, I decided to scarf one down straight away to see what it was like.
Oh my word.
I'm assuming that these things are designed for people who are on the verge of collapse. To someone very far from that point, they had the effect of making me giggle hysterically as I rode for about the next twenty minutes, weaving around like a drunken loon. The next section of the ride is a bit of a blur, and soon enough we were pulling over at the lunch stop. Pasta and doughnuts all round! By the time we set off again, it was beginning to really get warm. Whilst we were moving, all was well since we were at least getting some airflow, but whenever we stopped at junctions it was like stepping into an oven. Also at this point, we were beginning to hit the really chossy roads that I'd been fortunate enough to be packing front suspension for last year. I really do mean chossy, too, with quite a lot of the section between lunch and the last food stop seemingly midway back to turning into a dirt track. The heavy rain from last night was clearly in evidence too - the roads were covered in bits of branches, stones and earth which had been washed into the road by the torrential rain. Not fun riding for the full-on road bikes, but since we were riding a tourer (Beth) and a cross bike (me) we weren't to bothered. I distinctly remember Ed complaining about the bumpy surfaces on his audax bike (with the trashed bars), though.
Midway through this middle-of-nowhere only-barely-still-a-road section, we found the highlight of weird for the journey. A traffic jam behind a land rover which had fallen partly into a field. Rather than the other drivers get out to help, they thought it best to stop the cyclists who were mid-way through an event to get them to push the car back out of the field. Nice one, guys. Being pleasant types, push we did.
As we were approaching the last food stop, we saw a marvelous sight: A family out on their bikes. With a child trailer! And balance bikes! Sweet! I slowed down to have a chat with them, only to be overtaken by a pair of tandems. If only there'd been a trike and a unicycle too, I'd have had the full set...
After the last stop, the heat was beginning to tell. whilst the terrain now was mostly flat, I found this section to be a long grind (partly, I guess, because I knew most of the fun stuff was over). Up one last, then down a track to the river and we were back into Henley, and nearly at the finish. Since it was regatta time, the traffic was insane, and just before the town centre I was separated from the rest of the posse. Since we were nearly done (and the traffic was scary-busy), I headed off to the finish alone. Once again, I missed my chance to get a photo as the tandems pulled into the finish just ahead of me, with many oohs and ahhs. The photographer didn't even notice me slinking in past them. I claimed my medal, and then stuck around waiting for the others. Just a couple of minutes later, Ed rolled up, and then Beth... who stopped just short of the line, and made to start taking her bike apart and put it back in the car. She hadn't noticed the huge inflatable finish line, and just assumed that entering the car park again was the finish. We shouted her over, and she walked across the line, and we'd finished! Huzzah! Time to pick up our t-shirts and medals, splat down on the grass for a rest and marvel at the sheer number of bugs we'd managed to glue to ourselves using sunblock as we rode.
All in all, excellent fun. But only 100km, and I still felt pretty tired by the end of it so I was still far from confident that the 100 mile ride from York later in the year would go so smoothly...
Thursday, 6 August 2009
Thursday, 16 July 2009
The Amazing Windows Specific Bug
I've just spent a week and a bit tracking down possibly the weirdest, most pernicious bug I've seen all year. It was obviously going to be a doozy - the application crashed only when the system had multiple cores, only when the input images were all of different sizes, and the crashes were both intermittent and unpredictable. Oh, and in the middle of code that had been seriously hammered in the past, and in the wild, and shown no signs of falling over.
Where to start?
Well, first of all, that intermittent crashing? That looks like a threading issue. It could just be memory corruption, but running on the exact same input data in the exact same order gave significantly different times to failure each run. It just feels like a threading issue. That plus it's only apparent on multi-core machines. A check of the code reveals some worker threads that are spawned in the midst of some number crunching to make use of all the available cores - so if there's only one core, there're no more threads. Aha, this looks likely. Right, so, if there's a weird issue going on here, we might be able to track it down using Valgrind, right? right?
Wrong. Because on Linux... there is no issue. Exact same source code. No crashing.
Here's a pseudocode version of the section of code in question:
In the main thread, for each image processed:
And in each worker thread :
So, to explain a little, what we have here is a pool of threads that are created at one time and then re-used through the lifetime of the application. Whenever we get into the number crunching code, the threads have their operating data passed in and they're resumed. As soon as all the jobs are processed, the threads signal that they're finished. Once all the threads are done, the main thread pauses them all and the program goes on its merry way.
Anyone figured it out yet?
There's one last piece of interesting information, to do with the way that threads are paused on the two different platforms. On Linux, the thread won't honour the Pause() call immediately, but will pause the next time it gets to TestDestroy() (for those wanting to replicate this, I'm using wxThread though this behaviour is a result of the underlying threading of the target OS). On Windows, the thread is paused immediately.
The upshot of this is that you can't guarantee, when you resume a thread in Windows, where it's going to start execution from. Now, with this last piece of information, it should all fall into place. See where the thread sets its busy flag to false? Right, in your head pause the thread there. Now, next iteration the thread will be set up properly by the initialise call, which raises the busy flag... only for it to instantly lower the flag when the thread resumes. So now the thread is merrily doing its processing, but it's signalling that it isn't, and is ready to be paused.
That in itself, while worrying, isn't so bad. What can happen next is. The Pause() can now happen anywhere in the worker threads loop. So if the thread happens to still be working when the main thread starts checking the workers to see if they're ready to be paused, it's more than likely going to be paused in the middle of doing number crunching on the input data.
And then, next iteration, you change all the input parameters and resume the thread mid-calculation with totally different data. If you're lucky, that means you get garbage out for this one iteration. If you're not, and your worker is mucking around with variable length data, you just wandered off your allocated heap and into the wild unknowns of Segfault City.
There's an easy fix, of course. The issue only arises because when the thread is in a state that it flags as "pausable", it's still performing write operations on itself (constantly re-clearing the busy flag). If we consider that the pause, and any re-initialisation of the thread data, is asynchronous to the thread entry (which it is) then it's obvious this is going to cause trouble. Instead, we can make sure that once the thread has signalled it's ready to be paused it no longer performs any writes. Since it's not going to be writing, it's now safe to pause it, do some asynchronous writes, and then set it going again. (note: I say "safe". It's safe in this context. You could still really confuse the thread if it's checking badly-considered conditionals that you just monkeyed with the control parameters of, but let's take that as read).
Where to start?
Well, first of all, that intermittent crashing? That looks like a threading issue. It could just be memory corruption, but running on the exact same input data in the exact same order gave significantly different times to failure each run. It just feels like a threading issue. That plus it's only apparent on multi-core machines. A check of the code reveals some worker threads that are spawned in the midst of some number crunching to make use of all the available cores - so if there's only one core, there're no more threads. Aha, this looks likely. Right, so, if there's a weird issue going on here, we might be able to track it down using Valgrind, right? right?
Wrong. Because on Linux... there is no issue. Exact same source code. No crashing.
Here's a pseudocode version of the section of code in question:
In the main thread, for each image processed:
//Initialise the shared job queue (that all threads will pull jobs from) this->jobQueue.Initialise(jobData); //Initialise and resume the worker threads for (i=0; i<threadCount; i++) { this->threads[i]->Initialise(data); this->threads[i]->Resume(); } while (true) { //Keep pulling jobs off of the shared job queue until there's no more work to do job = this->jobQueue.GetJob(); if (job) { this->DoProcessing(job) } else break; } do { //Wait for the worker threads to finish for (i=0; i<threadCount; i++) { busy = this->threads[i]->CheckBusy(); if (busy) break; } } while (busy); //Pause all the workers until we need them again for (i=0; i<threadCount; i++) this->threads[i]->Pause();
And in each worker thread :
Thread::Initialise(data) { HandleData(data); //Signal that the thread is now working this->busy = true; } Thread::Entry() { while (!this->TestDestroy()) { job = this->jobQueue->GetJob(); if (job) { this->CrunchNumbers(); } else { //Signal that the thread is no longer busy this->busy = false; } } }
So, to explain a little, what we have here is a pool of threads that are created at one time and then re-used through the lifetime of the application. Whenever we get into the number crunching code, the threads have their operating data passed in and they're resumed. As soon as all the jobs are processed, the threads signal that they're finished. Once all the threads are done, the main thread pauses them all and the program goes on its merry way.
Anyone figured it out yet?
There's one last piece of interesting information, to do with the way that threads are paused on the two different platforms. On Linux, the thread won't honour the Pause() call immediately, but will pause the next time it gets to TestDestroy() (for those wanting to replicate this, I'm using wxThread though this behaviour is a result of the underlying threading of the target OS). On Windows, the thread is paused immediately.
The upshot of this is that you can't guarantee, when you resume a thread in Windows, where it's going to start execution from. Now, with this last piece of information, it should all fall into place. See where the thread sets its busy flag to false? Right, in your head pause the thread there. Now, next iteration the thread will be set up properly by the initialise call, which raises the busy flag... only for it to instantly lower the flag when the thread resumes. So now the thread is merrily doing its processing, but it's signalling that it isn't, and is ready to be paused.
That in itself, while worrying, isn't so bad. What can happen next is. The Pause() can now happen anywhere in the worker threads loop. So if the thread happens to still be working when the main thread starts checking the workers to see if they're ready to be paused, it's more than likely going to be paused in the middle of doing number crunching on the input data.
And then, next iteration, you change all the input parameters and resume the thread mid-calculation with totally different data. If you're lucky, that means you get garbage out for this one iteration. If you're not, and your worker is mucking around with variable length data, you just wandered off your allocated heap and into the wild unknowns of Segfault City.
There's an easy fix, of course. The issue only arises because when the thread is in a state that it flags as "pausable", it's still performing write operations on itself (constantly re-clearing the busy flag). If we consider that the pause, and any re-initialisation of the thread data, is asynchronous to the thread entry (which it is) then it's obvious this is going to cause trouble. Instead, we can make sure that once the thread has signalled it's ready to be paused it no longer performs any writes. Since it's not going to be writing, it's now safe to pause it, do some asynchronous writes, and then set it going again. (note: I say "safe". It's safe in this context. You could still really confuse the thread if it's checking badly-considered conditionals that you just monkeyed with the control parameters of, but let's take that as read).
Thread::Entry() { while (!this->TestDestroy()) { //Make sure we don't keep setting the busy flag to false over and over if (this->busy) { job = this->jobQueue->GetJob(); if (job) { this->CrunchNumbers(); } else { this->busy = false; } } else Sleep(1); //Oh, and may as well consume fewer resources while we're at it } }
Labels:
debugging,
linux,
programming,
threading,
windows
Tuesday, 2 June 2009
Driving me UP THE FRICKING WALL
I don't drive a lot. I don't even have a car. But since sprogging it's become much less practical to always travel by train, so I end up hiring a car once a month or so - like last weekend. We had a family wedding to go to, so Friday night I picked up the car and set off to Cambridge the following day. Pretty easy drive on the way there (and rather a pleasant weekend), but the journey back on Sunday night... honestly, I swear people were deliberately trying to make me crazy.
So, if you're also a driver, here're a few tips for how not to meet with my ire:
On an even more disturbing note, I stopped off for a coffee on the way back. And noticed the little vending machine thing in the toilets, and couldn't help wondering: Have I been doing long distance journeys wrong all these years? Does everyone else regularly need to buy condoms, disposable hand wipes, lube, a vibrator and a packet of painkillers when travelling? If so, what on earth do you need that exact combination of items for? Am I driving unsafely as a result of my ignorance? It's a worry, I must say.
Oh, and while on the subject of things that drive me up the wall, please everyone: Don't say "blah blah blah" when you mean "and so on" or "for example" or "words fail me, but you know what I mean". It really gets on my nerves, and makes you appear to have a stunted vocabulary.
On top of that, stop eating cod. All of you. The wishful thinking must end before the species does. There are many more tasty fish in the sea.
Grr.
So, if you're also a driver, here're a few tips for how not to meet with my ire:
- Don't overtake people when you're in a slower lane than they are. Really, seriously, don't. It's not cool. It's more than a little dangerous - for a start, the person you just overtook isn't expecting you to be there, so may well pull into you, and on top of that you just removed their ability to get out of the way of everyone else. You know, they may actually have been trying to get out of your way when you decided that you couldn't wait?
- Indicating is really useful. Was it maybe abolished and I didn't get the memo? How the hell am I supposed to know what you're doing if you don't indicate? My favourite trick there is when you indicate you're about to overtake, and then the car behind with no warning pulls out and overtakes you. Maybe they're going for bonus style points?
- Don't drive along the middle lane of an otherwise free-flowing motorway at 50 miles an hour. Asshat.
On an even more disturbing note, I stopped off for a coffee on the way back. And noticed the little vending machine thing in the toilets, and couldn't help wondering: Have I been doing long distance journeys wrong all these years? Does everyone else regularly need to buy condoms, disposable hand wipes, lube, a vibrator and a packet of painkillers when travelling? If so, what on earth do you need that exact combination of items for? Am I driving unsafely as a result of my ignorance? It's a worry, I must say.
Oh, and while on the subject of things that drive me up the wall, please everyone: Don't say "blah blah blah" when you mean "and so on" or "for example" or "words fail me, but you know what I mean". It really gets on my nerves, and makes you appear to have a stunted vocabulary.
On top of that, stop eating cod. All of you. The wishful thinking must end before the species does. There are many more tasty fish in the sea.
Grr.
Subscribe to:
Posts (Atom)