Poker Forum > Member Blogs
A Fish and his Chips
Chipaccrual:
You may or may not have been aware that Betfair had a few problems on Saturday afternoon/evening, meaning no access to their services for customers.
Unusually, they have posted up an explanation as to what went on, so customers can simply understand the situation.
--- Quote ---
When the website failed on Saturday, our first step was to disable Betfair for all our customers on the web, API and mobile services. Once we identified the actual problem, we determined that we needed our website "available" but with betting disallowed. We recovered the site internally around 18:00 and re-enabled betting as of 20:00 once we were certain it was stable.
Here is what actually happened:
After performing certain types of website changes, an issue developed that caused our servers to temporarily slow down, processing just one thing at a time (single threading) instead of thousands of user requests in parallel. This "single threading" behaviour was introduced some time ago to protect against occasional broken pages caused by serving content while it is changing. In tech speak, our servers weren"t thread-safe on certain types of content changes.
This has been an operational concern for several weeks as our traffic has reached record volumes week after week. While we had several operational protections in place to limit these types of changes during peak load, we missed an important one. Every 15 minutes, an automated process was publishing exactly the type of content that triggers the issue described above. Yesterday we hit a tipping point as the web servers reached a point where it was taking longer than 15 minutes to complete their update - essentially rendering the servers unusable.
Then in an attempt to quickly shed load, we triggered a process to disable some of the computationally intensive features on the site. Unfortunately, the way this was done triggered a complete recompile of every page on our site, for every user, in every locale. Under our normal Saturday usage, recovery took several hours.
After spotting the pattern, we"ve recognised this has been going on with varying impact since February 8, 2011. During periods of increased user traffic, our customers would experience this issue in the form of slow navigation or a "sticky" user experience. Yesterday was simply a tipping point, made worse by our recovery attempt.
We"ve fixed this problem now. We"ve disabled the original automated job and rebuilt it to update content safely. We"ve tripled the capacity of our web server farm to spread our load even more thinly. We"ve fixed our process for disabling features so that we won"t make things worse. We"ve updated our operational processes and introduced a whole new raft of monitoring to spot this type of issue. We"ve also isolated the underlying web server issue so that we can change our content at will without triggering the switch to single-threading.
We believe these changes will bring the stability we all desire for Betfair and thank you for your continued custom.
--- End quote ---
That explains it then. ;D
samuel_9:
I like the last part ................SO we wont make things anyworse?........nice to no they DONT no what there doing :-\
AAroddersAA:
--- Quote from: Chipaccrual on March 13, 2011, 09:58:23 AM ---
Clear Skys and then the Sky fell in.
Really enjoyed the day at the G Casino Luton for the Sky Poker Tour. 160+ runners with a 5k starting stack and 30 minute blinds. Had a quick catch up with Tighty and met up with Rich Baker and Dai, got my seat draw. Table 16, Seat 8. Last table, so probably first to break (Errr, wrong, would be last to break), not a problem though, shouldn"t be too tough a table.
Sat in seat 8, I had Team Sky Poker player Glitterbabe to my right. To his right was Julian Thew. To his right was James Hartigan. FML.
Really enjoyable table though. We were quickly made the feature table on the live stream, and away we went. Little to report in the first two levels. Won a couple of pots and was upto 6k.
Then got it allin preflop against Julian. AK v AQ. He"d been running quite good at the table, so everyone was quite shocked that he didn"t hit his queen outs. So, upto 12k, and get it allin preflop again with AK. This time against James Hartigan for his tourney life. James has KQ, and although we both made a flush, my ace of clubs was a nice comfort against his queen of clubs.
Now on 19k, and got three times average. I get another one allin preflop against a 5k stack. Pocket Jacks up against Ace Ten. Board comes AAxTx. Nothing wrong with it, just unlucky.
I then play Ace Four of clubs. I wasn"t completely happy with the way I played it, but actually it probably wasn"t that bad. Blinds were 300/600, folded round to me on the button. I raise to 1750, big blind calls. Flop comes out with two clubs. I bet out, BB calls. Blank on the turn. Big blind bets out, I reraise, BB flat calls and leaves himself about 2.5k back. River brings the killer club. Big Blind checks, I bet 2.5k, and eventually he reluctantly calls with his 9-10k stack now in the pot. I declare the nuts, and he just goes "oh".
Up to 25k, and that"s pretty much where I stayed for the next two hours. Just couldn"t find positions or cards. Tries a few steals that got played back on, managed to get a few through, but the levels passed, and before I knew it, I"d gone from three times average stack to less than average stack, but down to two tables.
Blinds were now 1.5k/3k/300. We are down to 19 players, and the top 17 are getting paid, with £220 a minimum cash. I"ve got 21k, average is now about 40k. If I can get a double up, I"m convinced I can do some damage in the latter stages of this.
It folds round to me in mid position, and I have pocket eights. Having not seen anything for ages, the temptation is to much not to ship it in. The player to my left pretty much instantly declares allin. Folds round the rest, and I"m up against poclet queens. A quiet "One Time Please Dealer" under my breath, but out came a queen on the flop to kill any hopes. he just had me covered, so I departed in 19th place.
Thoroughly enjoyed the 7 hours of play. The structure is a bit brutal, but means you have no option to play from the early stages. My problem is that I"d built a stack, but then didn"t push on. But the cards for those 2-3 hours were truly horrendous. I spent the last couple of hours with Dai Llewellyn to my right, and I"m sure he will vouch for all the Sighs and FFS every time I got dealt two cards. Dai managed to cash, so well played to that man.
All in all, it was a really enjoyable table to play poker on. I only actually had five showdowns in 7 hours.
AK v AQ ( For Tourney Life) - WON
AK v KQ - WON
JJ v AT - LOST
A4 v Mucked - WON
88 v QQ (For Tourney Life) - LOST
Don"t know where I get this ATC image from. ;D
Many thanks to Tighty and Tikay and all the guys at Sky, who work very hard to make these events great fun. Not sure if I"ll play another one this season, don"t know if I can face the one day structure. ;D
--- End quote ---
The final at DTD gotta worth a go, thats the one I want to play (it"s a bit above my bankroll tbh but I don"t mind taking a shot now and then). I will have a couple of goes at qualifying for Cardiff as well.
Chipaccrual:
--- Quote from: AAroddersAA on March 23, 2011, 20:09:54 PM ---
The final at DTD gotta worth a go, thats the one I want to play (it"s a bit above my bankroll tbh but I don"t mind taking a shot now and then). I will have a couple of goes at qualifying for Cardiff as well.
--- End quote ---
Unfortunately, the Sky Grand Final is the middle weekend of our Coast to Coast in September. I wonder if Jack and Geo fancy a slight detour. ;D
technolog:
--- Quote from: Chipaccrual on March 23, 2011, 20:18:36 PM ---
--- Quote from: AAroddersAA on March 23, 2011, 20:09:54 PM ---
The final at DTD gotta worth a go, thats the one I want to play (it"s a bit above my bankroll tbh but I don"t mind taking a shot now and then). I will have a couple of goes at qualifying for Cardiff as well.
--- End quote ---
Unfortunately, the Sky Grand Final is the middle weekend of our Coast to Coast in September. I wonder if Jack and Geo fancy a slight detour. ;D
--- End quote ---
We have a day off on the Sunday if that"s any good to you?
Personally I"m planning on having a bit of a thick head :)
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version