Archive for the ‘Uncategorized’ Category

SFU ResNet Stability (Jan/Feb/Mar)

Saturday, April 3rd, 2010

Instead of one post per month, I’m doing one post for January, February and March of 2010 because it’s the same story. ResNet hasn’t gotten any better, and if the outage numbers are any indication, managed to get worse.
January was notable for having 224 separate small outages, a new record, though March did come close with 214 separate outages. February clocked in at 107 outages, though it was likely lower due to the two-week Olympic break where many residents were not around (the two weeks of the break recorded 23 outages, while the other two had the remaining 84).

It appears that the ISP still has not configured their equipment properly, which in the case of a slow connection between Residence and SFU has been ongoing for over a year (the November post has more information on this issue).
Additionally, whenever the interconnect between SFU and Residence goes down, so does connectivity to SFU completely, which indicates that their router is not routing. This is in contrast with the way it should work, where a failed link is routed around, in this case going out over the ISP’s internet link and then back to SFU.
Why both of these issues continue to be issues, I can only guess at, but I suspect it is based on the ISP/Residence not caring enough to properly fix it.

Also noted were 13 outages on Jan. 4th and 25th, March 23rd and 14 outages on Feb. 5th. March 23rd also saw a few minute outage and drastically higher packet loss and latency into the evening. The reason given in was that someone had attacked the network internally, but there was no evidence provided, and given the veracity of previous statements, I would not put much weight behind it.
From my own logs and investigation, it looks more like someone, whether intentionally or not, was able to cause issues with the gateway due to poor network design/configuration.
Despite it being a preventable outage, no refund has been forthcoming, which was entirely expected due to Residence’s continued animosity towards taking steps to improve ResNet or treat their customers properly.

There was also an outage of a significant portion of an hour on the 15th of March and on the 29th of January. There was also notable instability, packet loss and latency on the evening of March 1st and 29th.
I realized when I was compiling these statistics that issues generally crop up in the evening, which is when the network is under the most load. In fact, upon more closely examining my logs, there is a definite trend in latency and packet loss throughout the day that peaks around midnight and falls off over the early morning hours, levels out in the late morning and afternoon and starts increasing around 5PM.
Which leads me to a conclusion I have previously reached, that the network capacity is too low and that the ISP is quite oversubscribed (though this is just further evidence, I reached that conclusion upon learning the 90:1 contention ratio).

Uptime was 98.1% in January, 98.9% (so close to 99%) in February and a more dismal 97.6% in March.
Downstream speed averages continued the trend, being at 10.3Mb/s in January, 11.0Mb/s in February and 9.8Mb/s in March, the first time the average speed fell below 10Mb/s, taking it out of the high-speed internet category (I consider 10Mb/s about minimum for something to be high speed, any lower than that and it’s not terribly usable for things requiring speed).
Average upstream bumped around .75Mb/s, for January, with February and March coming in at .89Mb/s and .71Mb/s respectively.

Average roundtrip time with the gateway was 38ms in January, 29ms in February, and a stunning bad 43ms in March. I say stunningly bad because it finally eclipsed the average roundtrip time on my Shaw connection, which goes to a much further distance, both in terms of the network and geographically.
Whereas Shaw’s gateway is downtown (roughly 15km away), ResNet’s gateway is on Residence, or a few hundred meters away and connected by low latency Ethernet and fiber, versus Shaw’s higher latency hybrid fiber and coax network.
As is a recurring theme, this just shows… You know how this goes, it’s overloaded.

The best summary I can come up with is that I am paying too much for too little. The ISP has not fixed problems that have been ongoing for over a year now, and Residence seems to be more interested in saving face than actually fixing issues.

SFU ResNet Stability (October)

Monday, November 2nd, 2009

Once again, I went through my logs pertaining to the stability of my ResNet connection for the month of October.  Overall, the connection was down more, but there was one whole day (out of 31) where no issues appeared. It also experienced two major outages.

For October, there were 109 outages of varying lengths, including a three hour outage on the 7th and a roughly seven hour outage on the 13th/14th (more on this one below).
There was also a roughly hour long outage (it wasn’t out for the whole hour, but it was too sow to be usable) early in the morning of the second, as well as large latency spike in the afternoon of the eighth and some during the day on the 19th.
The internet did work properly on the 11th, though this was likely due to reduced load do to many students not being in residence on account of Thanksgiving.

I was not actually at residence on the 13th, being home for the weekend and in transit from home, but I was able to monitor some of the situation from home via my Shaw connection.
The issues actually started on the 12, with noted high levels of packet loss and general instability appearing in the late afternoon and early evening. The ISP was notified, but likely due to the long Thanksgiving weekend, wasn’t able to do anything about it.
The 13th saw some of these issues clear up, but in the late afternoon, things started degenerating again. This lead to the internet going completely down around 6PM and staying out, with a few small exceptions for another 7 hours!
Additionally, routing to SFU, which seems to be hard-coded on both routers at the end of the route (which makes absolutely no sense to me) was not working until that evening, and it still continues (as of right now on November 3rd) to still not work properly.

I did inquire as to why the internet went down and spent so long down, and I was told that it was on account of an attack on the router. Why this was not fixed on the 12th when the issue first started cropping up has not been answered yet, and neither has why it took seven hours to fix a preventable problem.
Personally, I run software on my router (Snort, among others) that is really good at detecting and blocking potential attacks. I don’t know why the router wasn’t properly protected.

The internet connection was up a rather low 97% of the time, meaning that it was unusable for almost an entire day spread over the month!
Additionally, transfer rate was lower this month than last, with an average downstream speed of around 14Mb/s (versus ~17Mb/s for September) and around .75Mb/s upstream (versus .85Mb/s for September). Variation throughout the day increased further, with speeds dropping to around 5Mb/s during the slowest, peak hours.
Average ping on the internal network this month also crept up by roughly 13%.

Based on my observations from this month, I think my conclusion from last month that the network just isn’t designed to handle the load it is under is correct. The day of relative stability when lots of people weren’t here seems to indicate that when the load is low, stability is increased.
The large outage also highlighted the need for proper equipment, which represents a big factor in stability.

We have T-Shirts!

Friday, September 25th, 2009

I’d like to announce the availability of our new t-shirts: “Free as in Rum”.  This is taken from this post on my blog, and came about something like this:

Ffejery: “Is $foobar program Free as in Beer?”

Bob: “Not sure, but I’m sure it’s available on the Torrentz -grin-”

Ffejery: “So what would that make it… ‘Free as in Rum’?”

Behold:

Everything is Free.  This is the Third Kind.  Buy your t-shirts here.

Update: If you don’t get it, read this.

- The Ffejery