Welcome Guest Active Topics
CCL Messageboard » CCL News » CCL News and Events » Network Issues Resolved
Joe
Thursday, March 20, 2014 9:00:57 PM
Rank: Administration
Groups: CCL Feature Crew, Forum Admin, Member, Subscriber
Location: Nashville
Joined: 1/5/2007 | Posts: 2,146 | Points: 12,972
Today around 7:15 there was a network issue at our ISP which resulted in some down time. The issue has been resolved. Sorry for the outage. We will know more about the cause in the next few hours. Again, so sorry for the inconvenience. They had been doing a lot of upgrades the last couple of days, so I am sure that it had to do with some of the new hardware they were adding. But once we know for sure, I'll let everybody know.

Joe





Joe Butler
CEO / Founder

You can reach us by phone at (615) 264-4747
and you can e-mail us at helpdesk@comiccollectorlive.com

padreglcc
Thursday, March 20, 2014 9:18:29 PM
Rank: Beyonder
Groups: Approver, Approver Steward, CR-Guidelines, CR-Management, DC Host, Forum Admin, Member, Moderator, New Releases Host
Location: Bangor, Michigan
Joined: 5/6/2007 | Posts: 5,313 | Points: 592,035
Thanks for the update, Joe. We appreciate it.
“One of the most sincere forms of respect is actually listening to what another has to say.” ~ Bryant H. McGill

Please make sure you read and understand the Forum Rules here.

Need help? Feel free to contact one of the Moderators or Approvers.
NossetStash
Friday, March 21, 2014 7:56:07 AM
Rank: Newbie
Groups: Member
Joined: 7/13/2013 | Posts: 4 | Points: 137
OOOOOOOOOOOhhhhhhh! Nailbiting
So that's what it was! I freaked out and ran a scan on my computer thinking it was me! Oh wells, at least I know my computer is clean lol.
Thanks for the update!
Atilla2k
Friday, March 21, 2014 8:28:53 AM
Rank: Herald of Galactus
Groups: Approver, Member, Subscriber
Location: NY
Joined: 10/8/2007 | Posts: 1,875 | Points: 122,255
Thanks for the update.
Joe
Friday, March 21, 2014 6:08:43 PM
Rank: Administration
Groups: CCL Feature Crew, Forum Admin, Member, Subscriber
Location: Nashville
Joined: 1/5/2007 | Posts: 2,146 | Points: 12,972
Incident
Incident Date: March 20, 2014
Systems Affected: All Network Services
Product: All network access to servers hosted with ISP

Synopsis of events
At approximately 6:14pm, March 20, 2014 the ISP network experienced a critical outage with one of our primary clusters. Most network services were down for less than an hour with a full restoration of services was complete by 8:35pm MT.

Timeline (All times Mountain Standard Time)
06:14 PM: Network Outage observed by ISP External Monitoring. Management personnel and internal staff were contacted and advised of the outage.

06:21 PM: ISP Staff began the task of triage and repair on affected servers and associated network services within 10 minutes, working to bring critical infrastructure back up. Engineering staff began troubleshooting via the out of band network. Management and System Engineers were onsite at the data center troubleshooting the local servers and SAN environment.

07:04 PM: The primary cluster was reset and services started returning to normal.

08:30 PM: All systems and services affected by the network outage were restored.

Root Cause
The cause of the outage at this point in time is still not entirely known; our Engineers are still analyzing logs and graphs to determine the cause.

Network Engineering during troubleshooting observed abnormal traffic that contributed to the outage. All network hardware on the backbone never experienced any hardware failures; it appeared network redundancy was operational.



Resolution
A full restart of the equipment on the cluster core returned affected services to a normal state.

Corrective Actions to Prevent Reoccurrence
Network engineers are in the process of reviewing the incident, and will make recommendations for the infrastructure to prevent this specific issue from reoccurring.





Joe Butler
CEO / Founder

You can reach us by phone at (615) 264-4747
and you can e-mail us at helpdesk@comiccollectorlive.com

icarus201
Saturday, March 22, 2014 4:09:30 AM
Rank: Beyonder
Groups: Approver, CR-Policies, Forum Admin, Member, Moderator, Movies Host, Subscriber, UK Host
Location: United Kingdom
Joined: 8/17/2007 | Posts: 43,972 | Points: 798,496
I'm still getting some problems. It's 9am (GMT) and I've been experiencing severe slowness in the website responding, such as being timed out with 502 - Server Error when trying to log in or clicking on a new thread.

Marvel Universe® ~ No-one Ever Really Dies.

Make sure that you read and understand the forum rules here
icarus201
Saturday, March 22, 2014 4:51:51 AM
Rank: Beyonder
Groups: Approver, CR-Policies, Forum Admin, Member, Moderator, Movies Host, Subscriber, UK Host
Location: United Kingdom
Joined: 8/17/2007 | Posts: 43,972 | Points: 798,496
It's flip-flopping in and out every 10 minutes or so. I get either;



OR:



Kinda hard to keep interested when it takes up to 10x longer to do what you intended to.

Marvel Universe® ~ No-one Ever Really Dies.

Make sure that you read and understand the forum rules here
Joe
Saturday, March 22, 2014 1:42:30 PM
Rank: Administration
Groups: CCL Feature Crew, Forum Admin, Member, Subscriber
Location: Nashville
Joined: 1/5/2007 | Posts: 2,146 | Points: 12,972
i saw this happen today too. The engineers are working on it. Please post if you are experiencing this. I am going to include this link in my ticket.

Thanks.





Joe Butler
CEO / Founder

You can reach us by phone at (615) 264-4747
and you can e-mail us at helpdesk@comiccollectorlive.com

Joe
Sunday, March 23, 2014 9:53:27 AM
Rank: Administration
Groups: CCL Feature Crew, Forum Admin, Member, Subscriber
Location: Nashville
Joined: 1/5/2007 | Posts: 2,146 | Points: 12,972
looks like we still had a backup process running that was causing some disk io problems. Should me smoken now!





Joe Butler
CEO / Founder

You can reach us by phone at (615) 264-4747
and you can e-mail us at helpdesk@comiccollectorlive.com

MoonKnight1
Sunday, March 23, 2014 10:07:49 AM
Rank: Beyonder
Groups: Forum Admin, Guru, Member, Moderator, Sports Host, Super Seller
Location: Richmond, VA
Joined: 5/24/2010 | Posts: 5,037 | Points: 21,264
Shop at My Store
Thanks Joe!

That was a very swift response and fix. I'm glad you were able to resolve the issue so quickly!
Quality. Consistency. Variety. Reliable. Fair. Friendly

Hall Liquidations



padreglcc
Sunday, March 23, 2014 4:15:26 PM
Rank: Beyonder
Groups: Approver, Approver Steward, CR-Guidelines, CR-Management, DC Host, Forum Admin, Member, Moderator, New Releases Host
Location: Bangor, Michigan
Joined: 5/6/2007 | Posts: 5,313 | Points: 592,035
Joe wrote:
looks like we still had a backup process running that was causing some disk io problems. Should me smoken now!


Applause Thanks, Joe!
“One of the most sincere forms of respect is actually listening to what another has to say.” ~ Bryant H. McGill

Please make sure you read and understand the Forum Rules here.

Need help? Feel free to contact one of the Moderators or Approvers.
Users browsing this topic
Guest

Forum Jump

Access

You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Main Forum RSS : RSS

This page was generated in 0.192 seconds.
(0.098 seconds)

All images on comic collector live copyright of their respective publishers. © Copyright 2015, MidTen Media Inc. GOLO231