Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #1  
Old 14 Sep 2011, 15:25
Fergal C Fergal C is offline
 
Join Date: Dec 2004
Real name: Fergal Crawley
Consistent VB4 downtime over 5 days - Where do I go from here?

I'm extremely frustrated by what's happening on our business forum, that I've put a huge amount of time and money into. It runs on VB4 and is on an expensive VPS with Servint. Over the last five days the forum has been losing data from the database and hence going down regularly. Unfortunately this is very damaging to the community and our search engine rankings. The current downtime is the worst and it has now been down all day - a minimum of over eight hours since I first looked at it this morning.

I've opened a support ticket with VB.com and they have been very helpful. They feel that the issue is happening because of issues with the hosting account. The issue has been escalated to Servint's Engineers who have also been helpful, but they say its a vBulletin issue and that everything is running as it should on the hosting account.

I've paid a reputable and knowledgeable VB Admin to look at the problem and as of yet he hasn't been able to figure out the cause. He has a mirror of the site running on his machine and it's running fine, plus we have a test site on the same server which is also running fine.

These issues might be a little less frustrating if I use using free software on a shared hosting account, but I have not been shy about putting the needed resources into the forum and it has had severe downtime over the last five days.

Any ideas as to what I need to do to determine the cause of these issues?

How can I ensure that something like this doesn't happen again?

Thanks!
__________________
Business Advice Forum
Forum Promotion
[Previous username HostBider.com]
Reply With Quote
  #2  
Old 14 Sep 2011, 18:51
Lynne's Avatar
Lynne Lynne is offline
 
Join Date: Sep 2004
Real name: Lynne
When you say you are "losing data from the database", do you mean that literally - one minute it is there and the next it is not? Have you and your host looked into the possibility that you are being hacked? That someone is just playing with you?
__________________
Former vBulletin.org Staff Member

Try a search before posting for help. Many users won't, and don't, help if the question has been answered several times before.
W3Schools -
Online vBulletin Manual
If I post some CSS and don't say where it goes, put it in the additional.css template.
I will NOT help via PM (you will be directed to post in the forums for help.)
Reply With Quote
  #3  
Old 14 Sep 2011, 20:11
Fergal C Fergal C is offline
 
Join Date: Dec 2004
Real name: Fergal Crawley
Thanks Lynne, yes the host has looked at that possibility and they say that there is no evidence of the site being hacked.
__________________
Business Advice Forum
Forum Promotion
[Previous username HostBider.com]
Reply With Quote
  #4  
Old 14 Sep 2011, 21:05
realmr realmr is offline
 
Join Date: Aug 2008
is it possible u have a cron job running for ur particular domain alone ? Have u checked disabling add-ons and enabling them back on after a day or 2 to find the difference ? There could be an outdated add-on which could be causing the issue.
Reply With Quote
  #5  
Old 14 Sep 2011, 21:21
Digital Jedi's Avatar
Digital Jedi Digital Jedi is offline
 
Join Date: Oct 2006
Real name: Mark Daniel Martinez
Did you ever consider going back to a copy of the database from before the problems started occurring? True, that might be a last resort depending on the situation. But if you haven't had good up time since the problems started, maybe the loss will be negligible. At the least, you narrow down the potential list of problems.
__________________
Reply With Quote
  #6  
Old 15 Sep 2011, 15:38
Fergal C Fergal C is offline
 
Join Date: Dec 2004
Real name: Fergal Crawley
Originally Posted by realmr View Post
...Have u checked disabling add-ons and enabling them back on after a day or 2 to find the difference ? There could be an outdated add-on which could be causing the issue.
That's something we are doing now, thanks for the suggestion.

Originally Posted by Digital Jedi View Post
Did you ever consider going back to a copy of the database from before the problems started occurring? True, that might be a last resort depending on the situation. But if you haven't had good up time since the problems started, maybe the loss will be negligible. At the least, you narrow down the potential list of problems.
Thanks for the suggestion. We have been running now since yesterday evening without any major problems. However, that is something we will consider seriously if we run into problems again.

It now appears that the issue may have been caused by some old data that was backed up on the VPS. I am currently working with Servint to have this data deleted and hopefully that will resolve the issue once and for all.
__________________
Business Advice Forum
Forum Promotion
[Previous username HostBider.com]
Reply With Quote
  #7  
Old 15 Sep 2011, 18:07
Digital Jedi's Avatar
Digital Jedi Digital Jedi is offline
 
Join Date: Oct 2006
Real name: Mark Daniel Martinez
Originally Posted by Fergal C View Post
That's something we are doing now, thanks for the suggestion.


Thanks for the suggestion. We have been running now since yesterday evening without any major problems. However, that is something we will consider seriously if we run into problems again.

It now appears that the issue may have been caused by some old data that was backed up on the VPS. I am currently working with Servint to have this data deleted and hopefully that will resolve the issue once and for all.
Yeah, sometimes it's better to shut down temporarily then try to run with errors, particularly if they're compounding over time. Much harder for monetized sites to have to do, but it can save you and your users some frustration in the long run. Hope everything gets worked out.
__________________
Reply With Quote
  #8  
Old 18 Sep 2011, 10:40
Fergal C Fergal C is offline
 
Join Date: Dec 2004
Real name: Fergal Crawley
Today is the ninth day of serious downtime on my forum. My host Servint persisted in blaming the problems on the vBulletin software and only today said;

One of the causes may be the size of your /tmp directory. If the tmp directory fill up while the mysql socket is still trying to write information, then the database can become corrupt. I've removed the size of /tmp by remounting it on your server. I'll need to restore the databases from backup to get them working again
I've had a VPS with Servint for the last seven years and I have to say that I'm extremely disappointed by their lack of support over the last nine days. Servint are not a cheap host and downtime like that is not good enough for a forum that I have invested so much in.
__________________
Business Advice Forum
Forum Promotion
[Previous username HostBider.com]
Reply With Quote
  #9  
Old 18 Sep 2011, 11:57
SocialNetworkBuzz SocialNetworkBuzz is offline
 
Join Date: May 2007
Did you run repair on your database? Did you try restarting your VPS? Sometimes we over look the simplest things, resolved by a database repair or machine reboot =)
__________________
Reply With Quote
  #10  
Old 18 Sep 2011, 12:25
SCRIPT3R SCRIPT3R is offline
 
Join Date: Jan 2005
Why does your homepage address redirect to http://www.businessadviceforum.com/install/install.php? Might want to look into that.
Reply With Quote
  #11  
Old 18 Sep 2011, 14:36
Fergal C Fergal C is offline
 
Join Date: Dec 2004
Real name: Fergal Crawley
Thanks Stabatha and ScripT3R we did all of those things and they haven't been working. Servint seem to have identified what's causing the problem and hopefully they will fix it soon.
__________________
Business Advice Forum
Forum Promotion
[Previous username HostBider.com]
Reply With Quote
  #12  
Old 18 Sep 2011, 14:50
SCRIPT3R SCRIPT3R is offline
 
Join Date: Jan 2005
Originally Posted by Fergal C View Post
Thanks Stabatha and ScripT3R we did all of those things and they haven't been working. Servint seem to have identified what's causing the problem and hopefully they will fix it soon.
No you didn't, it's still redirecting to your intallation.php... in the wrong hands, this will cause all the errors you are describing. I can access your installation files right now and probably do some damage if I wanted to. This means anyone else can too.
Reply With Quote
  #13  
Old 18 Sep 2011, 17:05
borbole's Avatar
borbole borbole is offline
 
Join Date: Jan 2010
Indeed, I loaded the page and it is redirecting to the installer but it shows a firefox error message instead of the installation page. But I agree, the installer should not be left like that where everyone can access it as it poses a security risk.
__________________
My mods.
Reply With Quote
  #14  
Old 19 Sep 2011, 16:38
Fergal C Fergal C is offline
 
Join Date: Dec 2004
Real name: Fergal Crawley
The error may have appeared to have been caused by a re-direct, but it was actually caused by a DB issue. The site is now back up and running and Servint have confirmed that;

...the size of the /tmp directory was the issue. By default on our servers /tmp is it's own secure partition that is 100Mb in size. [Engineer's name removed] disabled the secure partition. This means that /tmp was bundled into the main server partition and so can use as much space as is left on your server. This should stop this from occurring again.
__________________
Business Advice Forum
Forum Promotion
[Previous username HostBider.com]
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 13:41.

Layout Options | Width: Wide Color: