Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #1  
Old 27 Oct 2014, 22:36
hilaryl hilaryl is offline
 
Join Date: Jan 2006
vBulletin Crashing Database each morning

Hi Everyone,

I'm going to try and be descriptive as possible, and hopefully get some help with a big issue we are having. We run a fairly big forum, 120,000+ users with over 7 million posts.

We currently are running vBulletin 4.1.2, and it's been running fine. Yesterday morning when trying to access the website, we got a database connection error. This essentially locked us out of accessing anything, when I try to log into phpmyadmin, I get the 'max-user-connections' message. So we had to get our host to restart the sql server.

After doing so, we were able to get back in to phpmyadmin, and could see that the 'searchcore_text' table was marked as crashed and needed to be repaired. The table is 3.6GB in size, so the repair took a very long time.

After repairing, the table no longer had the error on it, but the 'threadviews' table now was saying 'in use'. However there were no processes running on it - we thought it was a false positive.

After coming back in this morning, the website was saying database connection error again and our host is repairing the table again I believe.

Both times, the site seemed to go down around the 4am area.

I'm not sure what our best course of action is, the table is repairing, and saying it's fixed, but then we get database connection error again.

It's almost as if something is running at a specific time in the morning that locks a table, and then there's a massive build of queries trying to access that table, and that causes the connections to become overloaded and everything just shuts down.

What kind of query could be causing this though?

Please let me know what other information you need, we really need to get this sorted.

Thanks,
hilaryl
Reply With Quote
  #2  
Old 27 Oct 2014, 23:07
Zachery's Avatar
Zachery Zachery is offline
 
Join Date: Jul 2002
Real name: Zachery Woods
First, you should upgrade to 4.2.2
Second, you might want to move to new hardware, constant crashing is a bad sign for a server.
Third, you should invest in vBulletin sphinx from digitalpoint, or another vendor. The sphinx search system is 1000x better than what we're able to offer natively.
__________________
Looking for ImpEx?
Reply With Quote
  #3  
Old 28 Oct 2014, 01:26
hilaryl hilaryl is offline
 
Join Date: Jan 2006
Hi Zachery,

We are actually on 4.1.9 Patch Level 9 (I was incorrect earlier) - I don't imagine that makes things any better though.

We don't believe it's the hardware, because it's only started happening yesterday - and then again this morning. And there weren't any big changes made the day before the big crash.

I will talk to my host about vBulletin sphinx - but right now, we just need things going again.


We have managed to repair the tables and everything seems to be working now. With the exception of a glitch in certain threads where you can't get to the last page. One example being there are 12 pages, and even if you click on page 12, it only shows you the content of page 11. It even says page 12 in the URL.

I've read that this can fixed by Rebuilding Thread Information. Though there are 420,000 threads - should this be done with the forum offline?

Are there any other steps I should take after repairing database tables to kind of refresh the forum? Anything to do with the System Cache perhaps?

Thanks,
hilaryl
Reply With Quote
  #4  
Old 28 Oct 2014, 01:38
Zachery's Avatar
Zachery Zachery is offline
 
Join Date: Jul 2002
Real name: Zachery Woods
You should really upgrade to 4.2.2.
__________________
Looking for ImpEx?
Reply With Quote
  #5  
Old 28 Oct 2014, 01:50
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
If you have all security updates there is really no need to update unless the version you have is causing the crashes. Check server error logs and post them here
Reply With Quote
  #6  
Old 28 Oct 2014, 01:54
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Jul 2009
Real name: Chris
There is security fixes in 4.2.2 pl2 that is not in earlier versions, that is why it is advised to upgrade.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #7  
Old 28 Oct 2014, 01:58
hilaryl hilaryl is offline
 
Join Date: Jan 2006
Hi ForceHSS,

We do have the latest security updates - which is why we don't believe it's the version causing the issue.

In regards to the server error logs, I'll have a look through and see if there is anything worth posting.

My question before amount maintenance after a table repair, is there any advice anyone has for that?

Thanks,
hilaryl
Reply With Quote
  #8  
Old 28 Oct 2014, 02:41
AndrewSimm's Avatar
AndrewSimm AndrewSimm is offline
 
Join Date: Sep 2006
Is it crashing around the time you have a scheduled backup?
Reply With Quote
  #9  
Old 28 Oct 2014, 02:50
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
Originally Posted by hilaryl View Post
Hi ForceHSS,
In regards to the server error logs, I'll have a look through and see if there is anything worth posting.

Thanks,
hilaryl
If anything is in the server error logs even if you think its not worth posting post it as it might help
Reply With Quote
  #10  
Old 28 Oct 2014, 02:58
hilaryl hilaryl is offline
 
Join Date: Jan 2006
Originally Posted by AndrewSimm View Post
Is it crashing around the time you have a scheduled backup?
Actually yes, the back-up happens around the same time.

My host has moved the scheduled backup to an earlier time for tonight - seeing if that makes a difference.

Do you know of an issue that relates to the backup?
Reply With Quote
  #11  
Old 28 Oct 2014, 05:58
Zachery's Avatar
Zachery Zachery is offline
 
Join Date: Jul 2002
Real name: Zachery Woods
You should still upgrade to 4.2.2 for both security and bug fixes.
__________________
Looking for ImpEx?
Reply With Quote
  #12  
Old 28 Oct 2014, 06:45
hilaryl hilaryl is offline
 
Join Date: Jan 2006
Originally Posted by Zachery View Post
You should still upgrade to 4.2.2 for both security and bug fixes.
I agree we should upgrade to 4.2.2 - but I would prefer to upgrade an error free version of vBulletin. I think upgrading while having potential corruption issues would just cause more trouble then it would be worth.

It is definitely something we will look at after getting over this obstacle though.
Reply With Quote
  #13  
Old 28 Oct 2014, 12:39
Marv Marv is offline
 
Join Date: Jun 2002
Originally Posted by hilaryl View Post
Actually yes, the back-up happens around the same time.

My host has moved the scheduled backup to an earlier time for tonight - seeing if that makes a difference.

Do you know of an issue that relates to the backup?
Yes, we had something similiar. We found out that our ftp/server-repository had no space left and the backup task was setup to lock the tables and site until the backup was complete. Esp. the last point locked us out, flagged the site as inactive and we had to set the status back to active (deleting the unfinished backup task and restarting the server) via ssl first. Because there was no space left on the repository device (misconfiguration 6 yrs ago ) we had to alter the partition sizes used.
Reply With Quote
  #14  
Old 29 Oct 2014, 00:32
hilaryl hilaryl is offline
 
Join Date: Jan 2006
Originally Posted by Marv View Post
Yes, we had something similiar. We found out that our ftp/server-repository had no space left and the backup task was setup to lock the tables and site until the backup was complete.
Hi Marv,

Some great information there. Really appreciate it.

I have been speaking with my host and we are going to try a few things to hopefully isolate the cause of the crashes.

I'll come back with any new information I find.

Cheers!
Reply With Quote
  #15  
Old 29 Oct 2014, 01:25
AndrewSimm's Avatar
AndrewSimm AndrewSimm is offline
 
Join Date: Sep 2006
cPanel doesn't do a good job with large databases. Currently I use R1Soft CDP backup with the mysql extension and have no issues. I haven't used it to restore from any issues but I would like a more traditional backup option myself. It would be interesting to see whatever everyone else is using.
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 01:15.

Layout Options | Width: Wide Color: