Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #16  
Old 22 Jun 2014, 16:25
Ghostt Ghostt is offline
 
Join Date: Oct 2009
i just try to figure out how to fix this issue i have atm this query running 20 times with
600 query length

server has 8 cores running 2 websites.(wordpress and vbulletin)




.
i just installed mysql workbench and checked this long querys.

they are ALL from the same (sub )FORUmid 1101 only this forum has 750 000 threads.

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.


what can i do to prevent this query to process for testing? ive allready deactivated all setting if forum management . like thread views, search indexing, set forum incative, etc..
and i still see this query with forumid 1101

/edit
now i noticed that most slower querys 1-20 length atm are not "Sorting result" .
they are "sending data"


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.




_____________________________________

i checked debug in forumdisplay with id 1101.

i think this is the query:



Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.



/edit
also tested without forumdisplay.php shothread.php this query still apear.



this url for example loads very long: forumdisplay.php?f=1101&order=desc&page=101


maybe server managers assumption was right that vbthreads or some else table need innondb because it cant handle that mutch posts.

--------------- Added 22 Jun 2014 at 20:25 ---------------

so paul you also dont know what this query does exectly and how to fix? or at least test somehow to exclude this forum to process this query.

Last edited by Ghostt; 22 Jun 2014 at 21:02.
Reply With Quote
  #17  
Old 22 Jun 2014, 21:07
kh99 kh99 is offline
 
Join Date: Aug 2009
Real name: Kevin
Have you tried disabling the archive? Someone was having a problem like this a few weeks ago and it turned out to be spiders seaching the archive.
Reply With Quote
  #18  
Old 22 Jun 2014, 21:17
Ghostt Ghostt is offline
 
Join Date: Oct 2009
yes ive allready not activated archive. yes its also possible bots are crawling this pages causing more load. but ive tested allready Disallow: /*f=1101* in robots.txt for 5 minits no effect.

i just tested this httacess deny:

<files forumdisplay.php>
order allow,deny
deny from all
</files>

this query stopped...

how to deny forumdisplay.php?f=1101

for temporary solution

Last edited by Ghostt; 22 Jun 2014 at 21:35.
Reply With Quote
  #19  
Old 22 Jun 2014, 22:15
kh99 kh99 is offline
 
Join Date: Aug 2009
Real name: Kevin
I don't know how to do htaccess, but maybe you could go to Forum Permissions and edit the pemissions for Registered Users and make the forum unreadable.
Reply With Quote
  #20  
Old 27 Jun 2014, 16:54
Ghostt Ghostt is offline
 
Join Date: Oct 2009
ok its seems like vbulletin cant handle forumdisplay pages in very big forums with so many threads fast.
and search bots also overloading in forumdisplay with forumid 1101
we try innondb today
Reply With Quote
  #21  
Old 27 Jun 2014, 21:31
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
a lot of bots dont recognise robots.txt but follow some of the advice here http://antezeta.com/news/avoid-search-engine-indexing. For .htaccess use this
Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

just keep adding names to the string or use my mod for banning bots.

Your .htaccess file must reside in the forum root for this (you might have to set your control panel to view hidden files if you dont see it)
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #22  
Old 27 Jun 2014, 21:53
Ghostt Ghostt is offline
 
Join Date: Oct 2009
1. i dont want block all useragent especially not google....

i need to block them from only 1101 forumdisplay what ive done with this robot.txt line (hope this is correct):
Disallow: /*forumdisplay.php?f=1101&order=desc&page=*

but this is the emergency solution because its not a real fix of the bad performance of very big forumdisplay pages.
and how i see no one can realy help here .

and i allready use your ban spider addon. thanks for it, but we have sometimes problem with it.
if server crashes the cloudflare.com 502 error page is shown, after user refresh page. user get blocked/redirected from this addon.
you know why?
Reply With Quote
  #23  
Old 27 Jun 2014, 23:14
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
It will be to do with cloudflare caching i would imagine as my mod doesn't store any user/visitor details. As for your search engines try this, put this in your header template (or even forumdisplay but i think it must always use the header template)
Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Change x,y,z to be whichever forumid's you want to protect, change mysite.com to any url you want to redirect the spiders to and you should be golden
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #24  
Old 27 Jun 2014, 23:31
Ghostt Ghostt is offline
 
Join Date: Oct 2009
just tested htaccess code on the wordpress site on same server. connections/sec dropped from 200 to 10
with blocking this bastards: Baiduspider|yandex|anywordyoulike|like

so my robot.txt code not allways work you say?
seems to. but i will test your code tomorrow and check querys. thanks,
Reply With Quote
  #25  
Old 28 Jun 2014, 00:29
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
I put these in anywordyoulike|like to show you can include anything in the string, they're not actual bots Robots.txt is only obeyed by good legitimate (but maybe unwanted) bots, for the likes of Baidu...etc you can actually get in contact with them and ask them to stop indexing your site.
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #26  
Old 28 Jun 2014, 00:37
Ghostt Ghostt is offline
 
Join Date: Oct 2009
so baidu ignore robots.txt?
Reply With Quote
  #27  
Old 28 Jun 2014, 01:10
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
It seems that way as do many others like AhrefsBot, sosospider, Aboundex and even Bing to name but a few!

--------------- Added 28 Jun 2014 at 01:15 ---------------

For a more complete .htaccess block look here http://wpsecure.net/bad-bot-list/
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #28  
Old 28 Jun 2014, 11:53
Ghostt Ghostt is offline
 
Join Date: Oct 2009
i test it . SetEnvIfNoCase or rewrite Rules better?
both codes from the link dont work i think coz i see in cloudflare baidu still crawling.... first rewrite gives an error.
can i use this bots listed there for your write here?
or any better list. in your plugin ive a very big list but i think it will not work with that list because there are not complete spider names


RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider|yandex|bing) [NC]
RewriteRule .* - [R=403,L]

Last edited by Ghostt; 29 Jun 2014 at 01:06.
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 19:59.

Layout Options | Width: Wide Color: