vBulletin Mods

The Official vBulletin Modifications Site
https://www.vbulletin.org/forum/showthread.php?t=127868

orban 24 Oct 2006 23:49

Well if it's crashing the server it's not :<

Quote:

Just curious "orban" having done some more checks when doing just a user "Find Threads Started by user" it is over a minute with the size of files we have - and from what you are saying this is the standard vB result. While once you add an addional key - search string it all works much faster as it is using Sphinx (is that right?).

Is there a reason you didn't code that using Sphinx?
Yeah but you'd have to add a fake string to all posts...mm....that's what you do right?

ALanJay 25 Oct 2006 08:29

Quote:

Originally Posted by orban
Yeah but you'd have to add a fake string to all posts...mm....that's what you do right?

If only it was that easy :)

If you enter a space it thows it away and does the standard search if you enter a single character it just says there are no such matches. If you have a longer string it only finds occurences that match not all all of them :)

mute 25 Oct 2006 19:26

We're rolling out our sphinx search when we upgrade our site to 3.6.2 on thursday. I'm hopeful that it will live up to my testing, but I am a tad worried that the "find posts by user" searches will be a bit pokey. I'm don't think it warrants a lot of concern given how often that particular type of search is actually done though..

Orban, have you been working on any other surprises lately? :)

orban 25 Oct 2006 21:10

You mean "find posts by user" without key words yeah?

No, haven't had a lot time lately. ;(

ALanJay 25 Oct 2006 22:22

Quote:

Originally Posted by mute
We're rolling out our sphinx search when we upgrade our site to 3.6.2 on thursday. I'm hopeful that it will live up to my testing, but I am a tad worried that the "find posts by user" searches will be a bit pokey. I'm don't think it warrants a lot of concern given how often that particular type of search is actually done though..

Just to make clear that "find posts by user" is fine and works very fast it is "Find THREADS STARTED by user" that is solw (still uses the internal code in vB - I think from the comments made).

Overall we have been using sphinx search now for nearly a week and it seems to work very nicely, the DELTA file is generated every 5 mintes with the full file being rebuilt each night in the early hours of the morning. Our vB main data file is around 4Gb in size (Threads: 467,561, Posts: 11,271,241, Members: 173,321).

We have disabled the "Find Threads STARTED by user" option for non admin users from the search options as a temporary measure - I'm not sure anyone used it in anycase.

Obviously in an ideal world it would be nice if this was searchable but it isn't a deal breaker.

kmike 26 Oct 2006 06:49

AlanJay, what's your MySQL version? I remember dealing with a bug in MySQL 4.0.x when "find threads started by user" query didn't use the proper indexes. Repairing the thread and post tables fixed that, but after some time the problem crept back in.

Update: found the same bug in the bookmarks, it appears not only 4.0.x are affected: http://www.vbulletin.com/forum/bugs....iew&bugid=4159
Looks like an intermittent index loss or corruption.

Also, FYI sphinx-0.9.7-rc1 has been released.
Another update: forgot to say that the crash bug has been fixed in RC1.

ALanJay 26 Oct 2006 08:22

Quote:

Originally Posted by kmike
AlanJay, what's your MySQL version? I remember dealing with a bug in MySQL 4.0.x when "find threads started by user" query didn't use the proper indexes. Repairing the thread and post tables fixed that, but after some time the problem crept back in.

We are using 4.1 version of mySQL (clients 4.1.18 to 4.1.21 and server 4.1.19).

Though I suppose the real goal is to remove this from the standard vB search and put it into a search done by sphinx (if possible) as with tables as big as ours I can understand why the results might take a little time to be returned.

Quote:

Originally Posted by kmike
Update: found the same bug in the bookmarks, it appears not only 4.0.x are affected: http://www.vbulletin.com/forum/bugs....iew&bugid=4159
Looks like an intermittent index loss or corruption.

Thanks will take a look.

Quote:

Originally Posted by kmike
Also, FYI sphinx-0.9.7-rc1 has been released.
Another update: forgot to say that the crash bug has been fixed in RC1.

Interesting from discussions it sounds like there is quite a number of changes so I assume there might be more to upgrading than a simple rebuild once it is properly released.

orban 26 Oct 2006 11:35

Yeah, I just saw 0.9.7-RC1 got released.

I think I will wait for one more release candidate or even for the final version, because I'm sure there'll be bugs. Once it's release I'll create a new how-to. It will probably not just be a simple rebuild of indices, yeah.

ALanJay 26 Oct 2006 12:59

Quote:

Originally Posted by orban
Yeah, I just saw 0.9.7-RC1 got released.

I think I will wait for one more release candidate or even for the final version, because I'm sure there'll be bugs.

:) - as the current version works so well I'm in no hurry.

Quote:

Originally Posted by orban
Once it's release I'll create a new how-to. It will probably not just be a simple rebuild of indices, yeah.

I suspected as much. Good luck when he gets that far.

By the way did you have a sugestion as to where was the best place to do the arraywalk in your sphinx.php code?

Can it be implemented in the lines like?


ie something like


$cl->SetGroups2 ( array_walk($sphinx_groups2, "intvalArray") );

Obvioulsy with the function elsewhere in the code.

function intvalArray(&$item, $key)
{
$item = intval($item);
}

When I try I get another error:

Invalid argument supplied for foreach()

in Sphinxapi.php in the places that the int_val check takes place.

kmike 26 Oct 2006 13:22

Well, as the creator of the multi group column support for 0.9.5 I can honestly say that it was a terrible copy/paste hack. 0.9.7 has them implemented properly, and the resulting index takes much less space which is always good from the I/O standpoint.

orban 26 Oct 2006 13:25

Quote:

Originally Posted by ALanJay
By the way did you have a sugestion as to where was the best place to do the arraywalk in your sphinx.php code?

Can it be implemented in the lines like?


ie something like


$cl->SetGroups2 ( array_walk($sphinx_groups2, "intvalArray") );

Obvioulsy with the function elsewhere in the code.

function intvalArray(&$item, $key)
{
$item = intval($item);
}

Yeah that's alright.

Quote:

Originally Posted by kmike
Well, as the creator of the multi group column support for 0.9.5 I can honestly say that it was a terrible copy/paste hack. 0.9.7 has them implemented properly, and the resulting index takes much less space which is always good from the I/O standpoint.

Hm okay. I'll have a look then. Thanks for telling me :)

ALanJay 26 Oct 2006 14:16

Well having tried that:

$cl->SetGroups2 ( array_walk($sphinx_groups2, "intvalArray") );


I end up with errors from sphinxapi.php

Invalid argument supplied for foreach()

in the code that look in the array for the values to be checked as integers. I'll have to do some more playing when I have some time. :(

orban 26 Oct 2006 14:26

Oh I'm sorry.

array_walk doesn't return the new array.

Do
array_walk($sphinx_groups2, "intvalArray");
$cl->SetGroups2 ( $sphinx_groups2 );


---

Also at the moment rewriting sphinx.php for 0.9.7-RC1.

In sphinx.conf just minimal changes were necessary, sphinx.php quite some changes. Currently recreating indices so I can start playing :)

---

Running 0.9.7-RC1. Minimal changes to sphinx.conf, a huge change to sphinx.php and it's running :) Will upload upgrade howto and full howto later! Going to the gym now, need to get strong! "Strong Mind, Strong Body"!?

ALanJay 26 Oct 2006 14:55

Quote:

Originally Posted by orban
Oh I'm sorry.

array_walk doesn't return the new array.

Do
array_walk($sphinx_groups2, "intvalArray");
$cl->SetGroups2 ( $sphinx_groups2 );

Thanks - that seems to work. Curiously I am now getting assertion errors further down in the date element:

/// set timestamps to match
function SetTimestampRange ( $min, $max )
{
assert ( is_int($min) );
assert ( is_int($max) );
assert ( $min<=$max );
$this->_min_ts = $min;
$this->_max_ts = $max;
}

Which is most odd. Looks like in 3.0.x everything is held as text.

It appears adding:

$datecut = intval($datecut);

Just before datecut is first looked at seems to sort that out.

As I assume intval doesn't do anything harmful so it can be used generically.

Thanks oban for the guidence.

orban 26 Oct 2006 14:57

Yeah that's weird. Just call

SetTimestampRange

with SetTimestampRange ( intval ( ... ), intval ( ... ) );


All times are GMT. The time now is 02:44.

Powered by vBulletin® Version 3.8.14
Copyright © 2021, MH Sub I, LLC dba vBulletin. All Rights Reserved. vBulletin® is a registered trademark of MH Sub I, LLC
Copyright ©2001 - , vbulletin.org. All rights reserved.