[RedWingFan]

Originally Posted by UK Jimbo

Is the cronjob that's running the indexer creating any output?

Do you see any output if you run the indexer from the command line?

Can you copy/paste the command line you're using along with any output back here?

Here is the command line I use (within cron):

Block Disabled:      (Update License Status)
Suspended or Unlicensed Members Cannot View Code.

I just ran it with this output:

Block Disabled:      (Update License Status)
Suspended or Unlicensed Members Cannot View Code.

It's frustrating because it is an intermittent error--I kept trying to reproduce it but had no luck.

The output looks OK. But, it's updating the threaddelta.new.* files, not the threaddelta.* files (which remain at zero bytes, dated 5/29/09).

It's not a huge deal, since I can dump and regenerate all the indexes, but I just don't want this to happen if we put this on our "production" forum, and was curious to know how it happened.

[UK Jimbo]

Those .new files relate to the --rotate option and the indexer read more about it here: http://sphinxsearch.com/docs/current.html#ref-indexer

I would suspect that something preventing the indexer from rotating the files out perhaps the file permissions?

I'd suggest deleting all of the threaddelta files and then re-indexing.

[RedWingFan]

Originally Posted by UK Jimbo

Those .new files relate to the --rotate option and the indexer read more about it here: http://sphinxsearch.com/docs/current.html#ref-indexer

That was what I suspected--something like a temporary file, IOW.

Originally Posted by UK Jimbo

I would suspect that something preventing the indexer from rotating the files out perhaps the file permissions?

Nothing has changed on the server, so I'd attribute it to some kind of glitch in one (or all?) of those files dating back to May. Permissions and ownership matches all the others in the same directory, which was the first thing I checked.

I just tried deleting the threaddelta files, and reran the command line:

/usr/home/shtv/sphinx/bin/indexer --config /usr/home/shtv/sphinx/bin/sphinx_rr.conf threaddelta

This created a new set of threaddelta.* files. OK, so far so good. But then I go and retry with the --rotate option, and we're back to having the .new.* files, and the threaddelta.* files don't get updated.

In the same directory, the post.* and postmeta.* files are all working properly.

Deeply weird...I could see if I had changed the configuration of this mess awhile ago, but I actually haven't touched it since July last year, when I first installed it. No other changes on the server, and we have plenty of disk space.

Still poring over the Sphinx docs you pointed to...but am not seeing much else helpful yet.

Thanks!

[UK Jimbo]

have you tried restarting searchd and then trying it all over again?

Sounds like a strange one. Perhaps worth posting on the forum over at sphinxsearch.com

[RedWingFan]

Originally Posted by UK Jimbo

have you tried restarting searchd and then trying it all over again?

I'm willing to try anything. Although, the problem is with the indexer...unless something in searchd is somehow preventing the files from rotating properly (maybe searchd reporting that it's "busy", in other words, so the .new.* files don't get rotated in).

Originally Posted by UK Jimbo

Sounds like a strange one. Perhaps worth posting on the forum over at sphinxsearch.com

I'll do a search over there--thanks! I'll probably try dumping the entire index, regenerating a new one, restart searchd, etc., and start with a clean slate before pestering them too much over there.

[UK Jimbo]

Originally Posted by RedWingFan

I'm willing to try anything. Although, the problem is with the indexer...unless something in searchd is somehow preventing the files from rotating properly (maybe searchd reporting that it's "busy", in other words, so the .new.* files don't get rotated in).

In my mind it's 50:50 whether it's the indexer or searchd which is causing the problem. The indexer seems to be creating the index happily with the .new file name. After that it's the job of the search daemon to rotate the new index in.

With some of those .new files there what happens if you signal a restart to searchd with (not tested but I think this is correct).

Block Disabled:      (Update License Status)
Suspended or Unlicensed Members Cannot View Code.

That's the same method that the indexer uses to signal a rotate to searchd. I can't remember if searchd keeps a system log of it's activity. if so the restarts (and any possible problems) might be reported.

Good luck!

[RedWingFan]

I did:

Block Disabled:      (Update License Status)
Suspended or Unlicensed Members Cannot View Code.

...to stop, then restarted searchd. First time I ran the indexer, the .new.* files all were deleted. One hour and about 20 rotations later, they're still gone.

Thanks much--it worked!

[UK Jimbo]

Afraid I've not trawled the whole thread for this but here's a product that makes the install of the two small plugins even easier. Use at your own risk, etc...

[RedWingFan]

Originally Posted by UK Jimbo

Afraid I've not trawled the whole thread for this but here's a product that makes the install of the two small plugins even easier. Use at your own risk, etc...

Thanks! I'll try it out and let you know how it works. I'm about to pull the plug on fulltext and try Sphinx on our main (production) forum. Can't live w/o InnoDB tables, as I'm finding. I'll report back here, good or bad.

The hardest part is trying to remember what I did to get Sphinx running on our test forum. Working my way through the posts here, and my own notes...I think I'm getting it.

[UK Jimbo]

Originally Posted by RedWingFan

The hardest part is trying to remember what I did to get Sphinx running on our test forum. Working my way through the posts here, and my own notes...I think I'm getting it.

I'd recommend (this is true of any production roll-out):

• Backup test site
• Place a full copy of the live code (and db dump too if you think it's needed)
• Go through migration/roll-out process on your test platform making notes
• Repeat the above until you're confident of the steps you need to make
• Repeat on production. It'll be easy as you've just done it on the test platform!

[RedWingFan]

I have it running OK now.

Really basic (OK, stupid ) question. What do I need to turn OFF in vB to keep it from using its own internal search engine? IOW, aren't the built-in vBulletin search index tables going to be populated with the newest posts as they are made? From the way I see it, Sphinx bypasses the whole vB system, so vB shouldn't need to store any of the search terms anymore, unless another search process uses it. I've already dropped the fulltext indexes and changed the search type from fulltext back to vB internal, and am switching selected tables over to InnoDB to clear up the load issues we're having.

[amcd]

Originally Posted by RedWingFan

I have it running OK now.

Really basic (OK, stupid ) question. What do I need to turn OFF in vB to keep it from using its own internal search engine? IOW, aren't the built-in vBulletin search index tables going to be populated with the newest posts as they are made? From the way I see it, Sphinx bypasses the whole vB system, so vB shouldn't need to store any of the search terms anymore, unless another search process uses it. I've already dropped the fulltext indexes and changed the search type from fulltext back to vB internal, and am switching selected tables over to InnoDB to clear up the load issues we're having.

No.

First set the search type back to fulltext. This will tell VB not to populate the search tables (word and another one). The onus of maintaining search data now shifts to MySQL.

Then drop the fulltext indices from post and thread tables.

[RedWingFan]

I'll give that a try, thanks! My post and thread tables are converted to InnoDB, so they can't accept fulltext anyway.

Will doing this generate any kind of error, either from MySQL or vB? I don't think it will, but our visitors have a way of shaking out any type of rare bug or hiccup, when I least expect it.

[amcd]

If your tables are innodb, there will certainly be an error when you try to set search back to fulltext.

[RedWingFan]

Just looked at this again: if I try to change it back to fulltext, it will attempt to change the InnoDB tables back to MyISAM. Don't want that!

In the settings table, "fulltextsearch" has its "value" column set to "0". There are other parameters in that row. Think I'm safe to change "value" to "1" (and rebuild the datastore cache)?

I just have to comment that since changing tables back to InnoDB and implementing Sphinx, our forum runs SO much better now! I had fulltext previously. Searching for "Steve" as a search word, it would take 35-40 seconds to get results. (Steve is our forum's owner, so his name appears in most threads.) With Sphinx, the searches come back on average around 0.4 seconds. And with the InnoDB change, we don't have stacks of queries waiting in the queue anymore.

The only hiccup I've had is that I once again had a set of *.new.* files, this time for my post indexes. I killed and restarted searchd, and reran the update, and it all rotated properly. My clue was a forum member saying he couldn't search for his posts for the past few days. Sure enough, the stale indexes were dated around the time he was unable to find his posts. I may have to run a cron job to check for any *.new.* files in that directory, and possibly put together a shell script to kill and restart searchd.