Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #691  
Old 15 Jul 2009, 21:20
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
Originally Posted by UK Jimbo View Post
Is the cronjob that's running the indexer creating any output?

Do you see any output if you run the indexer from the command line?

Can you copy/paste the command line you're using along with any output back here?
Here is the command line I use (within cron):


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

I just ran it with this output:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

It's frustrating because it is an intermittent error--I kept trying to reproduce it but had no luck.

The output looks OK. But, it's updating the threaddelta.new.* files, not the threaddelta.* files (which remain at zero bytes, dated 5/29/09).

It's not a huge deal, since I can dump and regenerate all the indexes, but I just don't want this to happen if we put this on our "production" forum, and was curious to know how it happened.
__________________
-= N =-
Reply With Quote
  #692  
Old 15 Jul 2009, 21:57
UK Jimbo's Avatar
UK Jimbo UK Jimbo is offline
 
Join Date: Sep 2002
Those .new files relate to the --rotate option and the indexer read more about it here: http://sphinxsearch.com/docs/current.html#ref-indexer

I would suspect that something preventing the indexer from rotating the files out perhaps the file permissions?

I'd suggest deleting all of the threaddelta files and then re-indexing.
__________________
administrator: visordown.com
Reply With Quote
  #693  
Old 15 Jul 2009, 23:19
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
Originally Posted by UK Jimbo View Post
Those .new files relate to the --rotate option and the indexer read more about it here: http://sphinxsearch.com/docs/current.html#ref-indexer
That was what I suspected--something like a temporary file, IOW.

Originally Posted by UK Jimbo View Post
I would suspect that something preventing the indexer from rotating the files out perhaps the file permissions?
Nothing has changed on the server, so I'd attribute it to some kind of glitch in one (or all?) of those files dating back to May. Permissions and ownership matches all the others in the same directory, which was the first thing I checked.

I just tried deleting the threaddelta files, and reran the command line:

/usr/home/shtv/sphinx/bin/indexer --config /usr/home/shtv/sphinx/bin/sphinx_rr.conf threaddelta

This created a new set of threaddelta.* files. OK, so far so good. But then I go and retry with the --rotate option, and we're back to having the .new.* files, and the threaddelta.* files don't get updated.

In the same directory, the post.* and postmeta.* files are all working properly.

Deeply weird...I could see if I had changed the configuration of this mess awhile ago, but I actually haven't touched it since July last year, when I first installed it. No other changes on the server, and we have plenty of disk space.

Still poring over the Sphinx docs you pointed to...but am not seeing much else helpful yet.

Thanks!
__________________
-= N =-
Reply With Quote
  #694  
Old 15 Jul 2009, 23:35
UK Jimbo's Avatar
UK Jimbo UK Jimbo is offline
 
Join Date: Sep 2002
have you tried restarting searchd and then trying it all over again?

Sounds like a strange one. Perhaps worth posting on the forum over at sphinxsearch.com
__________________
administrator: visordown.com
Reply With Quote
  #695  
Old 16 Jul 2009, 03:07
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
Originally Posted by UK Jimbo View Post
have you tried restarting searchd and then trying it all over again?
I'm willing to try anything. Although, the problem is with the indexer...unless something in searchd is somehow preventing the files from rotating properly (maybe searchd reporting that it's "busy", in other words, so the .new.* files don't get rotated in).

Originally Posted by UK Jimbo View Post
Sounds like a strange one. Perhaps worth posting on the forum over at sphinxsearch.com
I'll do a search over there--thanks! I'll probably try dumping the entire index, regenerating a new one, restart searchd, etc., and start with a clean slate before pestering them too much over there.
__________________
-= N =-
Reply With Quote
  #696  
Old 16 Jul 2009, 09:00
UK Jimbo's Avatar
UK Jimbo UK Jimbo is offline
 
Join Date: Sep 2002
Originally Posted by RedWingFan View Post
I'm willing to try anything. Although, the problem is with the indexer...unless something in searchd is somehow preventing the files from rotating properly (maybe searchd reporting that it's "busy", in other words, so the .new.* files don't get rotated in).
In my mind it's 50:50 whether it's the indexer or searchd which is causing the problem. The indexer seems to be creating the index happily with the .new file name. After that it's the job of the search daemon to rotate the new index in.

With some of those .new files there what happens if you signal a restart to searchd with (not tested but I think this is correct).
Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

That's the same method that the indexer uses to signal a rotate to searchd. I can't remember if searchd keeps a system log of it's activity. if so the restarts (and any possible problems) might be reported.

Good luck!
__________________
administrator: visordown.com
Reply With Quote
  #697  
Old 18 Jul 2009, 03:57
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
I did:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

...to stop, then restarted searchd. First time I ran the indexer, the .new.* files all were deleted. One hour and about 20 rotations later, they're still gone.

Thanks much--it worked!
__________________
-= N =-
Reply With Quote
  #698  
Old 31 Jul 2009, 11:48
UK Jimbo's Avatar
UK Jimbo UK Jimbo is offline
 
Join Date: Sep 2002
Afraid I've not trawled the whole thread for this but here's a product that makes the install of the two small plugins even easier. Use at your own risk, etc...
Attached Files
File Type: xml product-sphinx.xml (830 Bytes, 62 views)
__________________
administrator: visordown.com
Reply With Quote
  #699  
Old 05 Aug 2009, 03:51
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
Originally Posted by UK Jimbo View Post
Afraid I've not trawled the whole thread for this but here's a product that makes the install of the two small plugins even easier. Use at your own risk, etc...
Thanks! I'll try it out and let you know how it works. I'm about to pull the plug on fulltext and try Sphinx on our main (production) forum. Can't live w/o InnoDB tables, as I'm finding. I'll report back here, good or bad.

The hardest part is trying to remember what I did to get Sphinx running on our test forum. Working my way through the posts here, and my own notes...I think I'm getting it.
__________________
-= N =-
Reply With Quote
  #700  
Old 05 Aug 2009, 09:08
UK Jimbo's Avatar
UK Jimbo UK Jimbo is offline
 
Join Date: Sep 2002
Originally Posted by RedWingFan View Post
The hardest part is trying to remember what I did to get Sphinx running on our test forum. Working my way through the posts here, and my own notes...I think I'm getting it.
I'd recommend (this is true of any production roll-out):
  • Backup test site
  • Place a full copy of the live code (and db dump too if you think it's needed)
  • Go through migration/roll-out process on your test platform making notes
  • Repeat the above until you're confident of the steps you need to make
  • Repeat on production. It'll be easy as you've just done it on the test platform!
__________________
administrator: visordown.com
Reply With Quote
  #701  
Old 05 Aug 2009, 16:10
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
I have it running OK now.

Really basic (OK, stupid ) question. What do I need to turn OFF in vB to keep it from using its own internal search engine? IOW, aren't the built-in vBulletin search index tables going to be populated with the newest posts as they are made? From the way I see it, Sphinx bypasses the whole vB system, so vB shouldn't need to store any of the search terms anymore, unless another search process uses it. I've already dropped the fulltext indexes and changed the search type from fulltext back to vB internal, and am switching selected tables over to InnoDB to clear up the load issues we're having.
__________________
-= N =-
Reply With Quote
  #702  
Old 06 Aug 2009, 09:34
amcd amcd is offline
 
Join Date: Oct 2004
Originally Posted by RedWingFan View Post
I have it running OK now.

Really basic (OK, stupid ) question. What do I need to turn OFF in vB to keep it from using its own internal search engine? IOW, aren't the built-in vBulletin search index tables going to be populated with the newest posts as they are made? From the way I see it, Sphinx bypasses the whole vB system, so vB shouldn't need to store any of the search terms anymore, unless another search process uses it. I've already dropped the fulltext indexes and changed the search type from fulltext back to vB internal, and am switching selected tables over to InnoDB to clear up the load issues we're having.
No.

First set the search type back to fulltext. This will tell VB not to populate the search tables (word and another one). The onus of maintaining search data now shifts to MySQL.

Then drop the fulltext indices from post and thread tables.
__________________
eXBii.com - Indian community
no XB no fun know XB know fun !
Reply With Quote
  #703  
Old 06 Aug 2009, 15:33
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
I'll give that a try, thanks! My post and thread tables are converted to InnoDB, so they can't accept fulltext anyway.

Will doing this generate any kind of error, either from MySQL or vB? I don't think it will, but our visitors have a way of shaking out any type of rare bug or hiccup, when I least expect it.
__________________
-= N =-
Reply With Quote
  #704  
Old 06 Aug 2009, 16:58
amcd amcd is offline
 
Join Date: Oct 2004
If your tables are innodb, there will certainly be an error when you try to set search back to fulltext.
__________________
eXBii.com - Indian community
no XB no fun know XB know fun !
Reply With Quote
  #705  
Old 11 Aug 2009, 17:24
RedWingFan's Avatar
RedWingFan RedWingFan is offline
 
Join Date: Oct 2004
Just looked at this again: if I try to change it back to fulltext, it will attempt to change the InnoDB tables back to MyISAM. Don't want that!

In the settings table, "fulltextsearch" has its "value" column set to "0". There are other parameters in that row. Think I'm safe to change "value" to "1" (and rebuild the datastore cache)?

I just have to comment that since changing tables back to InnoDB and implementing Sphinx, our forum runs SO much better now! I had fulltext previously. Searching for "Steve" as a search word, it would take 35-40 seconds to get results. (Steve is our forum's owner, so his name appears in most threads.) With Sphinx, the searches come back on average around 0.4 seconds. And with the InnoDB change, we don't have stacks of queries waiting in the queue anymore.

The only hiccup I've had is that I once again had a set of *.new.* files, this time for my post indexes. I killed and restarted searchd, and reran the update, and it all rotated properly. My clue was a forum member saying he couldn't search for his posts for the past few days. Sure enough, the stale indexes were dated around the time he was unable to find his posts. I may have to run a cron job to check for any *.new.* files in that directory, and possibly put together a shell script to kill and restart searchd.
__________________
-= N =-
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 10:23.

Layout Options | Width: Wide Color: