Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Dynamically Linked Source - Speed Up Your Multiple Site Network & Upgrades Details »
Dynamically Linked Source - Speed Up Your Multiple Site Network & Upgrades
Mod Version: 1.00, by Jafo232 (Coder) Jafo232 is offline
Developer Last Online: Nov 2018 I like it Show Printable Version Email this Page

This modification is in the archives.
vB Version: 3.7.2 Rating: (11 votes - 5.00 average) Installs: 51
Released: 14 Aug 2008 Last Update: Never Downloads: 305
Not Supported Uses Plugins Additional Files  

For the purposes of this documentation, "Network" refers to a group of forums on a single server or cluster of servers.

When 3.7.1 PL2 was released, I posted over at vbulletin.com lamenting the fact that I had 15 upgrades to perform and how it would be nice to have a multiple site license to lighten the burden of downloading each install separately. In the end, I spent much more time coming up with this system.

Dynamically Linked Source or DLS allows operators of two or more forums on the same server or cluster, for all intents and purposes, to use one set of source code instead of multiple sets. By using one set of source code, Op code caching is much more efficient as only one copy of the source is cached. This greatly reduces the amount of memory used caching the source code, and also makes it more efficient. It will lead to less fragmentation for systems using APC, XCache, eAccelerator, or other such methods of caching Op code.

The method uses symbolic links to a central repository of the source code. Therefore, upgrades to all sites on the network can be done quickly by just replacing the source. For patch level releases, this means the entire network can be patched with one upload. Upgrades to a new version can be done this way, or a new repository can be created and each site can be upgraded one at a time.

The issue of licensing has also been addressed. Once the product file is installed, and assuming the installation instructions were followed, each installation will report the proper license key when salting cookies, or reporting back to Jelsoft as Vbulletin does from time to time (mostly in the AdminCP).

I contacted the development team at Jelsoft and discussed this method before releasing it, and got the OK from Mike. I was initially concerned that it may break licensing or perhaps give away too much of the license reporting methodology, but now that I look back, it really does not do either.

In tests on a rather large network that contains seven forums with about 7 million posts, hundreds of thousands of registered users, and uses three load balanced servers the load has been reduced by almost 40%. You can see in the graph attached to this thread for the particulars.

I have tried to make the readme as clear as possible, but honestly that was never one of my strong points. I cannot emphasize more that before you implement this method, TEST it first in your sandbox. Also, make sure you backup your software and database before you use this.

I also have included a PDF that explains what is happening in greater detail.

Updates:

8/20/08 - This will work with the latest version of VBSEO (3.2.0 at this time). Add all the files to the DLS except the config file, and run the php-clone.php script.

This product was developed for Andy R from Social Knowledge and released here for others to enjoy.


Please don't forget to click install as I have worked quite a while on this.

Download Now

Only licensed members can download files, Click Here for more information.

Screenshots

Click image for larger version

Name:	Web Server Cacti.jpg
Views:	881
Size:	61.2 KB
ID:	85447  

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
  #46  
Old 27 Aug 2008, 21:02
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Well, if they are not caching in your windows environment, I don't see much of a point other than speedier upgrades. Getting Linux of course would solve all of this for you. (Couldn't resist)...
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #47  
Old 27 Aug 2008, 21:14
UaECasher UaECasher is offline
 
Join Date: May 2008
This mod is great
__________________
My Sites:CasherZ Forums: All in one forums
My Mods:
Quick Soft delete and Physical delete:
Save time and reduce spam
Reply With Quote
  #48  
Old 27 Aug 2008, 22:28
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Lightbulb

This utility automates the use if fsutil on Windows Servers. It has been tested on XP Pro and Windows Server 2003, and comes with the full source for Visual Basic 6.0.

Please try it first on a test environment and make a backup before linking the files.

I will not assume any responsibility for your test and/or production environment! Use on your own risk.
Attached Files
File Type: zip vBFSUtil.zip (19.2 KB, 7 views)
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #49  
Old 27 Aug 2008, 23:22
sockwater's Avatar
sockwater sockwater is offline
 
Join Date: Apr 2008
Originally Posted by y2ksw View Post
I've tried now both systems (junction, fsutil) on a clean test environment. Both systems are not cached once, but as if they were separate files. A clean 3.7.0 installation and copy produce each about 2.55 MB of uncompressed eAccelerator cached files on forum home (index.php).
  • The junction feature requires to rename the config.php files and is a one-liner.
  • The fsutil feature replaces the single files with a hardlink to the common files folder, and allows thus to keep some of the files in their original state. Thus, renaming of config.php and other files are not necessary. fsutil requires an utility to execute recursively. Please allow me to test it a little before releasing
  • Both features require to repeat the junction/fsutil calls on each copy. It is not possible to copy and paste those links, and doing so, will just retrieve the original files.
I wonder if APC handles junctions/hardlinks on Windows the same way eAccelerator seems to-- seeing the files as the same or different files. On the other hand, maybe this could be a feature request for eAccelerator/APC, maybe something can be tweaked in their code to make them recognize that hardlinks and junctions actually refer to the same file.
Reply With Quote
  #50  
Old 28 Aug 2008, 10:34
thompson thompson is offline
 
Join Date: Jul 2007
Real name: Thomas
very interesting. will try it.
__________________
My Babies:Sport Community | Bodybuilding - Fitness - Training
Reply With Quote
  #51  
Old 04 Sep 2008, 06:21
Amenadiel's Avatar
Amenadiel Amenadiel is offline
 
Join Date: Sep 2006
this seems amazing, I'm just starting to understand its full extents.

By the way, I'm studying the integration of two boards. They are on separate webservers and I plan to have independents database servers for each of them. It seems this mod can help me combine both webservers so both boards can use one logical webserver which is the load balancing between the current ones, but won't help me integrate users and passwords since DBs will still be on separate machines, am I right?
Reply With Quote
  #52  
Old 05 Sep 2008, 00:08
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Yes, that is correct. This will really do nothing as far as integration goes.
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #53  
Old 05 Sep 2008, 17:13
993ti 993ti is offline
 
Join Date: Dec 2005
Originally Posted by Jafo232 View Post
You can link any file just by adding it to the DLS source and running the php-clone.php script.
So if 2 forums have the same products that required uploads, it's best to include those files?
__________________
Twenteracingscene - Dutch car club (VB), Aibo forum (VB)
Streetcars (VB),
Reply With Quote
  #54  
Old 08 Sep 2008, 13:42
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Yes, you can add them.
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #55  
Old 10 Sep 2008, 05:12
sdsvtdriver sdsvtdriver is offline
 
Join Date: Mar 2005
When referring to Windows, are we referring to Windows running IIS or Apache (or is it not relevant)? If Apache, would using the alias command work?

Alias "/site1/forums" "C:/htdocs/forums"
Alias "/site2/forums" "C:/htdocs/forums"
Alias "/site3/forums" "C:/htdocs/forums"

Has anyone tried NtFS link? http://sourceforge.net/project/platf...roup_id=161885

linkd? http://support.microsoft.com/kb/205524
Reply With Quote
  #56  
Old 10 Sep 2008, 21:39
sockwater's Avatar
sockwater sockwater is offline
 
Join Date: Apr 2008
NTFS Link creates hardlinks and junctions and linkd creates junctions. Both of these have been confirmed to not work. Well, they work but they are cached as if they were separate files.
Reply With Quote
  #57  
Old 11 Sep 2008, 07:02
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Yeah, I doubt there is a Windows solution that will offer any cache relief. Of course, you could always go Linux, but who wants to start that discussion!
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #58  
Old 18 Sep 2008, 18:54
sockwater's Avatar
sockwater sockwater is offline
 
Join Date: Apr 2008
Originally Posted by Jafo232 View Post
Yeah, I doubt there is a Windows solution that will offer any cache relief. Of course, you could always go Linux, but who wants to start that discussion!
This might actually be a bug in the caching code that is not recoginzing Windows hardlinks/junctions in the same way it recognizes Linux symlinks. Since I use linux, I won't be pursuing this, but for those who have Windows servers, it might be worthwhile to submit this as a bug/feature request to APC/Xcache or whatever you're using.
Reply With Quote
  #59  
Old 23 Sep 2008, 03:55
Jafo232 Jafo232 is offline
 
Join Date: May 2004
If it happens with the big three opcode cache methods, I would then think it was something native with Windows. I dunno, not a .dll head here..
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #60  
Old 23 Sep 2008, 09:43
ManagerJosh's Avatar
ManagerJosh ManagerJosh is offline
 
Join Date: Feb 2002
Just wondering, are users/members of each forum independent? Or are they linked?
__________________
Latest News:

Nothing Going On here...just hanging around vBulletin.com
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 01:11.

Layout Options | Width: Wide Color: