Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Import External Images Details »
Import External Images
Mod Version: 1.0.5, by y2ksw (Coder) y2ksw is offline
Developer Last Online: Oct 2019 I like it Show Printable Version Email this Page

vB Version: 4.x.x Rating: (38 votes - 4.68 average) Installs: 303
Released: 07 Nov 2010 Last Update: 14 Feb 2011 Downloads: 1873
Supported DB Changes Uses Plugins Additional Files Translations  

This vBulletin 4 product imports external images in your posts from external servers or paths into a specific path, normally your forum's image folder.

Differently than the previous product Image Cache for vBulletin 3.x., it parses the posts at regular intervals for new images and replaces the external references in the background. Also, instead of using only a single folder, it creates folders for years and months, making it much easier to handle large quantities of images. The source and destination image URL's are stored into database, too, avoiding unnecessary duplicates.

It moves and optionally converts images from one place to another. It is a handy tool in order to keep your images local to your forums.

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.
Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

Download Now

Only licensed members can download files, Click Here for more information.

Supporters / CoAuthors

Screenshots

Click image for larger version

Name:	08-11-2010-11.32.53.jpg
Views:	3959
Size:	112.4 KB
ID:	123480   Click image for larger version

Name:	02-02-2011-22.06.39.jpg
Views:	2744
Size:	126.0 KB
ID:	126356  

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
  #151  
Old 04 Feb 2011, 13:31
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by Vodkaholic View Post
Yea i did manually a few times the other day still did'nt cache them tho hmm
Please turn logging of that task on, and check the logs. If an image has been imported, you can check it clicking the link in the logs ... those images should be then stored locally.

After checking, disable the logging because it may flood your logs with a tremedous number of entries, depending on the number of pasted images ...

And of course, tasks are not running until forums are on.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #152  
Old 04 Feb 2011, 15:16
BirdOPrey5's Avatar
BirdOPrey5 BirdOPrey5 is offline
 
Join Date: Jun 2008
Real name: Joe D.
Can you explain more about the "cache" option in the new Beta?
__________________
-Joe
Former vBulletin.org Staff Member

(@BirdOPrey5) Former vb.org Moderator. Fighting for a free & independent vb.org.
BirdOPrey5.com - Exclusive VB Mods! (Formerly Qapla.com) | Joe's Ultimate Off Topic
Note - I do not read my PMs often, do not expect quick replies.
Reply With Quote
  #153  
Old 04 Feb 2011, 20:13
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by BirdOPrey5 View Post
Can you explain more about the "cache" option in the new Beta?
Not much more than it says in the options. If the cache option is on (default), all image references are loaded into memory by a single query (and N fetches), if it is off, every image is individually searched in the database. Thus, if you have a large forum and a huge amount of images, PHP may run out of memory and the mod stops apparently to work ("could not allocate N bytes" error). In this case you have 2 options: empty the iei_img table, or switch caching off.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #154  
Old 04 Feb 2011, 21:28
BirdOPrey5's Avatar
BirdOPrey5 BirdOPrey5 is offline
 
Join Date: Jun 2008
Real name: Joe D.
Any estimate on what you'd consider a "huge amount" of images? 100,000? 1,000.000? 10,000,000? More than that?
__________________
-Joe
Former vBulletin.org Staff Member

(@BirdOPrey5) Former vb.org Moderator. Fighting for a free & independent vb.org.
BirdOPrey5.com - Exclusive VB Mods! (Formerly Qapla.com) | Joe's Ultimate Off Topic
Note - I do not read my PMs often, do not expect quick replies.
Reply With Quote
  #155  
Old 06 Feb 2011, 10:50
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by BirdOPrey5 View Post
Any estimate on what you'd consider a "huge amount" of images? 100,000? 1,000.000? 10,000,000? More than that?
That depends on the memory you are using for each PHP process and the length of the URL's. When the plugin stops working from one day to the other, you probably have reached this limit.

On one site we have 120.000+ images and it continues to work, thus maybe that's not huge. But we also give unlimited memory to the PHP processes and the server can hold up to 16 GB. Maybe a million is huge, in this context
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #156  
Old 06 Feb 2011, 14:10
BirdOPrey5's Avatar
BirdOPrey5 BirdOPrey5 is offline
 
Join Date: Jun 2008
Real name: Joe D.
Thanks.
__________________
-Joe
Former vBulletin.org Staff Member

(@BirdOPrey5) Former vb.org Moderator. Fighting for a free & independent vb.org.
BirdOPrey5.com - Exclusive VB Mods! (Formerly Qapla.com) | Joe's Ultimate Off Topic
Note - I do not read my PMs often, do not expect quick replies.
Reply With Quote
  #157  
Old 06 Feb 2011, 21:04
BadgerDog BadgerDog is offline
 
Join Date: Oct 2006
Real name: Doug
Went live today with 1.04 ....

Crossed my fingers and opened up most of our forums to start importing 10 posts at a time from the oldest to newest ....

First few runs seemed quicker with this ordering (old to new) than the testing we did with (new to old)... also turned OFF logging after first few runs ...

Let's see how long it takes to get through our site to the latest ...

y2ksw and BirdOPrey5 ... thanks for all the help from both of you ...

Regards,
Doug

Update: Ok, I started the live execution yesterday at about 4:00 pm (same time as this original post) and checked the first few logs. Everything seemed to be working according to the logs and the earliest threads seemed to be processed. Out of curiosity to see where it was at this morning, I turned LOGGING back ON and waited until cycle for cron had past. I checked logs and there wasn't anything there at all. So, I manually ran the process and watched it (see pic attached). It's been sitting like that for almost 15 minutes, just supposedly processing 10 posts, so I suspect it's hung and I have no idea how long it hasn't been running since I started it at 4:00 pm yesterday. I also don't know if there's been any database damage or not due to the hanging, but the vBulletin error logs don't show any SQL errors and the board seems to be running fine.

I assume I should quite the processing on the screen attached and do what next?

Update to update: In checking actually file storage on the server under /imported, it appears that the last pics imported happened at about 5:05 pm, so it appears that the import mod ran just over an hour after I had originally started it before it stopped working .... and hasn't worked since.

Another update: I have tried turning OFF the cache and that didn't make any difference.

Another update: For info purposes, we are running on a dedicated private host and Safe Mode is OFF, so that shouldn't be a time out problem. y2ksw suspects the job is timing out, so I've asked him for some debug code to be built into a test version of the cron job for manual run testing, so we can see where this is failing and perhaps why.
Attached Images
File Type: jpg Screen shot 2011-02-07 at 9.00.56 AM.jpg (33.9 KB, 40 views)

Last edited by BadgerDog; 08 Feb 2011 at 11:48.
Reply With Quote
  #158  
Old 11 Feb 2011, 14:05
PhilG PhilG is offline
 
Join Date: Apr 2003
I have just tested this and it's got a serious bug.

I have set "Forum URL replacement" to http://static.domain.com and alot of images get replaced with just that domain. For example there was an image "http://www.angelfire.com/oz/gbeze/images/circuit.gif" and it got replaced with "http://static.domain.com/

Apart from that great mod.
__________________
Electronic Circuits Forum
Reply With Quote
  #159  
Old 11 Feb 2011, 14:25
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by PhilG View Post
I have just tested this and it's got a serious bug.

I have set "Forum URL replacement" to http://static.domain.com and alot of images get replaced with just that domain. For example there was an image "http://www.angelfire.com/oz/gbeze/images/circuit.gif" and it got replaced with "http://static.domain.com/

Apart from that great mod.
Is your forum path starting with http://static.domain.com ? In other words, is your forum contained in the forum replacement or vice versa?

Could you please attach a setting with which I may reproduce this behaviour?

Thanks
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #160  
Old 11 Feb 2011, 20:51
sarangan sarangan is offline
 
Join Date: Jun 2008
It would be better if there was an option to exclude forums instead of include forums. :-) Looking forward to have that option also in the feature version. A super and nice plugin. I love it.. Thanks a lot friend.. :-)
Reply With Quote
  #161  
Old 11 Feb 2011, 23:58
PhilG PhilG is offline
 
Join Date: Apr 2003
Originally Posted by y2ksw View Post
Is your forum path starting with http://static.domain.com ? In other words, is your forum contained in the forum replacement or vice versa?

Could you please attach a setting with which I may reproduce this behaviour?

Thanks
I have attached the settings.

I have the main forum setup on www.domain.com and images get served from static.domain.com.

Edit: It seems to only happen to images that are broken/missing.

Regards,
Phil
Attached Images
File Type: jpg Capture.jpg (149.1 KB, 40 views)
__________________
Electronic Circuits Forum
Reply With Quote
  #162  
Old 21 Feb 2011, 15:51
BadgerDog BadgerDog is offline
 
Join Date: Oct 2006
Real name: Doug
Update:

With thanks to y2ksw for building in some debug code to a special beta version to troubleshoot problems with this mod, we have been able to isolate the issues that are causing hangs (cron lock-ups), which we and others may be experiencing. He's pretty busy at work, so suggested that we post our findings here so that possibly programmers such as BirdOPrey5 or others, might be able to help us figure out how to fix the lock up issue.

It appears that the mod locks up when it encounters pics referred to with the IMG tag from other sites such as photobucket etc., that are 1024x768 pixels or larger. I original thought that it also might have something to do with byte size as well, since we do have some inconsiderate posters who IMG link to external pics that are 1 Mb and even up to 3 Mb in size.

However, after checking over over 100 pics that the mod failed and hung-up on, there are a lot that are reasonably sized (100k to 300k), but are over 1024x768 in pixel size, so my assumption is the hangs are caused by the handling of pixel sizes as opposed to file sizes. The 1024x768 number is a guess, purely based upon the fact that the mod seems to process pics with that pixel width and height, but we haven't been able to find ANY pic over that sizing that hasn't locked up the mod.

We've been able to manually circumvent the problem by running the cron job manually a forum at a time. Once it hangs, we examine the Cron Error Log and look at the code. It looks like this at the point it's locked up and stopped processing:

=======================================
vB4 Import External Images
2011-02-16 08:00:02 345 Get forum ID's
2011-02-16 08:00:02 349 Get image tags
2011-02-16 08:00:02 358 Adjust options
2011-02-16 08:00:02 379 Cache registered image URL's
2011-02-16 08:00:02 177 SELECT oldurl, newurl
FROM iei_img
2011-02-16 08:00:02 386 For each record with images ...
2011-02-16 08:00:02 389 Include an/or exclude specific forum sections
2011-02-16 08:00:02 409 SELECT postid, p.dateline, pagetext
FROM post p
INNER JOIN thread t ON p.threadid = t.threadid
WHERE iei_parsed=0
AND forumid IN (72)

ORDER BY postid
LIMIT 0, 10
2011-02-16 08:00:02 413 Post: 61775
2011-02-16 08:00:02 417 Find all embedded images
2011-02-16 08:00:02 420 Searching for tag: img
2011-02-16 08:00:02 430 Search for images to be ignored
2011-02-16 08:00:02 445 If we haven't got the image yet, download
2011-02-16 08:00:02 453 Pseudo-loop
2011-02-16 08:00:02 456 Get file from remote site
2011-02-16 08:00:02 285 Use CURL for: http://www.thefirearmblog.com/blog/w...compressed.jpg
2011-02-16 08:00:03 324 Content size: 788141
2011-02-16 08:00:03 464 Get file extension
2011-02-16 08:00:03 467 Make path
2011-02-16 08:00:03 471 Get file name and/or index
2011-02-16 08:00:03 139 Get only the file name (no directory info)
2011-02-16 08:00:03 142 Strip out non alpha-numeric characters
2011-02-16 08:00:03 145 Set max file name length to 40 characters
2011-02-16 08:00:03 148 Increment number if file exists
2011-02-16 08:00:03 481 See if we have valid contents
===========================================
We check the post ID, call it up on the forums and invariably, it's an IMG reference to an off-site pic that's bigger than 1024x768 pixels.

We then SAVE the pic to our local drive. Delete the reference in the post to it. Use the vBulletin advanced ATTACH javascript to upload it to the post as an attachment and SAVE the edited post.

We re-start the cron job and it now moves past this post and proceeds normally until it runs into another post with an IMG referenced pic larger than 1024x768.

Using this workaround, we've been able to work through some of our forums, but it takes 6-8 hours of manual work to do so.

So, the question is, can someone help us by looking at the code and figuring out where *and why) it's locking up on these 1024x768 pics? Perhaps it needs a pixel re-sizer function built in front of it, BEFORE it begins to import the pic?

Second problem ...

We discovered an anomaly as to file name conventions (see attached pic). It appears that in some cases the mod imports and stores the pics, but for some reason it attaches ?t=xxxxxx or ?a=xxxxx after the .jpg extension (see attached pic). In the actual post, the filename is the same way, but of course the pic doesn't show.

If I manually rename the file in both the offending post and the associated /includes directory, it fixes the problem and the pics appear in the post. Unfortunately, so far, I can't find a simple way to locate all the filenames on the server that are messed up and also find them in the actual forums so I can start manually fixing posts.

This one y2ksw says is fixable by turning ON the Always Convert Images option. Y2ksw explained why this would stop the problem from occurring in future, but not being as technical as he is, I didn't quite understand it. We're still trying to figure how to locate all the pics that have been processed already and perhaps some kind of automated handling to adjust the file names on both the server and the associated posts, so they actually appear.

Anyway, this is an excellent mod, particularly for a research site such as ours, to enable us to preserve old posts and pics for posterity. With just a bit more de-bugging and sorting out this critical hanging problem, it would be perfect....

Hope this helps other users here who may be considering using this mod for their sites, particularly larger ones with lots of pics.

Regards,
Doug
Attached Images
File Type: jpg Screen shot 2011-02-21 at 10.24.30 AM.jpg (84.0 KB, 26 views)
Reply With Quote
  #163  
Old 22 Feb 2011, 17:17
BirdOPrey5's Avatar
BirdOPrey5 BirdOPrey5 is offline
 
Join Date: Jun 2008
Real name: Joe D.
I can confirm larger images aren't importing right, such as this one:
http://www.qapla.com/ju/DSC00261.JPG

Might be a sever limitation, not sure... will see if I can figure anything out.

Can you give me a link to an image that is giving you the second problem?
__________________
-Joe
Former vBulletin.org Staff Member

(@BirdOPrey5) Former vb.org Moderator. Fighting for a free & independent vb.org.
BirdOPrey5.com - Exclusive VB Mods! (Formerly Qapla.com) | Joe's Ultimate Off Topic
Note - I do not read my PMs often, do not expect quick replies.
Reply With Quote
  #164  
Old 22 Feb 2011, 20:08
dacho's Avatar
dacho dacho is offline
 
Join Date: Dec 2004
Real name: David
I have two questions please:
1. Is it possible to import the images to sub-domains or other domain?
2. If the files are not saved in my database, can also add a watermark option on imported images?
Reply With Quote
  #165  
Old 22 Feb 2011, 21:20
BadgerDog BadgerDog is offline
 
Join Date: Oct 2006
Real name: Doug
Originally Posted by BirdOPrey5 View Post
I can confirm larger images aren't importing right, such as this one:
http://www.qapla.com/ju/DSC00261.JPG

Might be a sever limitation, not sure... will see if I can figure anything out.

Can you give me a link to an image that is giving you the second problem?
Stand by ...

I think you may be onto something ...

y2ksw sent me an email and suggested I try to increase my PHP memory variable from the current 16 MB, which he thought was maybe is a little low. He recommended to give it at least 8 MB more (on linux, this setting (memory_limit) should be at /etc/php5/php.ini). I've asked our tech support to increase it from 16MB to 32MB just in case.

Once they do that, I'll start re-testing the big pics....

Thanks for your testing to validate the issue ...

Regards,
Doug
Reply With Quote
Reply

Tags
lisansdevir


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 04:14.

Layout Options | Width: Wide Color: