Making URLfilter easier on FLASH installations.

[Davesworld]

As a long time user of URLfilter as well as a big fan of Raqcop or IPCop FLASH installations I never liked how the urlfilter definitions update thrashed the flash drive for nearly an hour as it updates the databases for each definition category. This made me think, since the /tmp directory on flash installs is in a ram disk rather than on the physical flash drive and the download and database updates are done in a separate folder than the actual /var/ipcop/urlfilter/blacklists so it made sense to have all the thrashing done in a ramdisk (memory) and then copied to blacklists AFTER all the heavy work was finished.

I found that by editing /var/ipcop/urlfilter/bin/autoupdate.pl line 14 to:

my $target = "/tmp/download";

It takes just under 5 minutes to update the databases and then copy the new blacklists folder to /var/ipcop/urlfilter and you do not see your delicate flash drive being pounded for 45 minutes or so. I now allow mine to autoupdate daily but may back that down to a week or longer depending if I see many changes in the course of a week. I use Shalla Services as my choice.

A while back I did email Marco about using /tmp in IPCop 2 since ALL installations of IPCop 2, flash or disk, mount /tmp using TMPFS so ram would be used for the heavy lifting during definitions updates as I have done with a Raqcop flash install. I believe he is at least considering it but I'm sure there are greater priorities at the moment.

[Davesworld]

Well, now that I am testing builds of what will become Raqcop 2.1.0 in sync with the IPCop svn tree, urlfilter is working as of now.

I was successful in using /tmp as the place where the blacklists are downloaded and databases built. This time the file is /usr/local/bin/blacklistupdate.pl. I had the entire shallalist updated in five minutes and no hard drive activity until it copied the built lists to their destination.

Keep in mind that I have 500MB of space available in TMPFS which is 50% of the 1GB of ram on the system. In 2.x.x all systems have /tmp mounted using TMPFS which is basically a ramdisk.

[weizen_42]

Not sure how large the various Blacklists are, but I'd expect that you need > 128 MiB available space on /tmp

[Davesworld]

Probably so. I saw the commit and I think checking space in /tmp first before using the drive is the best solution that should work in all cases. I'll run it on something here shortly. I was pretty excited when I came home from work and saw the commit.

[Davesworld]

Yup, with 256MB of ram /tmp actually shows as 126M exactly. When trying to fudge the blacklist perl program to accept 125M, the blacklist gets jammed up during the database for porn and eventually the non updated databases as well as the updated ones get copied to the directory. The porn one is the biggest database to build using Shalla.

I took one of the 128MB sticks out and threw a 500MB in there for a mixed batch and with over 256MB in /tmp it worked beautifully. My recommendation is to run at least 500MB of ram on flash installs. I haven't tried  384MB yet on either drive or flash yet. With 256MB of ram the program will not allow it to try on /tmp since it shows as less than 128MB on /tmp in such a scenario.  Since you need at least 384MB to give you a /tmp of slightly less than 192MB I think it will work exactly as you modified it.

2.1.0 looks to be slated for April at the earliest is my guess since my svn build xml file shows 4 at the beginning of the non complete date. If so there'll still be plenty of testing.