RSSFeed download history

[reb01501]

Where is this stored? I want to do an analysis to determine the day of the week with the least download activity for a specific site

[RogerTheDodger]

In XP:

C:\Documents and Settings\<User>\Application Data\Azureus\plugins\rssfeed\rssfeed.options

This is for RSSFeed Scanner v 1.4.2 plugin.

I also installed the plugin in Vuze on Windows 10, but I couldn't find the ".options" file.

If you want to search for rssfeed.options, I recommend "Everything Search" utility from https://www.voidtools.com

The file is part text, part binary - not easy to inspect.

[reb01501]

Thanks. I guess this won't help me. Browsing through it, I see the download history, but I see no easy way to parse it. 

It's unfortunate an "export to text" was not provided.

[FeRDNYC]

Thanks. I guess this won't help me. Browsing through it, I see the download history, but I see no easy way to parse it. 

It's unfortunate an "export to text" was not provided.

There sort of is, since RSSFeed Scanner v. 1.4.10!

(1.4.12 is pretty moldy, BTW, Dodger — the current release is 1.5.3. Changes are mostly cosmetic, thogh there is an SSL fix. At least you have the magnet hosts fix from 1.4.11, that was a big one.)

If you go to Tools > Options > Plugins > RSSFeed, there's an "Import/Export" section of the plugin options, which includes an Export button allowing you to "Export configuration in JSON format to a file named 'rssfeed.options.json' in the plugin folder".

Be warned: My 2.1MB (!!!) rssfeed.options took quite some time to export as a 4.2MB (!!!!!) rssfeed.options.json, which still isn't in a format very conducive to parsing. (In the statistical-scraping sense, I mean. JSON is a standardized format designed specifically to be parsed by code, but to get any meaningful stats out of it you'd probably have to end up parsing the entire thing into some sort of data structure, and then analyzing that to collect your stats.)

Even then you may not get the information you're looking for. I'm not entirely clear what information you're trying to collect, but you say that you want data on activity over time ­— well, just like in the plugin view, the history entries in rssfeed.options[.json] aren't timestamped. (The "Age" column in the interface shows the time of the most recent refresh, for all entries. Even that info doesn't appear to be in the exported data, though.) So, information on historical activity that you could generate stats from just doesn't appear to be stored anywhere by the plugin. There's no exported equivalent to the "Download History" tab, that I can see.

Two possibilities that I can think of, neither of which involve the plugin's data:

1. If you're looking for statistics on your download activity, the torrent file timestamps in %APPDATA%\Azureus\torrents\ would tell you when each torrent file was downloaded. That might be a starting point, and I assume it's effectively the same way RSSFeed plugin constructs its "Download history" table, by noticing when each torrent was added to Vuze.
   
2. If you're looking for statistics on the availability of torrents at the tracker site, you might get more information by simply parsing the incoming RSS feed itself, separate from the plugin, and separate from Vuze itself for that matter. All of the entries in the feed should be timestamped with a <pubDate> element that will tell you when each one became available for download.
   

[RogerTheDodger]

<snip>
1.4.12 (sic) [ You meant 1.4.2, I know :) ] is pretty moldy, BTW, Dodger — the current release is 1.5.3. Changes are mostly cosmetic, thogh there is an SSL fix. At least you have the magnet hosts fix from 1.4.11, that was a big one.
<snip>

Be warned: My 2.1MB (!!!) rssfeed.options took quite some time to export as a 4.2MB (!!!!!)
<snip>



 

Thanks for the update, FeRDNYC.

I had some major CPU usage problems with v1.5.2 of the plugin - see here:
https://forum.vuze.com/Thread-Tiny-Irrit...er-Plug-in
 
The problem was high CPU usage causing high a CPU temperature - as if it was running in a loop that took a long time to exit.

I may install v. 1.5.3 of the plugin later, but cannot say when - a bit busy at the moment.

The problem with v. 1.5.2 appeared to be linked to the size of the rssfeed.options file. With a new installation, the high CPU and temperature doesn't occur (small rssfeed.options file). So it points to the plugin processing the options file and taking a longer time the bigger it is.

My rssfeed.options file is 6,747 Kbytes - whoops!

I was going to do some tests and let "parg" know the results, but got distracted. If I can find the time I'll post here, or back in my "Tiny Irritation" thread. :)

[FeRDNYC]

<snip>
1.4.12 (sic) [ You meant 1.4.2, I know :) ] is pretty moldy, BTW, Dodger — the current release is 1.5.3. Changes are mostly cosmetic, thogh there is an SSL fix. At least you have the magnet hosts fix from 1.4.11, that was a big one.
<snip>

Ooh, no, I'd misread and clearly didn't mean 1.4.2, since that means you don't have the 1.4.11 magnet processing fix. That one's a big deal, before it was put into place magnet links in the feeds were only loaded from the first host listed, so if that host wasn't reachable or no longer existed they'd either never load or take forever to time out to the next host (I forget which).

Until the plugin was fixed to use the entire list instead of just the first entry, I was actually loading all of my feeds through a PHP script I'd written that would download the original source feed, clean up the magnet links to remove hosts I knew wouldn't be accessible, put "preferred" ones at the top of the list, and then spit back out a modified RSS feed for the scanner to parse. I was very happy to throw away all that code with the release of 1.4.11.

Be warned: My 2.1MB (!!!) rssfeed.options took quite some time to export as a 4.2MB (!!!!!)
<snip> 

I had some major CPU usage problems with v1.5.2 of the plugin - see here:
https://forum.vuze.com/Thread-Tiny-Irrit...er-Plug-in
 
The problem was high CPU usage causing high a CPU temperature - as if it was running in a loop that took a long time to exit.

I may install v. 1.5.3 of the plugin later, but cannot say when - a bit busy at the moment.

The problem with v. 1.5.2 appeared to be linked to the size of the rssfeed.options file. With a new installation, the high CPU and temperature doesn't occur (small rssfeed.options file). So it points to the plugin processing the options file and taking a longer time the bigger it is.

My rssfeed.options file is 6,747 Kbytes - whoops!

It's the history. I take back what I said in my first comment about the download history not being present in the rssfeed.options file — it clearly is, in some form (though the timestamps are definitely not trivially readable), because I just trimmed it way down.

After noticing that I had a 2.1MB (encoded!) options file AND that my Download History tab contained entries going all the way back to 2013 (!!), I manually deleted everything older than a year from that list (just selected all those entries, and there's a "Delete selected torrents" option in the context menu), and after doing that my options file is down to a much more reasonable 722KB.

It seems to have improved the performance of the entire interface, too. (My problem is memory, not CPU usage, since a recent system failure forced me to rebuild my torrent box in an extremely memory-constrained system, so much so that Vuze doesn't fit in memory and spends a ton of time swapping. It's... painful. So I've been searching for every possible memory savings I can find. This appears to be another helpful one. Looks like it shaved about 200MB off the swap cosumption, cutting it down by 1/3. That'll help a lot. Hell, I'm gonna delete another half a year's history, I really don't have any need for more than 6 months' worth... not compared to the system resources saved by losing it.)

[FeRDNYC]

Heh. After cutting Download History to only the past 4 months, rssfeed.options is down from 2.1MB to 204KB, a > 90% reduction in size.

(I also lowered the Tools > Options > Plugins > RSSFeed configurables for days of history and max items to save, to 4 and 25 respectively (from 7 and 50), so that may have helped reduce the file size as well. But it was definitely deleting over 3 years' worth of accumulated history that made the most difference.)