[EDK4] Cronjob update all prices from eve-central

[prometh]

Hi there,

It seems like, there is currently no source for price stats we can use with EDK so i decided to write a script that does fetch all prices from eve-central and imports them.

This is work in Progress! Maybe someone that actually knows how to write php can take this as an inspiration.

This script is intended to be run maybe once a week beacause the execution time is roughly 12 minutes, so also nothing any free hoster will allow you to do.
It uses cURL and there is no fall-back.
Since i did want to use the original classes for importing prices it will generate a xml file in cache that uses the same formatting as compared to the no-ip file we are using by default.

What does the script actually do and why does it take so long?

Well first of all it will query the KB Database to get all typeIDs from kb3_invtypes that have a marketGroup (Thanks Salvoxia). Then it splits the list into chunks of 25 typeids and uses those to query prices from eve-central (25 at once) right know only from the forge (regionid can be changed multiples can be used but its hard coded right now). Whenever one of the 25 does not exist, eve-central will give an error message and not the values for the other ones, in this case i do 25 separate calls to figure out which one causes the problem. Theres a 1 second waiting time in between calls not to hammer the eve central server to much. After all that is done, everything is written to a file marketdata<date>.xml in the cache folder and from there it's being imported using the built-in fetch_value functionality, so the last line will be something linke xxxx values updates.

Todo:

Done... 1. write into an appropriate logfile
Not needed anymore... 2. Provide something like a blacklist (items that do exist in the EDK database but not on eve central) or a whitelist coming frome somewhere alse but the EDK database.
3. Figure out a clever way to get statistics instead of snapshots
(4. maybe host the result on my server to be used by everyone)

Credit goes to the development team of EDK as most of the code is already used in other cronjobs and Fuzzysteve (who documented the API usage in PHP fuzzwork.co.uk).

V 0.1
- Initial release

V 0.2
- Changed to file marketdata<date>.xml in the cache folder (maybe it will be good for something...)
- Added logfile marketdata<date>.log in the cache folder
- now using kb3_invtypes and filtering for MarketGroup -> Execution time down to around 12 minutes

V 0.3
- Fixed it under windows
- Adds a symbolic link to the latest xml called just "marketdata.xml" e.g. to share it
- Keeping only the last 7 days of individual xml files (to save disk space, each one is 1 MB)

V 0.4
- Added an option to use the 5% percentile price instead of the median, which gives more reasonable prices, especially for faction hulls and mods
- Added an option to use not just the Forge but multiple regions (takes slightly longer ~ +3 minutes).

Cheers,
prometh

[Salvoxia]

Hi,

wow, thank you for your work! It really makes me happy to see community contributions!

It's quite a coincidence that I just pushed a new item value fetcher to the repo (scheduled for release in the Phoebe Update on Tuesday). This fetcher is designed to query the CREST endpoint for average item prices (https://public-crest.eveonline.com/market/prices/) which returns the same prices used ingame for displaying item values.
Of course prices from eve-central are more up-to-date and even more accurate if you like Jita Prices, CREST data is more complete (and the run time is < 1second ).

So now one can choose which fetcher to use!

Regards,
Salvoxia

[prometh]

Not really I guess the changes you pushed will break the script I wrote

Edit: Nvm just saw that you're using a new class file...

I guess I'll just incorporate whats needed from the old value fetcher. Anyway the crest endpoint seems to be the right place to get marketable typeIDs. Btw. What format is the output, JSON? I was thinking about using the static db dump from ccp but that requires yaml for php which I don't think is very common...

[Salvoxia]

Hi,

yes, CREST outputs JSON.
For using the SDE with PHP I suggest getting a MySQL conversion (EDK database is based on Fuzzysteve's conversion: https://www.fuzzwork.co.uk/dump/). But if all you need are typeIDs that are available in the market take a look at the killboard's kb3_invtypes table. All items with marketGroupID != NULL should have a price in the ingame market.

Regards,
Salvoxia

[prometh]

V 0.2 Uploaded.

Thanks Salvoxia, to querry kb3_invtypes did the trick. Now theres only very few items left that can not be querried from eve-central:

Code: Select all

20:42:32 EVE Central price fetcher started...
typeID 367477 can not be querried.
typeID 367489 can not be querried.
typeID 367541 can not be querried.
20:54:41 8052 Items updated

Guess that's some recently added Dust stuff.
...and the Execution time is down to about 12 minutes so it might be something that can be used as a cron job maybe once a day.

For those who want to give it a try, you can find the daily dump (temporarly) here:

http://brgf.de/prices/marketdata.xml

Cheers,
prometh

[Salvoxia]

Gave it a try and worked pretty well, for one exception:

I tested it on a windows machine as well and after fetching the price data it threw an exception because of the directory separators (slashes) in the path to the XML file.
The cronjob already sets the include_path depnding on the OS, perhaps you could set yourself the correct directory separator there and make it work for windows machines as well?

Regards,
Salvoxia

[prometh]

Thanks i'll have a look, unfirtunately i can only test it under ubuntu...

[nefrit]

http://killboard.flame-bridge.ru/cron/cron_evec.php

504 Gateway Time-out

The server didn't respond in time.

[prometh]

Sorry about that, but the script takes quite long and if your php max execution time does not allow this it won't work.

If you have access to it you can try to increase or disable the limit usually in /etc/php5/cli/php.ini when executing it as a cron job or /etc/php5/apache2/php.ini when calling it in a browser. Otherwise you can at least use the prices using the link to the xml file for the import in the values section of the admin panel.

[prometh]

Btw. Salvoxia do you know which line causes the problem under windows? Was the xml written? I think the problem should be the line that uses file://... to import it? Thanks.