Most efficient way to pull all kill history

[Plain213]

Running into a number of issues when trying to pull kill history through zkb feed and ESI.

Code: Select all

2019/08/04 21:14:48 - Error processing involved party. Kill-ID: 78024979, KillID = 78024979
2019/08/04 21:14:48 - Error processing involved party. Kill-ID: 77794876, KillID = 77794876
2019/08/04 21:14:48 - Error processing involved party. Kill-ID: 77834885, KillID = 77834885
2019/08/04 21:14:48 - Error processing involved party. Kill-ID: 77865571, KillID = 77865571
2019/08/04 21:14:48 - Error processing involved party. Kill-ID: 77567938, KillID = 77567938
2019/08/04 21:14:48 - Stopped fetching before finding new kills due to safety limit (fetched 1400 kills in a row!). Try lowering your negative kill timestamp offset!

&

Code: Select all

is route is deprecated
[WARNING] Endpoint https://esi.evetech.net/v2/universe/names/ warning: 299 - This route is deprecated

In terms of Corp ESI I now get 0 kills posed and 1000 skipped.

Any suggestions?

[Salvoxia]

Hi,

first of all, do you have all the hotfixes from the "Known issues" section at the bottom of this post applied to your installation?

Second, the kills with error while processing involved parties: These are losses without an attacker. This should not happen and is a bug on CCPs side, and unfortunately EDK cannot handle such kills.

You do not have to worry about the deprecation warning, this won't affect your ability to fetch kills.

Unfortunately, ESI's kill history only goes back 90 days (it used to offer access to the full kill history, but not anymore). So if all kills from ESI are skipped, it means that they're already posted.

The best way to get your complete kill history is the way you most likely attempted: Using zKBFetch while setting the last kill date to your entity's creation date. But you have to have the hotfixes applied for zKBFetch to work properly right now.
Due to changes in zKB's API, zKBFetch can only fetch one day worth of kills each cycle. So if your Entity is 1 year old, you'll need at least 365 fetch cycles to make sure to get all kills.
If you already have the hotfixes and still get the error regarding the safety limit, could you please post your zKB fetch URL, the list kill timestamp you are using, as well as your negative time offset and NPC kill setting? Then I can try to analyze what's going wrong.

Best Regards,
Salvoxia

[Plain213]

Thanks,

I've just updated to the latest version as per your post so i'll start running the crons and see if it populates itself

Plain

[Plain213]

Keep getting the following error after every few runs.

Code: Select all

PHP Warning:  Module 'zip' already loaded in Unknown on line 0
2019/08/07 19:18:58 - Starting zKB Import
PHP Notice:  No suitable handler for query found. "select" in /var/www/kb/common/includes/class.dbcachedquery.php on line 263
PHP Fatal error:  Uncaught Error: Call to a member function getName() on boolean in /var/www/kb/common/includes/esi/class.helpers.php:47
Stack trace:
#0 /var/www/kb/common/includes/class.item.php(388): ESI_Helpers::getTypeNameById(25783, true)
#1 /var/www/kb/common/includes/class.esiparser.php(722): Item::getByID(25783)
#2 /var/www/kb/common/includes/class.esiparser.php(706): EsiParser->processItem(Object(Swagger\Client\Model\GetKillmailsKillmailIdKillmailHashOkVictimItems1), Object(Kill))
#3 /var/www/kb/common/includes/class.esiparser.php(213): EsiParser->processItems(Object(Kill))
#4 /var/www/kb/common/includes/class.zkbfetch.php(568): EsiParser->parse()
#5 /var/www/kb/common/includes/class.zkbfetch.php(464): ZKBFetch->processKill(Object(stdClass))
#6 /var/www/kb/cron/cron_zkb.php(56): ZKBFetch->processApi()
#7 /var/www/kb/cron/cron_zkb.php(29): getZKBApi(Object(ZKBFetch))
#8 {main}
  thrown in /var/www/kb/common/includes/esi/class.helpers.php on line 47

Fetch will then not continue

[Salvoxia]

Hi,

that's very interesting.
Basically, there's an item in the killmail which is unknown to the static data export from CCP, as well as to ESI. However, EDK has a fallback mechanism to cope with such items.
Could you please check in Admin -> Cache if you have Query Caching enabled (either File-, MemCache- or Redis-based)? If so, could you please check if disabling that caching solves this problem for you?
The fallback mechanism not catching this might have something to do with the PHP Notice logged right before the fatal error. And that seems to be generated by caching.
If that solves it for you, I'll take a closer look to identify the root cause.

Best Regards,
Salvoxia