Can’t pull all killmails with either ESI or zfetch

[edp]

I’ve tried things that used to work in the past, amply setting the date stamp to a very old date, or the date of a verifiable first kill, it nothing will pull older than a month or two. Is there something I’m missing?

[Salvoxia]

Hi,

sorry for my very late reply, I'm still here and not dead yet!

I just tested fetching from ESI as well as from zKB on my development killboard, both worked correctly.
So there doesn't seem to be anything wrong with the mechanisms and the "other side" per se.

Do you get any kind of error? Or simply no new kills?

Best Regards,
Salvoxia

[Ariea Thellare]

Hey,
I also have a problem with esi kill pulling.
It seems to not pull any kills for esi, or at least very sporadically.
There seems to be a problem with one specific kill mail:

Code: Select all

#!/usr/bin/env php 2018/10/03 23:05:01 - Starting ESI Import
2018/10/03 23:05:02 - ESI SSO Key: XXXXXXXXXX (type: corp)
2018/10/03 23:05:02 - 0 kills were posted and 1000 were skipped (1000 kills fetched)
2018/10/03 23:05:02 - Last kill ID: 72274208
2018/10/03 23:05:02 - Error processing involved party. Kill-ID: 72144624, KillID = 72144624
2018/10/03 23:05:02 - Spent 0.62288904190063s talking to ESI
2018/10/03 23:05:02 - Time taken = 1.0155599117279 seconds.

The kill in question is this one: https://esi.evetech.net/latest/killmail ... 4101bc6a6/
The problem is obviously the fact this mail has no proper attackers as eg. this one has https://esi.evetech.net/latest/killmail ... a7f3d06c5/

Now I would assume if the esi fetch class encounters an error in one of the killmails it just logs it and procedes, haven't really looked at the code though. Here it does not seem to be the case, even though it say it fetched 1000 kills.
Or could it possibly be a problem with Pagination? E-Uni easily has more that 1000 kills in the last 90 days so it should fetch more than 1000 kills per fetch or not?. i saw you implement pagination in the last patch, and iirc these problems occured in since the last patch.
So far I have ignored the error by pulling from zkb, but the breaking of zkbfetch forced me to look at it

EDIT:
I had a look at some code. Pagination is implemented here: https://github.com/evekb/evedev-kb/blob ... h.php#L184
you are looking for the "X-Pages" header field, yet ccp returns "x-pages" at least in my rest client. And iirc php array indices are case sensitive, so there might be a problem there? I can't test that currently, but it would by my suggestion after looking over the code a little.

[Salvoxia]

Hi,

thank you very much for your feedback and analysis!

It might very well be the case that pagination does not work correctly. Unfortunately, I do not have access to any entity with that number of kills, so I've been unable to test it at all.
The Swagger Interface shows it as "X-Pages", but I tried the call myself and you're right - it's "x-pages".

But - as I mentioned - I have no way to test it as well.
So if anyone is in that situation and has the opportunity to test if changing "X-Pages" into "x-pages" in the linked file does the trick, I would highly appreciate it.

Best Regards,
Salvoxia

[Ariea Thellare]

I can test it probably in 2-3 weeks on euni's server, right now I just have very limited access to internet. But I will let you know then unless somebody else tested it before.

[Ariea Thellare]

looks like the current version with capital letters is correct. so i guess that was not the problem. I guess further test is required, i will let you know how it goes.

[Salvoxia]

Hi,

thanks for testing!
Then maybe I messed up the break condition for fetching. I did check the theory again, seems still correct for me.
But maybe you'll find the flaw

Best Regards,
Salvoxia

[Rolnick]

Hi,
I have similar problem:

0 kills posted, 1000 skipped.
Error processing involved party. Kill-ID: 72757444, KillID = 72757444

Let me dig in too

Sometimes or now i get

PHP Notice: No suitable handler for query found. "select" in /usr/local/www/kb/common/includes/class.dbcachedquery.php on line 263
2018/10/20 13:56:54 - Error processing ESI API for someNoob (type: corp)
2018/10/20 13:56:54 - Error getting data: Could not resolve host: CREST_PUBLIC_URL(6)

or

PHP Notice: No suitable handler for query found. "select" in /usr/local/www/kb/common/includes/class.dbcachedquery.php on line 263
2018/10/20 13:58:57 - ESI SSO Key: someNoob (type: corp)
2018/10/20 13:58:57 - 0 kills were posted and 1000 were skipped (1000 kills fetched)
2018/10/20 13:58:57 - Last kill ID: 73013473
2018/10/20 13:58:57 - Error processing involved party. Kill-ID: 72757444, KillID = 72757444

Some crazy kill:
https://zkillboard.com/kill/72757444/

Code: Select all

2018/10/20 14:50:46 - Error processing{
    "damage_done": 0,
    "final_blow": true,
    "security_status": 0
}, KillID = 72757444

Is there a way to parse it?

[Salvoxia]

Hi,

I'm afraid EDK cannot handle kills in which the victim's ship decided to go boom all by itself (i.e. without any involved party) (first error).

I have absolutely no idea why the ESI cron job should try doing anything with the old CREST_PUBLIC_URL constant, which currently is only referenced in the CrestParser class, which in turn is no longer used anywhere.. except perhaps in some mod? (second error).

Best Regards,
Salvoxia

[Rolnick]

Hi,

It's not self-destruction, but he said that he was killed (two shots) by capital NPC on asteroid belt.
I will check mods

Regards
Rolnick