Can’t pull all killmails with either ESI or zfetch

Help and support in getting the EVE-Dev Killboard to run on your site.
Post Reply
User avatar
edp
Someday Author
Posts: 95
Joined: Fri Sep 25, 2009 19:10

Can’t pull all killmails with either ESI or zfetch

Post by edp » Mon Sep 10, 2018 14:35

I’ve tried things that used to work in the past, amply setting the date stamp to a very old date, or the date of a verifiable first kill, it nothing will pull older than a month or two. Is there something I’m missing?

User avatar
Salvoxia
Developer
Posts: 1528
Joined: Wed Feb 22, 2012 12:11

Re: Can’t pull all killmails with either ESI or zfetch

Post by Salvoxia » Sat Sep 29, 2018 16:55

Hi,

sorry for my very late reply, I'm still here and not dead yet! ;)

I just tested fetching from ESI as well as from zKB on my development killboard, both worked correctly.
So there doesn't seem to be anything wrong with the mechanisms and the "other side" per se.

Do you get any kind of error? Or simply no new kills?

Best Regards,
Salvoxia

User avatar
Ariea Thellare
Noob
Posts: 8
Joined: Mon May 14, 2018 03:12

Re: Can’t pull all killmails with either ESI or zfetch

Post by Ariea Thellare » Thu Oct 04, 2018 01:27

Hey,
I also have a problem with esi kill pulling.
It seems to not pull any kills for esi, or at least very sporadically.
There seems to be a problem with one specific kill mail:

Code: Select all

#!/usr/bin/env php 2018/10/03 23:05:01 - Starting ESI Import
2018/10/03 23:05:02 - ESI SSO Key: XXXXXXXXXX (type: corp)
2018/10/03 23:05:02 - 0 kills were posted and 1000 were skipped (1000 kills fetched)
2018/10/03 23:05:02 - Last kill ID: 72274208
2018/10/03 23:05:02 - Error processing involved party. Kill-ID: 72144624, KillID = 72144624
2018/10/03 23:05:02 - Spent 0.62288904190063s talking to ESI
2018/10/03 23:05:02 - Time taken = 1.0155599117279 seconds.
The kill in question is this one: https://esi.evetech.net/latest/killmail ... 4101bc6a6/
The problem is obviously the fact this mail has no proper attackers as eg. this one has https://esi.evetech.net/latest/killmail ... a7f3d06c5/

Now I would assume if the esi fetch class encounters an error in one of the killmails it just logs it and procedes, haven't really looked at the code though. Here it does not seem to be the case, even though it say it fetched 1000 kills.
Or could it possibly be a problem with Pagination? E-Uni easily has more that 1000 kills in the last 90 days so it should fetch more than 1000 kills per fetch or not?. i saw you implement pagination in the last patch, and iirc these problems occured in since the last patch.
So far I have ignored the error by pulling from zkb, but the breaking of zkbfetch forced me to look at it :)

EDIT:
I had a look at some code. Pagination is implemented here: https://github.com/evekb/evedev-kb/blob ... h.php#L184
you are looking for the "X-Pages" header field, yet ccp returns "x-pages" at least in my rest client. And iirc php array indices are case sensitive, so there might be a problem there? I can't test that currently, but it would by my suggestion after looking over the code a little.

User avatar
Salvoxia
Developer
Posts: 1528
Joined: Wed Feb 22, 2012 12:11

Re: Can’t pull all killmails with either ESI or zfetch

Post by Salvoxia » Thu Oct 04, 2018 16:23

Hi,

thank you very much for your feedback and analysis!

It might very well be the case that pagination does not work correctly. Unfortunately, I do not have access to any entity with that number of kills, so I've been unable to test it at all.
The Swagger Interface shows it as "X-Pages", but I tried the call myself and you're right - it's "x-pages".

But - as I mentioned - I have no way to test it as well.
So if anyone is in that situation and has the opportunity to test if changing "X-Pages" into "x-pages" in the linked file does the trick, I would highly appreciate it.

Best Regards,
Salvoxia

User avatar
Ariea Thellare
Noob
Posts: 8
Joined: Mon May 14, 2018 03:12

Re: Can’t pull all killmails with either ESI or zfetch

Post by Ariea Thellare » Thu Oct 04, 2018 16:54

I can test it probably in 2-3 weeks on euni's server, right now I just have very limited access to internet. But I will let you know then unless somebody else tested it before.

User avatar
Ariea Thellare
Noob
Posts: 8
Joined: Mon May 14, 2018 03:12

Re: Can’t pull all killmails with either ESI or zfetch

Post by Ariea Thellare » Sat Oct 20, 2018 02:45

looks like the current version with capital letters is correct. so i guess that was not the problem. I guess further test is required, i will let you know how it goes.

User avatar
Salvoxia
Developer
Posts: 1528
Joined: Wed Feb 22, 2012 12:11

Re: Can’t pull all killmails with either ESI or zfetch

Post by Salvoxia » Sat Oct 20, 2018 10:54

Hi,

thanks for testing!
Then maybe I messed up the break condition for fetching. I did check the theory again, seems still correct for me.
But maybe you'll find the flaw :)

Best Regards,
Salvoxia

User avatar
Rolnick
Noob
Posts: 3
Joined: Mon May 02, 2016 10:20

Re: Can’t pull all killmails with either ESI or zfetch

Post by Rolnick » Sat Oct 20, 2018 15:53

Hi,
I have similar problem:
0 kills posted, 1000 skipped.
Error processing involved party. Kill-ID: 72757444, KillID = 72757444
Let me dig in too

Sometimes or now i get
PHP Notice: No suitable handler for query found. "select" in /usr/local/www/kb/common/includes/class.dbcachedquery.php on line 263
2018/10/20 13:56:54 - Error processing ESI API for someNoob (type: corp)
2018/10/20 13:56:54 - Error getting data: Could not resolve host: CREST_PUBLIC_URL(6)
or
PHP Notice: No suitable handler for query found. "select" in /usr/local/www/kb/common/includes/class.dbcachedquery.php on line 263
2018/10/20 13:58:57 - ESI SSO Key: someNoob (type: corp)
2018/10/20 13:58:57 - 0 kills were posted and 1000 were skipped (1000 kills fetched)
2018/10/20 13:58:57 - Last kill ID: 73013473
2018/10/20 13:58:57 - Error processing involved party. Kill-ID: 72757444, KillID = 72757444
Some crazy kill:
https://zkillboard.com/kill/72757444/

Code: Select all

2018/10/20 14:50:46 - Error processing{
    "damage_done": 0,
    "final_blow": true,
    "security_status": 0
}, KillID = 72757444
Is there a way to parse it?

User avatar
Salvoxia
Developer
Posts: 1528
Joined: Wed Feb 22, 2012 12:11

Re: Can’t pull all killmails with either ESI or zfetch

Post by Salvoxia » Sat Oct 20, 2018 21:19

Hi,

I'm afraid EDK cannot handle kills in which the victim's ship decided to go boom all by itself (i.e. without any involved party) (first error).

I have absolutely no idea why the ESI cron job should try doing anything with the old CREST_PUBLIC_URL constant, which currently is only referenced in the CrestParser class, which in turn is no longer used anywhere.. except perhaps in some mod? (second error).

Best Regards,
Salvoxia

User avatar
Rolnick
Noob
Posts: 3
Joined: Mon May 02, 2016 10:20

Re: Can’t pull all killmails with either ESI or zfetch

Post by Rolnick » Sat Oct 20, 2018 22:42

Hi,

It's not self-destruction, but he said that he was killed (two shots) by capital NPC on asteroid belt.
I will check mods

Regards
Rolnick

Post Reply

Return to “Support”

Who is online

Users browsing this forum: No registered users and 5 guests