IRC logs for #farmOS, 2022-10-21 (GMT)

2022-10-20
2022-10-22
TimeNickMessage
[20:30:01]<paul121[m]><symbioquine[m]> "https://www.drupal.org/project/..." <- Welp that issue is a bit of a dead end. This one goes a step further and suggests the duplicated resource is only ever the 48th entry
[20:30:02]<paul121[m]>https://www.drupal.org/project/jsonapi/issues/3246611
[20:33:01]<symbioquine[m]>Yeah, I noticed that it wasn't a super useful issue when I went back to look
[20:33:26]<paul121[m]>I'm able to reproduce the general issue of duplicates:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/683d5bf9d3...)
[21:20:27]<paul121[m]>But I'm not seeing this pattern of it happening every 48th entry. Starts at 751 and then every 1-5th entry after (once the 7th entry and once the 10th entry) tends to be a duplicate. My test:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/e3a2f726b6...)
[21:22:30]<paul121[m]> * But I'm not seeing this pattern of it happening every 48th entry. Starts at 751 and then every 1-5th entry after (once the 7th entry and once the 10th entry) tends to be a duplicate. My test:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/2da3f2cda9...)
[21:24:26]<paul121[m]> * But I'm not seeing this pattern of it happening every 48th entry. Starts at 751 and then every 1-5th entry after (once the 7th entry and once the 10th entry) tends to be a duplicate. My test:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/39e70fb153...)
[21:25:26]<mstenta[m]>> https://www.drupal.org/project/jsonapi/issues/3246611
[21:25:28]<mstenta[m]>worth noting that's for the old `jsonapi` module, not drupal core
[21:27:38]<paul121[m]>that issue also said this issue only appeared when sorting by their `date`, sorting by `date,id` fixed the issue. but no apparent issue w/o sort πŸ€”
[21:28:00]<mstenta[m]>> https://www.drupal.org/project/drupal/issues/3098419
[21:28:00]<mstenta[m]>might be worth reopening and changing the title to make it more specifically about dupes
[21:29:48]<mstenta[m]>(better than opening a new one IMO because it has that good first comment from Wim Leers for context/reasoning about why default sorting doesn't work...)
[21:32:13]<mstenta[m]>(might also have a better chance of a quick follow up)
[21:33:21]<mstenta[m]>* worth reopening ~and changing, * about dupes~ EDIT: on second thought maybe wait on changing title
[21:33:48]<mstenta[m]>* worth reopening <del>and changing, * about dupes</del> EDIT: on second thought maybe wait on changing title
[21:38:26]<paul121[m]>wow very different behavior when sorting by `created`:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/be9af34b64...)
[21:39:16]<mstenta[m]>is it reproducible? or are the dupes random each time?
[21:40:45]<mstenta[m]>wait are you getting dupes with sort?
[21:42:19]<mstenta[m]>oh no really?
[21:44:49]<paul121[m]>well JSON:API results are cached as best I can tell. So if run the script immediately again it is the same result (also runs much faster). But if I clear cache, yes I seem to get the same result (dupes in the same order)
[21:46:04]<mstenta[m]>i'm still shocked to hear that you get dupes WITH sort
[21:46:06]<mstenta[m]>that's bad
[21:46:47]<paul121[m]>sorting by `drupal_internal__id` seems to work fine
[21:47:09]<mstenta[m]>oh ok well at least something works
[21:52:28]<paul121[m]>as well as sort by `id`:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/d654d5b97f...)
[21:53:17]<paul121[m]>anything else to test while I'm here?
[22:02:49]<mstenta[m]>log timestamp?
[22:03:41]<mstenta[m]>(would require a bit of refactoring)
[22:04:07]<mstenta[m]>but that will be a common one
[22:04:41]<paul121[m]>kinda related issue, just looking for hints: https://www.drupal.org/project/search_api/issues/2445391
[22:04:47]<paul121[m]>> So for me it seems that you need to have an "unique" sorting mechanism attached to get a stable search result.
[22:05:04]<paul121[m]>maybe we need to sort on fields that are unique
[22:05:54]<paul121[m]>there are very likely duplicated `created` timestamps for these plot assets (they are created in bulk)
[22:05:55]<mstenta[m]>true
[22:06:35]<mstenta[m]>can't possibly expect that kind of restriction though
[22:07:00]<mstenta[m]>need to be able to get a list of logs that might have the same timestamp
[22:07:07]<mstenta[m]>and not have dupes
[22:09:02]<paul121[m]>mstenta[m]: I only have 24 logs in my local dev
[22:09:45]<mstenta[m]>oh well 🀷 i imagine it would be the same as `created`
[22:10:33]<paul121[m]>but interestingly I believe the bug report to you and I mstenta was for duplicate logs
[22:12:09]<mstenta[m]>can you point your script at rothamsted?
[22:13:18]<paul121[m]>yeah I suppose so
[22:13:22]<paul121[m]>your server ready? turn up the juice? :P
[22:14:04]<mstenta[m]>sure - curious to see
[22:14:29]<mstenta[m]>(famous last words)
[22:26:04]<paul121[m]>2k input logs
[22:29:30]<symbioquine[m]>Turn down the page size to increase the likelihood of duplication where the timestamps fall across page boundaries.
[22:32:48]<paul121[m]>mstenta: is it rate limited?
[22:34:32]<mstenta[m]>nope
[22:37:28]<paul121[m]>hmm. just going kinda slow but I get this for `timestamp`:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/28b442a9d2...)
[22:39:04]<paul121[m]> * hmm. just going kinda slow but I get this for `timestamp`:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/9e019f0414...)
[22:40:24]<symbioquine[m]>mstenta[m]: I thought you were doing some throttling: https://farmos.discourse.group/t/sensor-data-storage-limits/624/13
[22:41:11]<mstenta[m]>> going kinda slow
[22:41:11]<mstenta[m]>well jeez you don't need to insult me πŸ™„πŸ˜†
[22:42:14]<mstenta[m]>symbioquine: i do rate limit data stream posts
[22:43:32]<paul121[m]>hahah well it was a pretty consistent # seconds/page, I wasn't sure
[22:46:05]<paul121[m]><symbioquine[m]> "Turn down the page size to..." <- worse!... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/4cbc7b138d...)
[22:47:48]<paul121[m]>so maybe that is a hint, duplicates relative to page boundaries? hmm
[22:47:49]<mstenta[m]>post these results to the d.o issue and reopen it (IMO)
[22:48:05]<mstenta[m]>seems like a pretty critical issue
[22:48:36]<mstenta[m]>wonder if it would be worth pinging @e0ipso in slack to see if he has any ideas where to start
[22:48:43]<mstenta[m]>i saw an issue by him when i was searching earlier that suggested this might be complicated
[22:48:49]<paul121[m]>I've got to run but will try and add later tonight or tomorrow!
[22:50:02]<mstenta[m]>https://www.drupal.org/project/drupal/issues/2979294
[22:50:23]<mstenta[m]>(maybe unrelated to this, and i haven't read it thoroughly, just gave me the sense that there are some tricky structural considerations with pagination generally)
[22:51:49]<mstenta[m]>symbioquine: just to confirm... you found this originally outside the context of farmOS.py correct?
[22:53:23]<mstenta[m]>another related concern that comes to mind... in a site with high usage (eg: automated log creation), would this make pagination unreliable more generally?
[22:53:24]<mstenta[m]>eg: if logs are being added WHILE you are requesting pages
[22:53:35]<symbioquine[m]>mstenta[m]: yep
[22:54:26]<mstenta[m]>> another related concern that comes to mind... in a site with high usage (eg: automated log creation), would this make pagination unreliable more generally?
[22:54:26]<mstenta[m]>and if so... would a solution be to add your own "greater than" filter and bypass pagination entirely?
[22:54:27]<symbioquine[m]>mstenta[m]: yeah, I think that's a general (and unsolvable) problem with this style of pagination.
[22:54:48]<symbioquine[m]>> <@mstenta:matrix.org> > another related concern that comes to mind... in a site with high usage (eg: automated log creation), would this make pagination unreliable more generally?
[22:54:48]<symbioquine[m]>>
[22:54:49]<symbioquine[m]>> and if so... would a solution be to add your own "greater than" filter and bypass pagination entirely?
[22:54:49]<symbioquine[m]>Yes, I think that would work
[22:54:53]<mstenta[m]>maybe that's what farmOS.py should do then... enforce a sort field, and always use a greater than filter with it
[22:55:18]<symbioquine[m]>You could keep requesting first pages of 50 where the timestamp filter of each page request is derived from the last log/asset in the previous request.
[22:55:20]<mstenta[m]>which is ironic because of this statement on the official pagination docs page:
[22:55:20]<mstenta[m]>> If you take only one thing away from this guide, it should be that **you should not construct your own pagination URLs.**
[22:55:20]<mstenta[m]>https://www.drupal.org/docs/core-modules-and-themes/core-modules/jsonapi...
[22:55:25]<symbioquine[m]>hahaha
[22:55:40]<mstenta[m]>🀦
[22:56:08]<symbioquine[m]>symbioquine[m]: Of course, this also has the flaw that if the data is modified such that a log's timestamp changes between page requests the same log could occur in multiple of those pages.
[22:56:09]<mstenta[m]>paul121: actually now that i think of it I think this is what Octavio ended up doing
[22:56:14]<mstenta[m]>symbioquine: yea, can't avoid everything like that i guess
[22:56:21]<symbioquine[m]>mstenta[m]: Yeah, it's a universal problem with paginating over datasets which don't expose an overall version pointer as part of the pagination API.
[22:56:31]<mstenta[m]>yea.
[22:56:38]<mstenta[m]>and why you set up your script to run via cron when all the farmers are sleeping :-P
[22:56:52]<symbioquine[m]>symbioquine[m]: i.e. if you have a dataset which is immutable and you can paginate over a specific revision pointer for the dataset as a whole, then you can get consistent pagination πŸ€“
[22:59:23]<mstenta[m]>paul121: we should also let Had know that she may still get dupes
[23:40:41]<paul121[m]><mstenta[m]> "paul121: we should also let..." <- Oh? I think the internal ID is okay. Maybe I missed something?
[23:41:02]<mstenta[m]>oh right. she's using that?
[23:41:10]<mstenta[m]>so strange that it's ok with some fields (unique only?) but not others
[23:46:43]<symbioquine[m]>One of those problems I was talking about goes away if the field you're sorting on is unique and immutable.
[23:46:54]<mstenta[m]>"those problems" plural? what else besides dupes?
[23:47:05]<symbioquine[m]>Different ways dupes can occur. Concurrent modifications vs indeterminate sort ordering.
[23:52:31]<symbioquine[m]>If they're unique, immutable, and always increasing/decreasing in value then all the problems go away :)
[04:43:30]* jgaehring[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* ircwaves[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* lauriewayne[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* symbioquine[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* mstenta[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* EricLarese[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* skipper_is[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* scrdcow[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* EvanKelley[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* evered[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* riotmiked[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* nzsnowman[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* Mo[m]1 has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* FarmerEd[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* paul121[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* zackmuma[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:30]* lordeddi[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:31]* donblair[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:32]* calbasi[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:35]* dazinism[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:43:35]* spitz234[m] has quit (Quit: Bridge terminating on SIGTERM)
[04:48:41]* calbasi[m] has joined #farmos
[05:16:34]* spitz234[m] has joined #farmos
[05:16:34]* dazinism[m] has joined #farmos
[05:16:45]* jgaehring[m] has joined #farmos
[05:16:46]* ircwaves[m] has joined #farmos
[05:16:46]* zackmuma[m] has joined #farmos
[05:16:46]* riotmiked[m] has joined #farmos
[05:16:46]* FarmerEd[m] has joined #farmos
[05:16:46]* skipper_is[m] has joined #farmos
[05:16:46]* EricLarese[m] has joined #farmos
[05:16:46]* lauriewayne[m] has joined #farmos
[05:16:46]* paul121[m] has joined #farmos
[05:16:47]* donblair[m] has joined #farmos
[05:16:47]* EvanKelley[m] has joined #farmos
[05:16:47]* mstenta[m] has joined #farmos
[05:16:47]* lordeddi[m] has joined #farmos
[05:16:47]* evered[m] has joined #farmos
[05:16:48]* scrdcow[m] has joined #farmos
[05:16:48]* symbioquine[m] has joined #farmos
[05:16:48]* Mo[m]12 has joined #farmos
[05:16:49]* nzsnowman[m] has joined #farmos
[11:48:08]<symbioquine[m]>ACTION uploaded a video: (193KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/cdsDDZzsnLt... >
[11:57:57]<mstenta[m]>this is truly a thing of beauty
[11:58:11]<symbioquine[m]>Thanks!
[11:58:11]<mstenta[m]>when asset link blog post symbioquine ??
[11:58:24]<mstenta[m]>πŸ˜„
[11:58:33]<symbioquine[m]>Good idea, soon (now) I guess...
[11:58:59]<mstenta[m]>Need to show the QR scanning bit to really set the stage
[12:00:03]<mstenta[m]>(We need a QR module which auto generates codes for each asset AND offers bulk printing option)
[12:00:04]<FarmerEd[m]>And the camera image importing
[12:00:36]<mstenta[m]>Then you just print em, cut em out, stick em on!
[12:00:48]<mstenta[m]>Put some postage tape over it for weather proofing - bam! good for a few seasons in most places!
[12:00:53]<symbioquine[m]>mstenta[m]: Yeah, I had that built into Rabbit Link (the inspiration for Asset Link)
[12:01:35]<FarmerEd[m]>mstenta[m]: Not here
[12:02:03]<mstenta[m]>Haha well maybe I'm thinking inside a barn where the food bin is kept... loosely indoors
[12:02:21]<symbioquine[m]>symbioquine[m]: Needed functionality to filter though. I ended up using the dev tools to remove the codes I'd already printed from the pages.
[12:02:49]<mstenta[m]>I'd put one on my chicken food bin, and one inside the chicken coop door
[12:03:02]<symbioquine[m]>I use this stuff: https://www.amazon.com/dp/B00NWVGOF4/
[12:03:42]<FarmerEd[m]>mstenta[m]: Even that can be a challenge sometimes.
[12:03:44]<symbioquine[m]>It's not cheap, but its still going strong outdoors ~2 years later (in the sun/rain just stapled to some cedar scrap wood which is drilled/wired on to the fence)
[12:04:10]<symbioquine[m]>I'll take a picture of one in a bit so show...
[12:05:04]<mstenta[m]>The other idea I had a while ago was to see if you could buy plastic QR codes in bulk... just randomly generated UUIDs... and then offer the ability to assign it to an asset in farmOS (so the opposite of generating it in farmOS)
[12:05:23]<FarmerEd[m]>Cool, could be worth a try.... It's either that or I'd need a CNC ingraver
[12:05:39]<mstenta[m]>Someone's gotta be producing weatherproof random UUID QRs right?
[12:06:01]<mstenta[m]>Feels like something pretty easy to mass produce
[12:06:15]<FarmerEd[m]>mstenta[m]: Or they should
[12:06:20]<mstenta[m]>Just get a big box full of em
[12:06:36]<mstenta[m]>Scan one to assign it to an asset, then glue/screw it to something
[12:06:56]<mstenta[m]>Who's feeling entrepreneurial?
[12:11:38]<symbioquine[m]>I've wanted to create/use functionality like that to allow re-using/upcycling QR codes from things like milk jugs that sometimes have unique batch numbers printed on them as QR codes.
[12:11:57]<mstenta[m]>That would be cool!
[12:15:29]<FarmerEd[m]>Collecting QR codes to reuse off of products reminds me of a shortlived games console from the 90's
[12:16:34]<FarmerEd[m]>Only it was barcodes
[12:17:58]<symbioquine[m]>ACTION uploaded an image: (463KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/deevDptkGFq... >
[12:18:46]<symbioquine[m]>The QR code itself has been replaced with a dummy version for privacy/security reasons, but you can get an idea of how well the text has weathered from the "L8" part.
[12:18:46]<mstenta[m]>Remember this thing? https://pluralistic.net/2022/10/20/benevolent-dictators/#felony-contempt...
[12:18:46]<FarmerEd[m]>Is that your waterproof paper?
[12:19:02]<symbioquine[m]>FarmerEd[m]: Yeah
[12:19:04]<mstenta[m]>Not bad symbioquine !
[12:19:19]<symbioquine[m]>That's after ~2 years outside south-facing in a field
[12:19:55]<symbioquine[m]>Printed with a standard laser printer
[12:21:27]<FarmerEd[m]>Are you in a typically wet part of the world?
[12:22:22]<FarmerEd[m]>symbioquine[m]: Would need to be, can't imagine inkjet lasting.
[12:24:49]<symbioquine[m]>FarmerEd[m]: ~73 cm of rain per year on average
[12:24:57]<mstenta[m]>paul121: not THAT is a bug report https://www.drupal.org/project/drupal/issues/3098419#comment-14758465
[12:25:01]<mstenta[m]>s/not/now/
[12:27:30]<mstenta[m]>I am so curious to understand why those duplicates occur... very tempted to put on my Drupal core snorkel 🀿
[12:30:18]<paul121[m]>haha yeah was just coming to post that update. had to jump on a call right as I was wrapping up so I got rushed at the end
[12:30:42]<paul121[m]>going to credit you two 🀝
[14:01:40]<mstenta[m]>Huh. Well according to @bbrala in Slack, something simple to test is commenting out this line: https://git.drupalcode.org/project/drupal/-/blob/9.4.x/core/modules/json...
[14:03:05]<mstenta[m]>That would tell us if `TemporaryQueryGuard` is to blame, which was @bbrala's hunch
[14:08:55]<paul121[m]>oh interesting. well I can try real quick...
[14:10:35]<mstenta[m]>yea! sounds like it would be an easy thing to test before digging any deeper
[14:17:19]<paul121[m]>womp. duplicate at the same spot, 751... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/d37b5e45d5...)
[14:18:01]<mstenta[m]>alright well good to know it's not that... that class looked kinda scary haha
[14:18:46]<paul121[m]>there is really something to be said about sorting by values that are not unique. for example, plots by `created`:... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/2990b4aeb3...)
[14:19:08]<paul121[m]>not the first page! but the second page, second entry
[14:19:45]<mstenta[m]>can you elaborate? what specifically are you seeing?
[14:20:04]<mstenta[m]>(grokking that code, but if you have it summarized in your head...)
[14:20:39]<mstenta[m]>happens a lot when logs have the same created time?
[14:20:46]<mstenta[m]>(assets i mean)
[14:21:17]<paul121[m]>Yes, it seems
[14:22:33]<paul121[m]>Also, interesting that it was the second entry on second page (using default page size 50 here). In this core code they query N+1 (51) to see if there is a next page..... so that 52nd entry wasn't included in the first query...
[14:22:36]<paul121[m]>https://git.drupalcode.org/project/drupal/-/blob/9.4.x/core/modules/json...
[14:22:41]<symbioquine[m]>I believe you could add a secondary sorting parameter like `drupal_internal__id` to avoid that problem
[14:22:52]<symbioquine[m]>It just needs to make the sorting consistent right
[14:23:07]<mstenta[m]>Mm that makes sense symbioquine
[14:23:11]<symbioquine[m]>https://www.drupal.org/docs/core-modules-and-themes/core-modules/jsonapi...
[14:24:14]<paul121[m]>symbioquine[m]: good idea, that works :-)... (full message at <https://libera.ems.host/_matrix/media/r0/download/libera.chat/4cc2b20d6b...)
[14:25:30]<paul121[m]>I gtg entertain puppy, unfortunately JSON:API pagination is not one of his interests
[14:25:50]<symbioquine[m]>paul121[m]: understandably :)
[14:27:55]<mstenta[m]>why not?? 🐢
[14:28:30]<symbioquine[m]>insufficient squirrels and butterflies involved?
[14:32:45]<FarmerEd[m]>ACTION uploaded an image: (35KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/DeOLnFNwDpv... >
[14:34:59]* BjrnBrala[m] has joined #farmos
[14:35:15]<BjrnBrala[m]>just to have this ready should i need to check here or ping ;)
[14:36:32]<mstenta[m]>Hello BjΓΆrn Brala !
[14:36:40]<BjrnBrala[m]>^^
[15:00:46]<mstenta[m]>Curious all your thoughts on this: https://www.drupal.org/project/farm/issues/3316925
[15:01:04]<mstenta[m]>Just thought of it while investigating JSON:API stuff, so whipped up a quick issue and did a quick skim of our classes
[15:42:20]<paul121[m]>ACTION uploaded an image: (110KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/xWvJbhPnwjw... >
[15:42:44]<mstenta[m]>drupup!
[15:42:53]<paul121[m]><paul121[m]> "I gtg entertain puppy, unfortuna..." <- We'll work our way towards that...
[15:48:03]<FarmerEd[m]>ACTION uploaded an image: (117KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/ZwYXxgODbyG... >
[15:48:32]<symbioquine[m]>^ "After a long day programming"
[15:59:18]<FarmerEd[m]>Think she's done for the night.