| [09:44:31] | <mstenta[m]> | Thanks for sharing edbob! |
| [09:45:27] | <mstenta[m]> | I think it's a great exercise to learn the farmOS data model by "mirroring" it in another system. |
| [09:46:12] | <mstenta[m]> | This is something we've talked about before... thinking about the "farmOS data model" as an independent piece from the "farmOS software" |
| [09:46:18] | <mstenta[m]> | And the current implementation (in Drupal) is the "reference implementation" |
| [09:46:59] | <mstenta[m]> | Different systems can benefit from the data model, while focusing on the things that matter to them. |
| [09:47:49] | <symbioquine[m]> | The biggest flaw IMHO in the current reference implementation (not farmOS' fault, but core Drupal's) is the weird lack of transactions when committing database changes. |
| [09:48:36] | <symbioquine[m]> | I mean flaw, relative to the ideal of what the data model could be. |
| [09:49:23] | <symbioquine[m]> | Though I personally would prefer more things to be immutable and function as a ledger of sorts - which could obviate many of the race conditions that the lack of useful commit transactions creates. |
| [09:50:38] | <symbioquine[m]> | All that said, I'm not sure if it's likely to be a real problem or just a theoretical one so that's why I haven't tried to work on it. |
| [09:52:56] | <mstenta[m]> | > this can be thought of as "farmOS lite" in a way... (full message at <https://matrix.org/oftc/media/v1/media/download/AV_j3hkqq3e4uIOBhW3fMiAp...) |
| [09:53:31] | <mstenta[m]> | I think the power of a shared data model is best leveraged when the systems built for it compliment each other, and solve unique/distinct problems. |
| [09:53:49] | <mstenta[m]> | So that's what I would be most excited to see. :-) |
| [09:57:32] | <mstenta[m]> | > I have found a couple of limitations (IIUC) with the API which prevent the "wrapper" mode from being *quite* as good |
| [09:57:32] | <mstenta[m]> | edbob I'd love to hear what you're finding too! |
| [10:01:44] | <mstenta[m]> | edbob: Also curious if you have decided how to deal with farmOS's modularity. eg: If someone installs a module in their farmOS server which extends the data model, does this get replicated? Or are you targeting a specific module set? |
| [11:45:22] | <edbob[m]> | hey thanks for the comments! i'll hop on today's call, maybe discuss some of the points raised. and/or i'll add more thoughts here afterward |
| [11:46:10] | <mstenta[m]> | Cool! |
| [11:46:36] | <mstenta[m]> | Good seque... |
| [11:47:02] | <mstenta[m]> | @room: The farmOS weekly dev call starts in ~13 minutes. All are welcome! https://meet.jit.si/farmos-dev |
| [11:47:34] | <mstenta[m]> | symbioquine: if we have some time maybe we can talk about SVGs :-) |
| [11:47:59] | <mstenta[m]> | I'm also working on https://www.drupal.org/project/farm/issues/2513056 if anyone's interested in seeing that |
| [16:09:38] | <edbob[m]> | <mstenta[m]> "> I have found a couple of..." <- we touched on this in the call but for sake of others: the main problem is around pagination. API wants to give me one page at a time but not tell me how many there are. as of now this means i can only (reasonably) pull a single page of 50 results, with no way for user to get "more" - admittedly this is partly "my" problem since framework is built around the notion of showing e.g. "1 - 20 of |
| [16:09:38] | <edbob[m]> | 500 results" and pagination link/buttons to navigate. (which is also what farmOS itself provides.) it could be worked around by a "doom scroll" type UI but not sure if i'll bother since i mostly am focused on the "mirror" path. that lets me access "native" app DB via SQL and gets me my typical pagination feature |
| [16:18:19] | <mstenta[m]> | edbob: Thanks for reporting back for folks in chat! This reminds me of another idea I had while we were talking, which I forgot to say... |
| [16:18:37] | <edbob[m]> | <mstenta[m]> "> this can be thought of as "..." <- yes i am trying to be mindful of this, avoid competing for resources. i have 2 hopes though, one is that mostly it's just my own effort for a while, so ideally not steal any "other" resources per se. and the 2nd is that when i've done enough to really "debut" things, barriers will be low enough that some may *want* to play around because it's more fun than Drupal :) ha we'll see, but |
| [16:18:37] | <edbob[m]> | anyway i want to stay in my lane and please let me know when i'm pushing too hard etc. |
| [16:18:37] | <edbob[m]> | i wish i already had a "killer feature" to offer, but so far it's mostly just a "jailbreak to python" and from there anything else could be tacked on. my suspicion/hope is that this makes customization more accessible to some folks, but that will require building some more wrapper libs maybe to make it even easier |
| [16:19:57] | <mstenta[m]> | In the context of the "API only" approach, one idea might be to just run multiple requests to get the full list ahead of time, and then build the UI accordingly. I don't know if it's a good idea, though. 😆 |
| [16:20:36] | <mstenta[m]> | I assume the API only approach is quering farmOS on the server side via Python? |
| [16:21:24] | <mstenta[m]> | There is a way to say "only give me the entity title" (or maybe uuid would be better)... so then you could paginate through all the results, build a temporary set so you know how many there are, render your pages/tables accordingly, and then query for the specific items to get more information. |
| [16:21:30] | <mstenta[m]> | https://drupal.stackexchange.com/questions/269590/jsonapi-query-to-retur... |
| [16:21:35] | <edbob[m]> | yes querying API on server side w/ python |
| [16:21:54] | <mstenta[m]> | Just an idea! Definitely not saying it's the best way to do this! But if you wanted to maintain an API-only option, that's one way to do it perhaps. |
| [16:23:24] | <mstenta[m]> | Ultimately it feels like the mirror approach is all around going to be easier to work with, though. |
| [16:23:50] | <edbob[m]> | that's interesting..i'll give it some thought at least. would it mean making e.g. 50 queries to fill in each record when showing a page though? |
| [16:23:53] | <mstenta[m]> | One way to think about this is as a cache. |
| [16:24:35] | <mstenta[m]> | Which then begs the question... where do we store the cache? And when are we recomputing it? |
| [16:24:49] | <edbob[m]> | probably a cache would make more sense, possibly even use DB table and sort of "import on the fly" but not expose CRUD for those tables of course |
| [16:25:01] | <mstenta[m]> | And how do we deal with changes that may happen on the server from another user, while you are working with an older copy. |
| [16:25:24] | <mstenta[m]> | (Maybe these problems don't matter in your case, but thoughts to consider as things scale...) |
| [16:26:15] | <mstenta[m]> | Any application that's working with farmOS as a server may have similar problems... and both Field Kit and Asset Link have explored ways of tackling this |
| [16:26:52] | <mstenta[m]> | In those cases, they aren't trying to maintain a full mirror, so then it becomes a question of deciding which data is necessary (eg: for offline use) |
| [16:27:13] | <edbob[m]> | yes it really depends on how one uses the pair of systems, how they should be configured etc. in previous projects i was able to do real-time sync between systems but it all hinges around "knowing" when something changes. not sure if drupal provides any such hook, haven't looked yet. but if it does, then i have framework bits for the rest of it |
| [16:29:07] | <mstenta[m]> | Just did a quick search, maybe one of these would be helpful... |
| [16:29:07] | <mstenta[m]> | https://www.drupal.org/project/webhooks |
| [16:29:11] | <mstenta[m]> | https://drupalzone.com/tutorial/headless-drupal/104-using-webhooks-for-e... |
| [16:30:30] | <edbob[m]> | cool thanks! those appear to be what's needed, will bookmark |
| [16:31:21] | <mstenta[m]> | I haven't looked closely, so I can't vouch for them... but the webhook concept would work to notify your system of changes |
| [16:31:29] | <mstenta[m]> | Otherwise it's a pull process, looking for changes. |
| [16:31:46] | <mstenta[m]> | Octavio MartÃn Duarte did a lot of work on a "farmOS mirror" system that works that way. |
| [16:42:43] | <mstenta[m]> | https://gitlab.com/our-sci/software/mirror_farmos |
| [17:02:05] | <edbob[m]> | huh interesting - thanks |
| [17:07:40] | <edbob[m]> | <mstenta[m]> "edbob: Also curious if you..." <- also touched on in call: my current plan is mostly targeting the farmOS core, but also probably some "common" modules - so far the 'farm_eggs' module is included in my mirror etc. long term, it would be possible for "mirror modules" to exist for the python side as needed, but i don't have any plans for that yet. standard python packaging applies, which basically means writing a "module" is |
| [17:07:40] | <edbob[m]> | already possible..but will not bother unless/until it makes sense. my app may need some way to discover which farmOS modules are installed, e.g. an error may occur if i query for something which isn't there. i semi-expect that and will take a closer look when it comes up |
| [17:14:20] | <Greg[m]> | Happy to share or walk through our work, or Octavio can. We've done a lot of data mirroring for sure, and data model extraction |
| [17:17:43] | <edbob[m]> | thanks - i don't see why i shouldn't take you up on that ;) maybe in next week dev call, or..? feel free to share any other links in the meantime. i do have some general tasks to wrap up before i get to the trickier parts though, so not a blocker yet |