Entity change listener should not receive info on all entities #581

Activity

1. timmclean

   added

   performance

   area/store

   on Nov 16, 2018
2. timmclean

   added this to the Hosted Service milestone on Nov 16, 2018
3. Jannis commented on Nov 23, 2018

   

   Jannis

   on Nov 23, 2018

   Last edited by Jannis

   Contributor

   More actions

   Wouldn't that require maintaining (temporary) subscription state in Postgres? Unless we'd use one channel per subgraph ID + entity name and have EntityChangeListener listen on multiple of these channels.

   The following goes beyond what the issue is about but I wanted to capture my thoughts about subscriptions and performance:

   ---

   Thinking in sets

   Making subscriptions performant is a hard problem. If we take sorting and nesting out of the equation, a (subscription) query corresponds to a set of entities of a different types. Let's make a couple of assumptions:

   1. We identify and temporarily store this information for each subscription.
   2. To update a subscription, we simply rerun its query and send the result to the client.

   After a subscription is submitted, several things can happen. First, let's look at the simple cases:

   • An entity from a subscription set is removed.
   • An entity from a subscription set is updated.

   In both cases it is easy to find out which subscriptions are affected and need to be updated. In case of a removal, we can remove the entity from the subscription set.

   The more complicated scenario is when a new entity is added that could belong into the subscription set, or when an existing entity that is not in the subscription set is updated in a way that moves it into the subscription set. Identifying whether that is the case is hard — that's why we currently just rerun all subscription queries to make sure.

   Thinking in graphs

   Thinking of the "reach" of a query in terms of a set is not very good. Let's try thinking about the reach of a query as a graph. Here's a first attempt that ignores sorting*:

   1. For every entity in the initial query result, there is an entity node in the query graph.

   2. For every collection of entities (whether at the top level or through a singular or plural relationship), there is a relationship node in the query graph.

      1. Each relationship node is tagged with filter conditions, including relationship conditions.

      2. For each relationship node, there is a directed edge from the the relationship node to the entity nodes for the entities in that collection.

   3. For every relationship field in the query, there is an edge from the entity nodes that the field was resolved for to the relationship node for that relationship. The edge is labeled with the field name.

   Now, let's look at how we handle entity changes:

   1. An entity is added. Identify all relationship nodes. Check if the filter and relationship conditions of any relationship node apply to the entity. If so, add an entity node and an edge from any relationship node for which the conditions apply to the entity, to this new entity node. Update the query result.**

   2. An entity is updated

      1. If the corresponding entity node exists in the query graph, follow the edge back to its relationship node. Check if the filter and relationship conditions still apply to the entity. If yes, keep the node and update the query result**.
      2. If no corresponding entity node exists in the query graph, treat the entity as if it has been added (see 1.).

   * — Sorting is a little tricky, as it's not a simple condition that applies to just one node. One possible solution here could be to also tag the relationship nodes with sorting info (orderBy, orderDirection) and, when testing a new/updated entity node for inclusion/removal, resort the nodes in the set and add the new entity node and/or remove an existing entity node to/from the query graph and relationship node.

   ** — Updating the query result could either be done by simply re-running the query. In that case, we could construct a new query graph from the result and wouldn't have to modify the graph separately. If we don't rerun the query, we have to either keep the query result around and update it or derive the query result from the updated query graph.

   These are my current thoughts. I'd love to spend a bit more time on this and identify a simple, fast and memory-efficient way to do this.

   👍React with 👍1Reacted by Tim McLean
4. Jannis commented on Sep 16, 2020

   

   Jannis

   on Sep 16, 2020

   Contributor

   More actions

   Difficult, also graph-node-level subscriptions will go away in the network.
5. Jannis

   closed this as completedon Sep 16, 2020