HigherOrderGraph / modified MultiOrderModel

[vineetbansal]

A HigherOrderGraph class of arbitrary order, constructable from temporal_graph/event_graph/path_data. It delegates most of the work to the MultiOrderModel class which does the iterations on the lifting (with cached=False). It is also constructable using from_aggregated, and has its own .lift method.

The MultiOrderModel class now has HigherOrderGraphs in its layers.

None of the tests for MultiOrderModel needed modifications and still pass, which is reassuring. Tests for HigherOrderModel and MultiOrderModel that assume HigherOrderModel in layers are coming next.

This PR assumes that the EventGraph branch is merged, as it builds on top of it.

Typical workflow using these new classes:

def data() -> pp.TemporalGraph:
    r"""

        a           d
          \        /
            c  (hub)
          /        \
        b           e

    """
    return pp.TemporalGraph.from_edge_list(
        [
            ("a", "c", 1), ("c", "d", 2),   # a -> c -> d
            ("b", "c", 3), ("c", "e", 4),   # b -> c -> e
            ("a", "c", 5), ("c", "d", 6),
            ("b", "c", 7), ("c", "e", 8),
        ]
    )

t = data()
DELTA = 1
eg = EventGraph.from_temporal_graph(t, delta=DELTA)
h1 = HigherOrderGraph.from_temporal_graph(t, order = 1)
assert h1.order == 1

h2 = HigherOrderGraph.from_event_graph(eg)
assert h2.order == 2

paths = PathData(IndexMap(list("abcde")))
paths.append_walk(("a", "c", "d"), weight=3)
paths.append_walk(("b", "c", "e"), weight=3)

h1b = HigherOrderGraph.from_path_data(paths, order = 1)
h1c = HigherOrderGraph.from_event_graph(eg, order = 2)

h5 = HigherOrderGraph.from_event_graph(eg, order=5)  # Create order 5 ho (but still has to go through 2->5 algorithmically)

print("\n=== HigherOrderGraph (order 2) ===")
print("order:", h2.order)                   # 2
print("nodes:", h2.nodes)                   # [('a','c'), ('b','c'), ('c','d'), ('c','e')]
print("edges:", h2.edges)                   # [(('a','c'),('c','d')), (('b','c'),('c','e'))]
print("weights:", h2.data.edge_weight)      # [2., 2.]

assert h2.order == 2
assert h2.n == 4                            # 8 events collapsed into 4 nodes
assert h2.n_first_order == 5
assert h2.first_order_mapping.to_id(0) == "a"

h3 = h2.lift()
assert isinstance(h3, HigherOrderGraph)
assert h3.order == 3
print("order-3 nodes:", h3.nodes)  # [('a', 'c', 'd'), ('b', 'c', 'e')]


MAX_ORDER = 2

# build MultiOrderModel from TemporalGraph
m = MultiOrderModel.from_temporal_graph(t, delta=DELTA, max_order=MAX_ORDER)

for k, layer in sorted(m.layers.items()):
    print(f"  layer {k}: order={layer.order}  n={layer.n}  m={layer.m}")
    #   layer 1: order=1  n=5  m=4
    #   layer 2: order=2  n=4  m=2
    assert isinstance(layer, HigherOrderGraph)
    assert layer.order == k
    assert layer.n_first_order == t.n

# build MultiOrderModel from EventGraph
m_via_eg = MultiOrderModel.from_event_graph(eg, max_order=MAX_ORDER)
assert m_via_eg.layers[2].edges == m.layers[2].edges

# build MultiOrderModel from PathData
paths = pp.PathData(pp.IndexMap(list("abcde")))
paths.append_walks(node_seqs=[("a", "c", "d"), ("b", "c", "e")], weights=[4, 4])
m_paths = MultiOrderModel.from_path_data(paths, max_order=MAX_ORDER)

[M-Lampert]

The PR looks good already. I made some comments; let me know if you have any questions. I think some of the questions that I raised should be discussed together with the others in our next meeting. I said so in the comments as well.

[M-Lampert]

There could actually be a zeroth-order De Bruijn graph and it is currently an open question whether that is something that should be included in the MultiOrderModel or not. See #172
This is probably something that we should discuss together in the next meeting if this is something that we want the HigherOrderModel to be able to represent or not.

[M-Lampert]

I think that name is confusing since it requires an unaggregated graph as input. So the function should either be named aggregate or from_unaggregated.
We could also think about giving the unaggregated graph a more consistent name throughout the whole repository. Currently, it doesn't have a distinct name since it is just the representation for a step in between with the final result being the higher-order DeBruijn Graph. Some naming suggestions:

• Higher-Order (path-)occurence graph
• Higher-Order line graph
• Lifted graph

If and how we name this intermediate unaggregated higher-order graph, is probably also something we should discuss in the next meeting.

[M-Lampert]

I think this lift should use the pre-aggregation representation rather than self.data.edge_index.

At this point, edge_index and edge_weight have already been coalesced by from_aggregated(). Lifting them constructs paths implied by the aggregated De Bruijn topology, which can combine occurrences that were never observed consecutively in the source data and can propagate already-aggregated weights incorrectly.

inverse_idx already lets us reconstruct the pre-aggregation node sequences via:

node_sequence = self.data.node_sequence[self.data.inverse_idx]

If from_aggregated() also retains the occurrence-level edge index and weights (e.g. pre_aggregation_edge_index and pre_aggregation_edge_weight), then .lift() can call lift_order_step() on those tensors and aggregate only the result. That would make h.lift() consistent with the unaggregated iteration used by MultiOrderModel.

[M-Lampert]

Should we check if cached or max_order == 1: here?

[M-Lampert]

Since we have a dedicated higher-order graph class now, we can remove the higher-order workarounds in Graph like the node_sequence attribute that is only needed for orders larger than 1.

[M-Lampert]

This is also something only necessary for higher-order graphs and can be removed now.

[M-Lampert]

From now on, Graph is exclusively used for order-1 graphs, so this property is not necessary anymore.

[M-Lampert]

Suggested change

	summed. First-order nodes not traversed by any path remain as isolated nodes.
	summed. First-order nodes not traversed by any path remain as isolated nodes.
	Warning: This is a projection, not an inverse transformation
	This method does not reconstruct the original first-order graph from
	which this higher-order graph was built. Instead, it maps each higher-
	order node to either the first or last first-order node in its represented path.
	Consequently, the result preserves flow encoded by the higher-order model
	under the selected projection, but may differ from the original graph in its
	edge set. In particular, isolated first-order edges cannot be recovered.