Optionally track changes on a per-column basis so that clients of contiguous iteration can efficiently skip entire tables.

[pcwalton]

Bevy systems currently have no way to skip processing tables that contain no components that have changed since the last frame. There is extremely strong evidence that these systems are preventing Bevy both from scaling down to normal scenes on very low end hardware, and up to very large scenes on modern hardware. For example, there's currently no feasible way for Bevy to scale to the roughly 4.5 million entities that the Unity megacity demo has, even on modern hardware, much less the 7-year-old hardware that the 2019 demo was designed to run on.

The root cause is that Bevy has a variety of systems that examine every component that changed in some way from the previous frame. These systems are frequently the most expensive systems in typical apps: transform propagation, visibility propagation, and mesh extraction, to name a few. At the moment, these systems have no way to avoid checking the change tick of every entity to determine what changed. This has become a severe bottleneck as Bevy scales to larger scenes, particularly on lower-tier embedded hardware which often has slower memory and shares a single memory bus with the GPU.

This commit provides a way for such systems to avoid checking every row in tables by exposing a per-column summary tick to clients of the contiguous iteration feature. A summary tick is simply a field, present on each table column corresponding to a component that opts in to the feature, that records the value of the last change tick that was written to any cell in the column. Because keeping the change tick up to date adds some overhead to the Mut<T> type, which is performance-critical, components that wish to track summary ticks must opt into the feature like so:

#[derive(Component)]
#[component(summary_tick)]
struct MyComponent {
    ...
}

Contiguous iteration is a Bevy feature, seldom used at present, that directly exposes table columns to the query client as wrapped arrays. This makes it a natural mechanism for exposing information about columns. Because contiguous iteration yields entire tables at a time, it provides a straightforward way for clients to examine information about table columns to determine whether the client should process the table. Additionally, contiguous iteration is restricted to dense queries, which is precisely the restriction that we want in order to ensure that the summary tick is useful for systems that need to skip tables if no relevant components in the table have changed since the last time the query ran. Consequently, this PR uses the contiguous iteration feature as the preferred mechanism for exposing the summary tick to clients (but see the end of this commit message for further potential follow-ups).

The design rationale for this feature is covered in the comments in PR #25157, in order to keep this commit message relatively tidy. Please see those comments for more information. Because this PR has had so much discussion, I opted to keep it minimal and to defer potential improvements to follow-ups.

I measured the performance of updating components via Mut in various scenarios.

• Without summary ticks, the performance is unchanged (8.8846 µs main vs. 8.7340 µs this PR), showing that the correctly predicted branch on whether the component has a summary tick has no effect in a memory-bound task like component iteration.

• With summary ticks iterating sequentially, the performance is a 1.75x regression (8.7340 µs with no summary tick vs. 15.670 µs with a summary tick) which is to be expected as the summary tick adds a dependent load.

• With summary ticks iterating in parallel, the performance is a 1.14x regression (104.91 µs with no summary tick vs. 119.21 µs with a summary tick).

I did a performance test in which I created a version of extract_meshes_for_gpu_building that leverages parallel contiguous iteration from PR #25264 and summary ticks from this commit. After applying summary ticks to all the relevant components, running many_cubes --instance-count 1600000 --no-cpu-culling results in an improvement in extract_meshes_for_gpu_building of 5.13 ms to 0.039 ms, a 132× speedup.

I believe that the small regression in parallel performance--which is what matters for bulk component update--is well worth a massive across-the-board increase of Bevy performance in most applications. Note that extract_meshes_for_gpu_building is just one of the many systems that will benefit from this improvement.

All measurements were performed on a 2021 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz, 32 GB DDR4 RAM. Note that I quite intentionally chose older hardware here.

[pcwalton]

Note to self: this needs to also update the column changed tick when new entities are added.

[pcwalton]

This PR is the culmination of many, many, many discussions both in Discord and PR #23519. The outcome of these discussions is summarized here.

Here, "change indexes" means "any solution for the problem of detecting changes at a granularity higher than that of the individual row": i.e. any solution that allows large numbers of rows to be skipped at once without having to fetch the change ticks for each and every row.

Design constraints

The most important design constraint is that the regression in mutating component performance must be kept to a minimum. In particular, Mut<T> must not have synchronized read-modify-write instructions. On x86(-64) and ARMv6 or later, Relaxed atomics are "free" in that they compile to the same instruction as a regular store (though note that false sharing can still be an issue). However, read-modify-write instructions, such as atomic additions and bitwise operations, are very expensive compared to normal stores. Therefore, we wish to ensure that the only synchronized instructions that we use are relaxed stores, and not atomic OR operations as for example a bitfield would require.

It may be surprising at first that table- (or archetype-)level summary ticks require synchronization at all. After all, Mut<T> doesn't need synchronization when updating the existing per-row, per-component change tick. The most basic reason why table-level summary ticks need synchronization of some kind is par_iter_mut(). The presence of par_iter_mut() makes it entirely possible, and common, for a single system to spawn multiple threads that all write to the same table. Therefore, some amount of synchronization is required in order to ensure that par_iter_mut() worker threads don't race with one another.

Unfortunately, par_iter_mut() isn't the only race condition that can happen. It's possible for these two systems to run concurrently:

fn foo(q: Query<(&mut DenseComponent, &SparseSetComponentA), Without<SparseSetComponentB>>) { ... }
fn bar(q: Query<(&mut DenseComponent, &SparseSetComponentB), Without<SparseSetComponentA>>) { ... }

Suppose that some entities have a DenseComponent and a SparseSetComponentA and no SparseSetComponentB, while other entities have a DenseComponent and a SparseSetComponentB and no SparseSetComponentA. In that case, both systems will be accessing the same table mutably, with different change ticks. Any proposal for change indexing will need to have a solution to this problem. This issue sank many previous proposals.

An additional design constraint is that the proposal should ideally not diverge too far from the existing Changed semantics. The semantics of Changed are that the component has changed since the last time the system ran. It is not that the component changed this frame. Systems can run 0, 1, or many times per frame. The fact that a solution for change indexes shouldn't diverge too far from existing Changed semantics isn't a hard and fast rule, and proposals can sacrifice this constraint as long as they're clear about the semantics they are upholding. But it's something to keep in mind when evaluating the "cleanliness" of potential solutions.

Finally, any proposal for change indexes should be fast enough to perform well on low-end devices, including those with low DRAM bandwidth, and should ideally be simple to implement and iterate on.

How this solution works

The solution in this PR is based on contiguous iteration. Contiguous iteration is a little-used Bevy feature that takes a dense query and returns query data that represents entire table columns (or subsets thereof). This patch augments the query data representing a column with a new method that allows the consumer of the query to ask whether any component instance in the column has changed since the last time the query was run. This solves the problem, as systems that use such queries can simply ask whether the columns they’re interested in are changed before iterating over those columns’ contents; if all the columns in question are unchanged, then the entire table can be skipped. In essence, contiguous iteration provides objects that allow consumers of a query to ask questions about table columns as a whole, not individual rows, which is precisely what is needed in order to accelerate change detection.

There’s another, subtle yet crucial, advantage of contiguous iteration for this problem. Summary ticks can actually run backwards in certain situations involving sparse components. Consider the two systems above:

fn foo(q: Query<(&mut DenseComponent, &SparseSetComponentA), Without<SparseSetComponentB>>) { ... }
fn bar(q: Query<(&mut DenseComponent, &SparseSetComponentB), Without<SparseSetComponentA>>) { ... }

Suppose that Bevy runs the two systems concurrently. Then they will have different change ticks while simultaneously mutably accessing DenseComponent. If the this_run change tick of bar’s query is later than the corresponding change tick of foo’s query, then it’s possible that bar will mutate a DenseComponent that it’s iterating over, set the summary tick of the DenseComponent column to its this_run value, and then foo will later mutate a DenseComponent that it’s iterating over and set the summary tick of the DenseComponent column to its this_run value. This will cause the summary tick to run backwards. It means that a last-changed tick for a component may be later than the summary tick for that column.

This seems like an unfortunate complication. However, note that this issue only occurs when sparse queries are involved. Suppose that, in addition to the above, we have this dense query that uses contiguous iteration and checks summary ticks to avoid examining tables that had no changes to any DenseComponent:

fn baz(q: Query<&mut DenseComponent>) {
    for cs in q.iter_contiguous() {
         if !cs.any_changes() {
             continue;
         }
         for c in cs {
             // … do work …
         }
    }
}

Imagine that all those queries are unordered relative to one another, so Bevy could schedule them in any order. The scenario we want to prevent is a situation in which a change tick for a DenseComponent goes from after baz’s change tick to before baz’s change tick. This would cause baz to miss changes that it should have picked up.

But observe that the only time a change tick can go backwards is when two queries are scheduled concurrently and both have component filters that mention sparse set components (i.e. both queries are sparse queries). A dense query has exclusive access to the tables that it touches. In fact, this fact is central to what makes contiguous iteration safe at all, as handing out mutable access to an entire table column while other sparse queries are reading the table would be undefined behavior. A dense query that touches a table will never be scheduled concurrently with a sparse query. Therefore, even though summary ticks can run backwards, a summary tick that represents a time after a dense query will never be overwritten with one that represents a time before that dense query. So, if the consumer of a dense query asks the specific question “was any component in this column mutated since the last time this query ran?”, the answer can always be correctly determined by checking the summary tick.

Thus, by scoping the use of summary ticks to contiguous iteration, which also require dense queries, the fact that summary ticks can run backwards is rendered harmless. We must, of course, document that summary ticks can only be correctly used to detect changes since the last contiguous iteration. But, as long as we design the supported API appropriately to discourage misuse of summary ticks, this approach constitutes a simple solution that renders the pitfalls of summary ticks harmless.

Alternatives

The landscape of potential solutions to this problem is vast, and a huge number of alternatives have been proposed over the months of discussion leading up to this PR. Below are descriptions of other potential solutions and how they fare relative to the design criteria specified above.

1. Do nothing. This has the advantage of not regressing Mut<T> performance at all. However, there's abundant evidence that Bevy is having significant trouble achieving good performance on normal scenes for hardware from a few years ago, because of the memory bandwidth problems that come from checking every row of the same table over and over every frame.

Not only is there real-world evidence, but a simple back-of-the-envelope calculation based on DRAM speeds also demonstrates the bottleneck quite clearly. Suppose that we’re on a modern desktop machine with DDR5-4800 RAM, which runs at 38.4 GB/s. Say that we need to check 16 components per frame (which is similar to the number that the expensive passes check), that each change tick is 4 bytes, and that we have 4.5 million entities like the Unity megacity demo. Then we have (1000 * 4 * 4500000 * 16) / (38.4 * 1024 * 1024 * 1024) = 6.985 ms just to check change ticks. To hit 120 FPS, we have 8.333 ms per frame. So you’re looking at 83.82% of the frame budget spent just checking change ticks to match a Unity demo from 7 years ago, on powerful modern hardware, even if you get the maximum theoretical memory throughput that you will never hit in practice, even if you had infinite CPU cores.

2. Search change ticks using SIMD. We may well want to do this on top of this PR; it's a valid optimization and can be done in a straightforward manner with contiguous iteration. However, SIMD by itself doesn't really help very much, because the problem isn't the ALU cost of checking the change ticks: it's the memory cost. SIMD by itself doesn't improve memory bandwidth. Simple measurements of SIMD acceleration by @stuartparmenter have demonstrated an improvement of 20% or so for checking tables with no changes, which is insufficient to solve the problems we're seeing.

3. Search change ticks once per frame, and write the IDs of tables that changed this frame into a resource for rendering to use. This still has an O(n) search of all change ticks: somewhat better than the current situation, in which we have multiple systems that all search change ticks, but not hugely, because every change tick is still being checked. Moreover, because this scan runs before transform and visibility propagation, it requires that transform propagation and visibility propagation remember to keep the "dirty tables" list up to date whenever they perform changes. Since those systems are highly parallelized, that means that the "dirty tables" list must be a concurrent data structure, or else thread-local lists of dirty tables must be drained into the set after the parallel iterations, adding overhead and complexity. It also adds a maintenance burden to transform and visibility propagation, because now whenever those systems need to update change ticks, they must remember to keep the "dirty tables" list up to date. Finally, this system only works for rendering and not for other systems that might benefit from fast change detection, such as physics.

4. Have a flag per column that specifies whether any component in that column has changed this frame. This means that only one system can be the consumer of each such flag. That's because the system that consumes the flag must be the one to clear the flag, or else it would miss changes that occurred in between the time when the system ran and when the flag was cleared. This could work for rendering, visibility, and transform propagation, but keeping track of the timing here is a fragile nuisance.

5. Divide each table into pages, and have a summary tick per page. This should technically work and might well be a good follow-up for this PR. However, I would caution that I haven't had a lot of luck with page-based solutions, because the birthday paradox means that changing a small number of entities tends to dirty a lot of pages, and it doesn't take long before almost every page in the table is dirtied, at which point the pages are pure overhead. Checking multiple summary ticks per column is slower than checking a single summary tick per column, and if the overhead of paging doesn't buy us much, then there's no point in using it.

6. Divide each table into pages, and have a "changed" flag per page. This solution combines the disadvantages of (4) and (5). Additionally, it adds the unique problem that you probably want those flags to be packed into a bitfield for memory savings, at which point you're adding atomic RMW operations to Mut<T>.

7. Reduce the ALU cost of checking change ticks by widening them to 64-bit. Currently, checking to see whether a change tick has changed has a couple of branches in it in order to check for wraparound. Widening a change tick to 64 bits so that it will realistically never overflow would reduce this ALU cost. However, the issue was always memory bandwidth, not ALU. Widening change ticks from 32 bits to 64 bits would therefore be a significant regression in overall performance.

8. Accumulate column summary ticks into thread-local storage, and after each system update the summary tick for each column by taking a max over each thread's per-column summary tick. This adds a lot of overhead, as well as complexity around running per-system/per-query "cleanup" jobs, to each system for no particularly compelling gain. The scheme in this PR uses only relaxed atomics, which are "free" to begin with in that they compile to regular loads and stores on x86(-64) and ARMv6+.

9. Have per-archetype, per-component summary ticks rather than per-table, per-column summary ticks. The problem here is that multiple archetypes can point to the same table. Dense iteration works per-table, not per-archetype. You can't skip archetypes while iterating over a table, because entities of an archetype can be scattered throughout the table; in other words, archetypes don't cluster together. During dense iteration (what all the performance-critical systems in question use), if you want to skip unchanged tables, you would have to go from the table ID to all the archetypes that reference that table and check all the summary ticks for those archetypes. There's currently no mapping from table ID to all the archetype IDs, so we'd have to start maintaining that mapping for good performance. And there's no benefit to doing so, because we can't skip individual archetypes when iterating anyway. Since contiguous iteration is per-table, it's best to store the change summaries per-table.

10. Update the summary tick for a table column when mutably iterating over that column rather than in Mut<T>. In other words, whenever we reach a table in an iteration, for each component C, eagerly mark the column corresponding to each &mut C as potentially mutated by updating its summary tick. This can be done and avoids any overhead in Mut<T>, but it's quite conservative--even potentially accessing a column is enough to cause full table scans of change ticks. Given that the cost of updating the summary tick in Mut<T> is low, the extra precision of updating the summary tick only when a component is actually mutated seems worthwhile.

11. Introduce a Static or Stationary component (or its inverse) so that apps can tell Bevy explicitly whether entities are likely to change in such a way as to require re-rendering. We may in fact want to do this in the future, in order to mark objects that should receive baked lightmaps, for example. However, this by itself not only adds a burden to apps that want to achieve high performance but also is incompatible with objects that are usually static but occasionally dynamic. For instance, imagine you have a level with coins that the player can pick up. When a coin is picked up, it plays some sort of animation, but otherwise the coins are static (or animated entirely on the GPU so as to save on CPU overhead). Only on the small numbers of frames after the player has picked up a coin is any member of the coin table changing transform. Should the coins be marked Static or not? This is a difficult question to answer for developers, who would be better served by a system that can automatically skip unchanging tables on the specific frames in which those tables don’t change.

Future work

This commit constitutes the minimal version of the feature, and many follow-ups are possible:

• The query iteration logic could learn to accelerate Changed for regular queries, not just contiguous ones. This would only work if the query contains a set of filters that cause only rows with changed summary ticks to be selected and either (1) is dense or (2) iterates over tables that contain archetypes with no sparse sets. I opted to leave this to a follow-up because the logic is rather complex, and we haven't settled whether this would be worth it, given the complexity of these conditions. For instance, if we started relying on such a feature throughout rendering, then modifying the query to add a new component, and forgetting to add a summary tick to the component, would cause the query to lose its acceleration silently--and our ability to catch such performance regressions in CI is presently poor.

• Contiguous iteration could additionally be made parallel. This is necessary to make practical use of the summary ticks feature and is implemented in the companion PR Implement parallel contiguous iteration. #25264.

• Contiguous iteration could be accelerated with SIMD.

Because of the enormous amount of discussion that this feature has generated, I opted to implement only the minimal version of this feature and leave other improvements to follow-ups.

[pcwalton]

The new commit adds a read before the write, which fixes the performance problem on Tiger Lake. I believe this has to do with the store reorder buffer in the CPU forcing serialization of reads, but I'm unsure.

I should caution that, in general, the component mutation benchmarks do so little that microarchitectural quirks on specific models of CPUs can start to dominate the results. We should be careful about making decisions naively from their results, as actual component mutation tends to have a lot more arithmetic intensity than our microbenchmarks. Specifically, microbenchmarks of our mutation perform one vector load, one vector addition, and one vector store, but real systems tend to do a lot more than that, and so the regressions in the Mut benchmarks should not be taken as representative of performance of real games. Rather, the significant improvements demonstrated in Tracy (e.g. in many_cubes) should be taken as more representative (not to imply that many_cubes is representative, simply more representative than trivial microbenchmarks). See the discussion in alternative (1) for why I believe doing nothing is not a viable option.

[pcwalton]

I went ahead and marked this ready for review, pending the results of CI.

[eugineerd]

I think massive increase in efficiency more than makes up for a minor performance loss in iteration speed for normal queries, especially since contiguous_iter_mut is a more preferred api for batch updates either way due to it's better auto-vectorization capabilities.

With the summary_tick infrastructure in place, I also think it would be interesting to try applying alternative 10 (conservative summary tick) to all components since it'll be essentially free compared to #[component(summary_tick)] path during iteration, but might still give some easy wins in general (future work, not this PR).

[eugineerd]

Setting summary tick should probably be done in Column::initialize and Column::replace instead as that's where all the other Ticks are set.

Or maybe in BundleInfo::write_components since sparse set also uses these methods and there is no reason to add any additional overhead to it.

[eugineerd]

This should also set summary tick (ContiguousComponentTicksMut::mark_all_as_changed, github ate it)

[eugineerd]

This needs to be tested on other architectures, and I think it should be cfg-guarded if it actually only makes sense on x86-64.

[eugineerd]

Since summary ticks are only supported for table components, should this be a T::has_summary_tick() && matches!(T::STORAGE_TYPE, StorageType::Table) instead?

[eugineerd]

Same here, maybe even an assert since there's already one for layout?

[eugineerd]

Should this also check summary tick?

[eugineerd]

There are some weird formatting artifacts here it seems

[eugineerd]

Suggested change

	false, // has summary tick
	false, // no summary tick