This page describes a list of TODO items related to multi-threading. It's probably helpful to look at the dav1d version of this page also to understand the background of the design.

Design

The design of dav2d's multi-threading is similar to dav1d's. In the most complete form, dav2d uses a 3-pass decoding pattern to decode multiple frames together. In this design, a frame is split into the following tasks:

• init & init_cdf;
• entropy parsing for one tile-sbrow (pass 1);
• motion vector & warp matrix resolution for one tile-sbrow (pass 2);
• block reconstruction for one tile-sbrow (pass 3);
• deblock, cdef (including ccso), wiener (including gdf) for one 64px-row;
• film grain for one Nx32 or Nx16 slice.

The 3-pass system was chosen so that multiple frames can process data concurrently with efficient dependencies between them. For example:

• a frame's entropy decoding will typically rely on the entropy decoding of some dependencies to have finished, either because the output CDF of the dependent frame is the input CDF for our frame, or due to - for any block in the frame - temporal dependencies (CCSO, segmentation), which are signaled in entropy_progress (frame.progress[0] for the reference).
• a block's motion vector & warp matrix resolution depends (if use_ref_frame_mvs=1) on temporal motion vector projections from previous frames. These are signaled right before postfilters start running in frame.progress[1] (reference).
• a block's reconstruction relies on pixels in reference frames (after inloop-postfilters), which are tracked using frame.progress[2] (reference) & lowest_px[] (the current frame's motion vector range).
• intrabc relies on before-postfilter data, so in order for postfilters to be able to run inline with block reconstruction, we copy the block reconstruction data. The original version is used for intrabc (and intra prediction), and the copy is used for inloop-postfiltering and eventually returned to the user.

TODO

Note for the items below: some are research questions. If you intend to work on this, you should implement & test whether multi-threading performance actually increased. The result may be negative.

• frame-mt (!606)
• (if intrabc=1) copy recon so that postfilter and recon can run concurrently (!617)
• tip frame reconstruction multithreading
  • skip pass=1 (entropy coding), which is just an empty task
  • split into tiles
  • reduce superblock size? Possibly 64x64 facilitates frame-mt better, test to see if this is true.
• in multi-pass decoding, once we get to pass>0, anything could be stateless. For example. if pass=0 stores in adjacent b entries (rather than positional), we can just iterate over these b's (assuming they store a bx/by position, perhaps within sb) and call decode_b or recon_b directly from task_thread.c without needing to call recon_sb at all. The advantage of this would be that we don't need ts->frame_thread[].partition[] anymore.
• This might also allow partial-runs, stashing intermediate state if lowest_px[]-style conditions are not fullfilled, and resuming later once they are.
• if we remove all type comparisons and assume that warpmatrix is valid, we can remove shear from the mvres pass entirely (for 3-pass frame-mt)
• for single-tile-col files, we might not need entropy_progress tasks, we can just run that inline with the entropy-tile tasks and signal progress when the task finishes.
• we should only wait for p_b in check_tile() for the mv-res pass if that ref exists in mfmv_mask, not for all refs. Something similar is true for entropy, we only need to wait for the refs relevant for ccso & segmentation (if active), not all. (!722)
• Av2Block.mtxbak and Av2Block.matrix can probably be unions over each other tpo reduce the size of Av2Block. Possibly matrix can be reduced from 24 bytes to less, at least for the [2-5] bits.
• refactor postfilter code to run in 64px-high (instead of superblock-size high) slices for better multi-threading/cache (!654)
• error code forwarding to user when frame-mt=1
• would be nice to merge the multiple pass functions in recon_tmpl.c back together with a few more branches, saves space and probably makes it easier to understand what's going on also... This is mostly the coef parsing loops. Right now, the luma tx_part splitting code is duplicated between coef parsing and reconstruction because of this, which is not very pretty.
• in the threading subsystem, some of the postfilter-conditions are still AV1-style, e.g. the LR task scheduling depends on LR being set in the seqhdr, but ignores GDF. Likewise, the CDEF task scheduling depends on CDEF presence but ignores CCSO.