The goal here is to create a libavm-v11v12v13v14.0v14.1.0-compatible bitstream decoder.

General:

• convert to v12 (CTC samples; !56)
• convert to v13 (CTC samples; !384);
• convert to v14 (CTC samples; !597);
• convert to v14.1 (CTC samples; !605);
• convert to v15 (CTC samples; !693);
• convert to av2-normative (!843);
• multi-frame hdrs (!825)
• content interpretation OBU (!544)
• bridge frames
• lossless (!535)
• partial-lossless (particularly in postfilters; !564, !583, !586, !587, !588)
• qm
  • qm obus (default qms only)
  • user-defined qms
  • per-segment selectable qm levels
• parity hiding (!701)
• reduced-txtp-set (--reduced-tx-type-set=X; @juliobbv)
• delta_q (!515)
• segmentation (with 16 segids, not 8; !511; @KyleSiefring)
• max-DPB=16, not 8 (!805)
• remove BS_64x4 and BS_4x64 - these should be inaccessible with the aspect ratio constraints (edit: actually, the ii_masks[] will specifically need these, but not other places, so perhaps it makes sense for them to be moved to the end of the enum, and have an additional N_BS_SIZES (one for coded, another for chroma) so that CDF tables etc. don't have entries for 4x64/64x4, but ii_masks[] does.
• many CDFs have stale / unused items. Look for 16384 in bits etc. Many of the coef coding use 4 tx sizes for elements where there is only 2 or 3. See e.g. txtp_inter_tx_set where setidx=1 only exists for 16x16 and setidx=0 only for 4x4/8x8. So this could change from [2][4][3] to [3][3].
• some header bits have "magic values", e.g. for tip.frame_mode, "1" means "as_reference" whereas "2" means "frame_output". It would probably be useful to create (or re-use existing) enums for this, and then document which enum is meant to be used for each field while still using only 8 bits (uint8_t or similar) to actually store the type.
• CDF cross-tile averaging (requires a multi-tile file; @rbultje; !8)
• more dead code removal after that: reconstruction-related things w.r.t. dualfilter (!136), SGR / decomposable-wiener (!496), obmc (!413), filtered intra prediction (!11), super-resolution (!412)
• I would like c->refs[8] to include c->cdf[8], and then to have an equivalent (for all active refs) in f->refs[7] - to include everything [7] in f->* where it makes sense, probably refpoc, refdir, absrefdir, refrefpoc, refp, gmv_warp_allowed, etc.
• remove f->sr_cur, since super resolution no longer exists (!412).
• mv should move from 14+sign (int16_t) to 16+sign (int32_t?). We can use the extra bits to store information regarding refs or anything else that needs to be stored for refmvs purposes and save some memory. I wonder if it's possible to store 4 MV components in less than 128 bits, e.g. 96 or even 80? (Think of memory usage here.) we made refmvs_block 64byte which is probably as good as it gets for now
• CI should run debugoptimized, not release. (!263)
• update all copyright years (!411).
• t->scratch can probably be made leaner, need to analyze what data needs to stay unclobbered for how long.
• in decode_coefs(), sign bits can be chunked in sets of (up to?) 32, which should be slightly more efficient than doing them 1-by-1.
• For arm and x86 in particular, we should probably remove all SIMD for code that is no longer there (if we haven't done that already) - paticularly for things like avx512/ssse3 which we haven't started with yet - and %if 0 all other code to reduce binary size and build time from unused things. (!680, !681, !682, !683, !684; !696)

Inter parsing (@rbultje):

• TIP:
  • base implementation (!71)
  • tip frame reconstruction (requires tip block reconstruction to work; !321);
    • skip spatial context initialization (t->a, t->l; !565);
    • skip spatial MV initialization and updates - but keep temporal portions. (!601)
  • refdist[] is now calculated in multiple places (header reading / TIP, submit_frame() in decode.c and in refmvs, perhaps this could be shared to reduce redundant code & operations.
• refmvs (!95)
  • base implementation (!95);
  • opfl/refinemv (!202);
  • compound newmv^2 warp_causal
  • top edge handling (4x4->8x8) at (horizontal) sb boundaries (done for MVs, but not for warp). Likewise, warp derivation (in decode.c) at SB boundaries also needs to be adjusted (!229).
  • tmvp_step_size=2 should not mask bits in TMVPs, but rather increase blocksize (8x8->16x16) and reduce allocated blocks by 4x; (actually this can change frame-to-frame so this may not work)
  • remove weight from mvstack[];
  • 32bit MVs (!386);
  • to workaround 1146, we may need to generate warpMVs on-the-fly rather than storing them pre-warped.
    • We should probably move MV resolving from symbol coding into the reconstruction process (!565). This would also eliminate the need for struct SBEdgeCtx in env.h.
  • We should probably split spatial and temporal MV "splatting" into separate calls, or alternatively integrate it into splat (thus removing save_tmvs()) for non-refined MVs, and then manage temporal MV splatting 1-by-1 for refined MVs. (!332)
  • refmvs.c offsets references by one for easier separation between intra (ref=-1) and intrabc (ref=0), but that introduces significant complexity when mixing refmvs.c and non-refmvs.c code, since you always need to offset references by one in comparisons. Consider if it is worth undoing this (!418).
  • "extended" / mixed references are right now not implemented; these mix 0/1,1/0,0/2,2/0,1/2,2/1 of previous candidates for compound. (!258)
  • during the "refbank seeding" (from the top edge) at the start of each non-top SB, if a top block had a width of 4, because of the 8x8 granularity requirement, we add the same block twice. This is silly but normative. It would be faster if we added the block once but then increased the count (hits in the seeding function and hits[0] in the state keeper) by 2 instead of 1, to save a few cycles (see also #1259)
  • we don't check drl_idx against the mvpstack[] size, which allows entering uninitialized memory. (!310)
  • we could possibly remove refmvs_block.warp_type, since the reason for its existence (a bug in v12) was resolved upstream. (!668)
  • we need to re-enable the mechanism where load_tmvs() can be called per-tile_sbrow (instead of per-frame_sbrow) so tile threading can use this. (!545)
• LD samples
• bru (backwards reference update?)
• tx-part in partitions larger than 64x64 - there is one symbol per 64x64 sub-block. (apparently this is not a thing)
• in functions like get_snglref_ctx() and alike, we check whether any (not how many) matching refs exist along each edge (top, left); this means we can skip the top/right if w==4, and bottom/left if h==4.

Transforms:

• Primary transforms
  • identity (for H/V_{DCT,ADST_FLIPADST} - perhaps don't transpose the ones where identity is the 1st 1d-itx; same for IDTX
  • special "64x64" for subsampled chroma (in v12, this is for luma also and replaces the regular dct64; !115)
  • Reduce number of function pointers for inverse transform DSP interface, right now it's 25x16 which is a lot. (!140)
  • re-enable eob-based shortcuts in C code. (!517)
  • Secondary transforms
    • Entries (9, 10, 12) in ist_8x8_kernel are identity. Look into reducing the size of the kernel (!384).
• CCTX (for chroma; !365)

Intra reconstruction (@psilokos):

• directional prediction (including mrls and multi-line)
  • in all calls to filter_edge(), from is always 0 and to is always size. These parameters can presumably be removed/merged to simplify the interface.
• dip (data-driven intra) prediction (!11)
• palette prediction (!174)
• hook-up basic chroma intra prediction (!191)
• cfl prediction (!222) and mhccp (!304)
  • improve DSP interface: we discussed merging cfl_dc into cfl_ac, and splitting cfl_ac into cfl_ac_{exp,imp}licit, implicit calculating den and num for the alpha derivation, since that uses some of the regressed Y values calculated when computing Y edge DC. This avoids storing all the regressed Y values and allows to process them right away (sum all for DC, sum and mul some to UV values for alpha), compute num and den for alpha derivation for U & V from these and store these 4 in a buffer, or do the alpha derivation in the ASM directly, not sure yet. (!589)
  • measure rate of 0 alpha values for CFL_IMPLICIT to see whether it's worth rearranging the cfl branch in intra pred to execute DC_PRED instead.
  • MHCCP: revise imat buffer layout to have a fixed distance between edges (see !552) this is not needed, it would actually make the simd more complex
• Smooth Predictor don't use all the pixels in the edges, would could reduce the edge values to only those that are required. This refers to bottomleft/topright, where for all non-dip modes, we generate th top/right pixels and tw bottom/left pixels, even though smooth only appears to need 1. (!415)
• intrabc/morph-pred (!181)
  • also add hex_dump() calls after prediction (both in dav2d as well as to our AVM patch-set) to assist in mismatch debugging. (!176)
  • move alpha generation into its own function so that it can be shared with implicit CfL. (!222)
  • intrabc motion vector range restriction code is currently #if 0'ed, we should update & re-enable it
• we call prepare_intra_edges() with ibp=1 and then unset the flag afterwards when angle_delta&1. Unsetting this flag before the call to prepare_intra_edges() leads to pixel mismatches. I'm not sure why, the only thing affected by this flag is that we generate extra pixel edges, which (if ibp=0) appear unused. Look into this, this may mask a tiny bug in which edges are needed for intra modes in prepare_intra_edges(). [edit] I think the reason for this is orip, which requires top, left and top/left edges, but there is no provision for this. Ibp appears to mask this deficiency (!419).
• the edge preparation for inter-intra coding (particularly for n_bl) uses a loop, but we only ever need a single px, so we should be able to use a simplified code flow here. It's also only relevant for SMOOTH_PRED, so maybe a branch would be helpful here (!416).

Inter reconstruction (@rbultje):

• translational, warp, compound averaging, weighted averaging, inter/intra, wedges (!136)
• frame re-ordering (for situations where show-existing-frames packets are implicit; !187)
• bacp (boundary-aware compound prediction; !182);
  • move gen_mask() to a DSP interface, possibly without duplication between LBD and HBD (!721)
  • or alternatively, remove gen_mask() and do in-place mask generation in a new bacp_avg() DSP function. This doesn't help for tip/refinemv/opfl (because each sub-block has its own MV and therefore BACP output), but the regular case can be done at full block size.
  • there's some duplicate code before the call to gen_mask() that can be shared in a function and/or macro. (!331)
• difference-weighted prediction (!177);
• cwp (compound weighted prediction; !178) and maybe (?) bawp (block-adaptive weighted prediction)
• opfl/refinemv (!202)
  • refactor ref-area concept so that we don't end up calling emu_edge() multiple times per ref/block. Ideally, we can do implicit-sad before emu_edge even if the extended sad_refine_mc()s need emu_edge, and then (if needed) call emu_edge once for the whole sadrefine-sz block (16x16 or 8x8).
  • this is also important for chroma, where non-tip 4xN/Nx4 blocks always use emu_edge() even though they don't have to.
  • we should probably never use the destination buffer as a scratch (p0), because we can't guarantee that it is large enough. When reducing other buffers, we can presumably place both pixel p0[72 * 24] and pixel p1[72 * 24] in a struct along with int16_t mid[2][64 * 64] and then this should be fine. (!333)
  • opfl_stride can always be 8 (removed in !403)
• bawp (!228)
• tip (!175)
• size of comp-pred intermediate buffers can be reduced from 128x128 to 64x64 (!333)
• clean-up the bottom half of derive_warpmv() (get rid of mvd[] & ret, do in-loop replacement; !280)
• warp-extend at sb boundaries should access top at 8x8 (not 4x4) granularity (!335; still needs correct offsets in caller; !381)
• the SAD check before OPFL in TIP blocks could be integrated with the sad-refine or opfl-refine functions to reduce loads (!417). (decided to not do this for now)
• union OpflMvDeltaBlock [..] opfl[8 * 8]; in Dav1dTaskContext can move to stack in opfl_pred() and tip_pred() (done in !403)
• chroma (!403)
• would be nice if some form of DEBUG_B_PIXELS with inter-support could be made to give identical output to AVM and tested automatically in CI to prevent regressions (!421).

Postfilters:

• deblock (different from AV1 deblock; !114)
  • chroma (!401)
  • disable-deblock-on-tile-boundaries
  • delta-q / segmentation support (@KyleSiefring; !533)
  • the mask-generation in lf_apply_tmpl.c would probably be much simpler if we did deblock on 64xN/Nx64 edges instead of (up to) 256xN/Nx256 edges, and SIMD might not be worse off (except maybe avx512-for-chroma).
• cdef (sounds like it's similar to AV1 cdef; !133)
• ccso (new; !197)
  • chroma (!488)
• restoration (pixel-classified wiener and non-separable wiener; neither are like the separable wiener AV1 has, and SGR is gone; !378, !449)
  • chroma (!501)
  • introduce a new LR DSP function prototype for chroma so that we don't have to put the cross-component luma entries in the WienerParams struct, which is kind of ugly. (!736)
  • remove old SGR / decomposable-wiener code (!496)
  • optimize by using mask to skip zero-coefs
• guided detail filter (new; !494)
• film grain (@rbultje; !532)
• I believe in v14, CCSO and GDF gained the ability to use smaller (down to 64x64?) block sizes, we should probably implement that. (!787)

SIMD integration (@gramner):

• See separate page.