In the CPU->GSP messaging path, the code reads the write pointer from GSP, reads the response and advances the read pointer.
A LOAD->LOAD ordering is required after the write pointer read and the data read. Add it as this is currently missing. A LOAD->STORE ordering is required after the data read and the advance of read pointer. Currently a Rust `SeqCst` barrier is used, which roughly maps to `smp_mb(Full)`; this however does not order DMA operations (notably on ARM, the generate barrier orders inner shareable and not outer shareable, which is ordered by `dma_mb`). This ordering does not need to be in between read pointer read and write, because it's for ordering between the ring buffer data and the pointer; the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only), so move it out to cmdq.rs for clarity. Signed-off-by: Gary Guo <[email protected]> --- drivers/gpu/nova-core/gsp/cmdq.rs | 8 ++++++++ drivers/gpu/nova-core/gsp/fw.rs | 9 --------- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs index 94c2790d943d..229d49a67077 100644 --- a/drivers/gpu/nova-core/gsp/cmdq.rs +++ b/drivers/gpu/nova-core/gsp/cmdq.rs @@ -22,6 +22,8 @@ aref::ARef, barrier::{ dma_mb, + Full, + Read, Write, // }, Mutex, // @@ -339,6 +341,9 @@ fn driver_write_area_size(&self) -> usize { (MSGQ_NUM_PAGES, tx) }; + // ORDERING: Ensure data load is ordered after load of GSP write pointer. + dma_mb(Read); + // SAFETY: // - `data` was created from a valid pointer, and `rx` and `tx` are in the // `0..MSGQ_NUM_PAGES` range per the invariants of `gsp_write_ptr` and `cpu_read_ptr`, @@ -436,6 +441,9 @@ fn cpu_read_ptr(&self) -> u32 { // Informs the GSP that it can send `elem_count` new pages into the message queue. fn advance_cpu_read_ptr(&mut self, elem_count: u32) { + // ORDERING: Ensure read pointer is properly ordered with a LOAD->STORE ordering. + dma_mb(Full); + super::fw::gsp_mem::advance_cpu_read_ptr(&self.0, elem_count) } diff --git a/drivers/gpu/nova-core/gsp/fw.rs b/drivers/gpu/nova-core/gsp/fw.rs index fae2eae495d1..b213385c1f01 100644 --- a/drivers/gpu/nova-core/gsp/fw.rs +++ b/drivers/gpu/nova-core/gsp/fw.rs @@ -46,11 +46,6 @@ // TODO: Replace with `IoView` projections once available. pub(super) mod gsp_mem { - use core::sync::atomic::{ - fence, - Ordering, // - }; - use kernel::{ dma::Coherent, dma_read, @@ -76,10 +71,6 @@ pub(in crate::gsp) fn cpu_read_ptr(qs: &Coherent<GspMem>) -> u32 { pub(in crate::gsp) fn advance_cpu_read_ptr(qs: &Coherent<GspMem>, count: u32) { let rptr = cpu_read_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES; - - // Ensure read pointer is properly ordered. - fence(Ordering::SeqCst); - dma_write!(qs, .cpuq.rx.0.readPtr, rptr); } -- 2.54.0
