[go-nuts] io.CopyN occasionally fails with unexpected EOF on large ranged reads

[Kanak Bhatia]

Hi all,

I'm encountering an intermittent issue while using io.CopyN to copy 129MB 
object from reader to writer.

The problem:
On some iterations, io.CopyN(dst, src, n) fails with unexpected EOF even 
though the same logic succeeds in most runs. The failure typically happens 
after 125–130 MB and returns ~1MB less data than expected.

type debugReader struct {
    r          io.Reader
    totalBytes int64
    nextLogMB  int64
}

func (d *debugReader) Read(p []byte) (int, error) {
    n, err := d.r.Read(p)
    d.totalBytes += int64(n)
    if d.totalBytes >= 125*1024*1024 {
        currentMB := d.totalBytes / (1024 * 1024)
        if currentMB >= d.nextLogMB {
            fmt.Printf("🔵 Read %d MB so far\n", currentMB)
            d.nextLogMB = currentMB + 1
        }
    }
    return n, err
}

type debugWriter struct {
    w          io.Writer
    totalBytes int64
    nextLogMB  int64
}

func (d *debugWriter) Write(p []byte) (int, error) {
    n, err := d.w.Write(p)
    d.totalBytes += int64(n)
    if d.totalBytes >= 125*1024*1024 {
        currentMB := d.totalBytes / (1024 * 1024)
        if currentMB >= d.nextLogMB {
            fmt.Printf("🟡 Written %d MB so far\n", currentMB)
            d.nextLogMB = currentMB + 1
        }
    }
    return n, err
}

func SafeCopyN(dst *debugWriter, src *debugReader, n int64) (int64, error) {
    fmt.Printf("\n🔁 Starting SafeCopyN for %d bytes\n", n)
    written, err := io.CopyN(dst, src, n)
    fmt.Printf("📊 Total bytes read:    %d\n", src.totalBytes)
    fmt.Printf("📊 Total bytes written: %d\n", dst.totalBytes)

    const allowedDrift = 128 * 1024

    if err != nil {
        diff := n - written
        if err == io.ErrUnexpectedEOF && diff <= allowedDrift {
            fmt.Printf("⚠  Accepting unexpected EOF: copied %d of %d bytes 
(drift: %d bytes)\n", written, n, diff)
            return written, nil
        }

        fmt.Printf("❌ CopyN failed: wrote %d of %d bytes, err: %v\n", 
written, n, err)
        if src.totalBytes != dst.totalBytes {
            fmt.Printf("📉 Mismatch: %d bytes read but not written\n", 
src.totalBytes-dst.totalBytes)
        }
    } else {
        fmt.Printf("✅ Successfully copied %d bytes\n", written)
    }

    return written, err
}


---

Here's a real log from one of the failed runs (test runs 20 times, fails 2):


🔁 Starting SafeCopyN for 135264256 bytes
🔵 Read 125 MB so far
🟡 Written 125 MB so far
🔵 Read 126 MB so far
🟡 Written 126 MB so far
🔵 Read 127 MB so far
🟡 Written 127 MB so far
📊 Total bytes read:    134215680
📊 Total bytes written: 134215680
❌ CopyN failed: wrote 134215680 of 135264256 bytes, err: unexpected EOF
📉 Mismatch: 0 bytes read but not written

Notes:

This only happens 1 to 2 times out of 20 iteration.

No server-side encryption involved.
---

Question:

Has anyone seen something similar? Could this be:

Timeout/internal truncation in io.CopyN?

Should io.CopyN tolerate io.ErrUnexpectedEOF more gracefully in this case?


Any insights or guidance appreciated!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/56d2219e-32ab-4950-ab49-83b5fbbfc4fen%40googlegroups.com.