Thanks.
Depending on how the increment compares to the file size, file->bytes might
be up to 1.5x faster (on my machine at least) than the workaround. But the
workaround is still fast enough.
(define (file->bytes2 file-name)
(define size (file-size file-name))
(define bs (make-bytes size))
(call-with-input-file
file-name
(lambda (in)
(let loop ((i 0))
(cond ((= i size) bs)
(else (define end (+ i (min 1073741824 (- size i)))) ;; 1GB
increment
(read-bytes! bs in i end)
(loop end)))))))
On Wednesday, July 15, 2020 at 6:32:36 PM UTC-4, Matthew Flatt wrote:
>
> The `file->bytes` function uses the file size with `read-bytes`, and it
> appears that the Mac OS `read` system call errors on requests of 2GB or
> more. The right fix is for the `read` call within Racket (at the rktio
> layer) to limit the size that it passes, and I'll make that change.
>
> Meanwhile, you could work around the problem by limiting the size of an
> individual request: Allocate a byte string and then use a sequence of
> `read-bytes!` calls to read the file in increments. Each time, use the
> number of read bytes to increment a starting position into the
> destination byte string (which is the third argument to `read-bytes!`).
>
> Matthew
>
> At Wed, 15 Jul 2020 15:05:22 -0700 (PDT), Greg Rosenblatt wrote:
> > Hi, I'm getting an error while using file->bytes to load a moderately
> large
> > file:
> >
> > > (time (void (file->bytes "my-7.6GB-file")))
> > ; error reading from stream port
> > ; port: #<path:/Users/greg/my-7.6GB-file>
> > ; system error: Invalid argument; errno=22
> > ; context...:
> > ; /Applications/Racket v7.7/collects/racket/file.rkt:768:6: temp218
> > ; /Applications/Racket
> > v7.7/collects/racket/private/more-scheme.rkt:336:52
> > ; eval-one-top
> > ; /Applications/Racket
> v7.7/share/pkgs/xrepl-lib/xrepl/xrepl.rkt:1478:0
> > ; /Applications/Racket v7.7/collects/racket/repl.rkt:11:26
> >
> > Is there a limit to the size of files that can be used with file->bytes?
> >
> > I was preferring file->bytes because it seems much faster than manually
> > reading from a port. If file->bytes is not appropriate here, can
> somebody
> > recommend another fast approach?
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "Racket Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to [email protected] <javascript:>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/racket-users/e99edda0-06ed-4164-b7bd-46f8a458
>
> > c6c8o%40googlegroups.com.
>
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-users/b6d1ff87-c775-4117-8f88-7816f6434212o%40googlegroups.com.
