[issue5396] os.read not handling O_DIRECT flag

Change by Jakub Wilk : -- nosy: +jwilk ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org

[issue5396] os.read not handling O_DIRECT flag

Logan Gunthorpe added the comment: Paul's solution works in 3.7 if you set the buffer size to zero when calling fdopen. fd = os.open("my_file", os.O_DIRECT | os.O_RDWR) f = os.fdopen(fd, "rb+", 0) m = mmap.mmap(-1, 4096) f.readinto(m) This is according to a comment in _pyio.py: # If remaini

[issue5396] os.read not handling O_DIRECT flag

Michael Mol added the comment: I wound up writing it in C++ instead, and my then-employer eventually opened my code. https://github.com/VirtualInterconnect/diskstress -- ___ Python tracker __

[issue5396] os.read not handling O_DIRECT flag

Paul added the comment: I've dug into stracing this python program in 2.7 vs. 3.7. directread.py import mmap import os fd = os.open('/dev/dm-2', os.O_DIRECT | os.O_RDWR) # mapped block device fo = os.fdopen(fd, 'rb+') m = mmap.mmap(-1, 4096) fo.readinto(m) Python 2.7 result: ... open("/de

[issue5396] os.read not handling O_DIRECT flag

Paul added the comment: Michael, I ran into the same issue as you. I got it to work by changing the mmap size to 8K. d = os.open(disk_file_path, os.O_RDWR | os.O_DIRECT | os.O_SYNC | os.O_DSYNC) readbuf = mmap.mmap(-1, 8192) os.lseek(d, 0, os.SEEK_SET) fo = os.fdopen(d, 'rb') fo.readinto(re

[issue5396] os.read not handling O_DIRECT flag

Michael Mol added the comment: I need to remember not to try to write quick programs in Python that need O_DIRECT. My use case: I'm attempting to write a program that forces the disk to seek to a particular place, as part of burning in the disk. My algorithm goes: 1. Seek to the beginning of

[issue5396] os.read not handling O_DIRECT flag

Jean-Paul Calderone added the comment: > I think the policy is to mirror all possible O_* constants, even if they > are of no use in Python. For example, we also have os.O_DIRECTORY. Not disagreeing with the conclusion of this ticket, but I would like to point out that os.O_DIRECTORY isn't use

[issue5396] os.read not handling O_DIRECT flag

Antoine Pitrou added the comment: Hello Eduardo, > 1) to allocated an aligned buffer it is as simple as allocate 4096 + len > (buffer) and truncate address to 4k boundary. Yes, but it is wasteful, especially since the common case is not to use O_DIRECT. Also, os.read does not allocate a buffer

[issue5396] os.read not handling O_DIRECT flag

Eduardo Aguiar added the comment: Hi, I think I have a few more issues you can consider: 1) to allocated an aligned buffer it is as simple as allocate 4096 + len (buffer) and truncate address to 4k boundary. 2) I wrote a floppy imager, and without O_DIRECT, it gives me 8 sectors (4k = kernel

[issue5396] os.read not handling O_DIRECT flag

Antoine Pitrou added the comment: I don't think we will try to support O_DIRECT. First, it is a rather platform-specific flag. Second, Python's memory allocator doesn't allow specifying an arbitrary alignment value. Third, I don't even think O_DIRECT can make a positive difference in a Python pr

[issue5396] os.read not handling O_DIRECT flag

Changes by Antoine Pitrou : -- priority: -> normal type: behavior -> feature request versions: +Python 2.7, Python 3.1 -Python 2.6 ___ Python tracker ___

[issue5396] os.read not handling O_DIRECT flag

New submission from Eduardo Aguiar : At posixmodule.c (line 6306) static PyObject * posix_read(PyObject *self, PyObject *args) { int fd, size, n; PyObject *buffer; if (!PyArg_ParseTuple(args, "ii:read", &fd, &size)) return NULL; if (size < 0) {