Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-26 Thread Lance Dockins
To clarify, I am NOT using regexp as a replacement for locations. Some of the things that I need to do might be possible with locations. Most aren’t. I have to test more than just the request_uri and logically I have to use AND/OR and nested conditions. So it’s largely unfeasible with core Nginx

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-26 Thread Dmitry Volyntsev
On 9/26/23 8:30 AM, Lance Dockins wrote: Up until now, I had assumed that string.match types of statements were just transparently calling PCRE behind the scenes so that the associated memory from the PCRE call was being freed after use.  Maybe that’s not even an accurate read on how Nginx is

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-26 Thread Lance Dockins
Dmitry, I’ve been testing this more and I think that there’s more going on here than I was originally thinking. I have some js_set code that I have to run to properly route and filter requests and I noticed that it was consuming around 1mb per request (which woudl then carry to later header fil

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-22 Thread Lance Dockins
I am checking the content type, yes. But in my case, I’m just switching between body parsing methodologies depending on the body type. I do actually have a few scenarios where I have to evaluate body data in multipart submissions, but I only actually need the parts of the multipart form that are

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-21 Thread Dmitry Volyntsev
On 9/21/23 4:41 PM, Lance Dockins wrote: That’s good info.  Thank you. I have been doing some additional testing since my email last night and I have seen enough evidence to believe that file I/O in NJS is basically the source of the memory issues.  I did some testing with very basic command

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-21 Thread Lance Dockins
That’s good info. Thank you. I have been doing some additional testing since my email last night and I have seen enough evidence to believe that file I/O in NJS is basically the source of the memory issues. I did some testing with very basic commands like readFileSync and Buffer + readSync and

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-21 Thread Dmitry Volyntsev
On 9/21/23 6:50 AM, Lance Dockins wrote: Hi Lance, See my comments below. Thanky you, Dmitry. One question before I describe what we are doing with NJS.  I did read about the VM handling process before switching from Lua to NJS and it sounded very practical but my current understanding is

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-21 Thread Lance Dockins
Thanky you, Dmitry. One question before I describe what we are doing with NJS. I did read about the VM handling process before switching from Lua to NJS and it sounded very practical but my current understanding is that there could be multiple VM’s instantiated for a single request. A js_set, j

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-20 Thread Dmitry Volyntsev
On 20.09.2023 20:37, Lance Dockins wrote: So I guess my question at the moment is whether endless memory use growth being reported by njs.memoryStats.size after file writes is some sort of false positive tied to quirks in how memory use is being reported or whether this is indicative of a memo

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-20 Thread Lance Dockins
Thank you, Maxim. I’ve been doing some testing since I reached out earlier and I’m not sure whether I’m looking at a memory leak in Nginx/NJS or whether I’m looking at some sort of quirk in how memory stats are being reported by Nginx. All that I know is that my testing looks like a memory leak

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-20 Thread Maxim Dounin
Hello! On Wed, Sep 20, 2023 at 11:55:39AM -0500, Lance Dockins wrote: > Are there any best practices or processes for debugging sudden memory > spikes in Nginx on production servers? We have a few very high-traffic > servers that are encountering events where the Nginx process memory > suddenly

Re: Debugging Nginx Memory Spikes on Production Servers

2023-09-20 Thread Manuel
Hello, apparently you could look into dmesg. There should be a stacktrace of the process. Also you could somehow start nginx with gdb. You could also log all request and then when the server crashed try to replay them to be confident that the crash is reproducible. What does ChatGpt says? 😅