The MergeInserter is doing a merge sort because there is no usable
descending index on system_log.time. The storage layer's page cache (the
ConcurrentCache) is filling up because you have to fault-in the entire
contents of system_log. The logic in MergeInserter.insert() does not
seem to be smart enough to realize that it is close to exhausting memory
and needs to spill a merge run to disk.
I would recommend throwing more memory at your JVM or adding a
descending index to system_log.time.
On 2/8/21 1:41 AM, Peter Ondruška wrote:
OutOfMemoryError: GC overhead limit exceeded happens when JVM spends way much
garbage collecting. Could it be your heap is too small? One way to check if
DateTimeFormat may be leaking memory is to run query without this function.
-----Original Message-----
From: John English <[email protected]>
Sent: Monday, February 8, 2021 10:28 AM
To: Derby Discussion <[email protected]>
Subject: Out-of-memory errors
In the last few days I've suddenly had a bunch of OOM exceptions. I'm using
Derby 10.9.1.0, Oracle Java 1.8.0 on Ubuntu 64-bit, and haven't upgraded for a
while (probably years, looking at those numbers).
The place where they happen is in a call to executeQuery() in a method which
displays a view as a table. Analysing the heap dump for the latest one with the
Eclipse memory analyser shows this:
One instance of "org.apache.derby.impl.store.access.sort.MergeInserter"
loaded by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0xf04231b0"
occupies 134,841,800 (64.65%) bytes. The memory is accumulated in one instance of
"org.apache.derby.impl.store.access.sort.SortBuffer", loaded by
"org.eclipse.jetty.webapp.WebAppClassLoader @ 0xf04231b0", which occupies 134,841,496
(64.65%) bytes.
One instance of "org.apache.derby.impl.services.cache.ConcurrentCache"
loaded by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0xf04231b0"
occupies 43,766,832 (20.98%) bytes. The memory is accumulated in one instance of
"org.apache.derby.impl.services.cache.ConcurrentCache",
loaded by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0xf04231b0", which
occupies 43,766,832 (20.98%) bytes.
The query itself was:
SELECT DateTimeFormat(t_time,null) AS t_time,
facility,event,details,name,username,sector,item
FROM system_log_view
ORDER BY time DESC
NULLS LAST
FETCH NEXT 20 ROWS ONLY
The view is nothing special except that t_time is a duplicate of the time
column (the timestamp of the log entry) used to create a separate formatted
copy for display purposes:
CREATE VIEW system_log_view AS
SELECT time AS t_time,
facility,
event,
details,
name,
username,
sector,
item,
time
FROM system_log;
The stack trace shows the error is occurring inside the call to DateTimeFormat,
which is again nothing special:
public static final String formatDateTime (Timestamp date, String
locale) {
if (date == null) {
return null;
}
else {
String fmt = translate("d-MMM-yyyy 'at' HH:mm",locale);
return translatePhrases(fmt.format(date),locale);
}
}
Here's the start of the stack trace:
java.sql.SQLException: The exception 'java.lang.OutOfMemoryError: GC overhead
limit exceeded' was thrown while evaluating an expression.
at java.text.DigitList.clone()Ljava/lang/Object; (DigitList.java:736)
at java.text.DecimalFormat.clone()Ljava/lang/Object;
(DecimalFormat.java:2711)
at java.text.SimpleDateFormat.initialize(Ljava/util/Locale;)V
(SimpleDateFormat.java:645)
at
java.text.SimpleDateFormat.<init>(Ljava/lang/String;Ljava/util/Locale;)V
(SimpleDateFormat.java:605)
at java.text.SimpleDateFormat.<init>(Ljava/lang/String;)V
(SimpleDateFormat.java:580)
at
database.Functions.formatDateTime(Ljava/sql/Timestamp;Ljava/lang/String;)Ljava/lang/String;
(Functions.java:51)
Does anyone have any idea what might be happening, or what I can do to find out
more?
TIA,
--
John English
