Re: neither CPU nor IO bound, but throttled performance

[Magnus Hagander]

On Thu, Feb 21, 2019 at 12:34 AM Gunther  wrote:

> Hi, I have an Amazon Linux based Postgresql 11 server here on a t2.medium
> EC2 instance.
>
> It is serving 24 worker processes that read jobs from a queue (thanks to
> SELECT ... FOR UPDATE SKIP LOCKED!) and do jobs some of which are reading
> and writing business data to the database, others are only reading, and
> some don't hit the business data at all, only the queue.
>
> Everything flows quite nicely. Except, I don't understand why I can't max
> out the CPU or the IO, instead, IO is almost negligible yet the CPU is at
> 30% hardly hitting 50%.
>
> Here I give you a view of top:
>
> top - 23:17:09 up 45 days,  2:07,  4 users,  load average: 20.32, 18.92, 13.80
> Tasks: 338 total,  24 running, 111 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 28.7 us,  2.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 68.7 
> st
>
>
If I read that right, it's about 70% "steal". The description for this is
"the percentage of time spent in involuntary wait by the virtual CPU or
CPUs while the hypervisor was servicing another virtual processor.".

So basically, the CPU is spent dealing with other peoples VMs on the same
hardware. Welcome to the multi-tenant cloud.

In particular, I believe T series instances get a limited number of CPU
"credits" per hours. My guess is you've hit this limit and are thus being
throttled. T series are not intended for persistent workloads. Either way,
this is probably a question better asked at the Amazon EC2 forums rather
than PostgreSQL as you'll find more people who know the EC2 interactions
there.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: neither CPU nor IO bound, but throttled performance

[Gunther Schadow]

Thank you Magnus. 68% steal. Indeed. You probably hit the target. Yes.

That explains the discrepancy. I need to watch and understand that CPU 
credits issue.


regards,
-Gunther

On 2/21/2019 4:08, Magnus Hagander wrote:
On Thu, Feb 21, 2019 at 12:34 AM Gunther > wrote:


Hi, I have an Amazon Linux based Postgresql 11 server here on a
t2.medium EC2 instance.

It is serving 24 worker processes that read jobs from a queue
(thanks to SELECT ... FOR UPDATE SKIP LOCKED!) and do jobs some of
which are reading and writing business data to the database,
others are only reading, and some don't hit the business data at
all, only the queue.

Everything flows quite nicely. Except, I don't understand why I
can't max out the CPU or the IO, instead, IO is almost negligible
yet the CPU is at 30% hardly hitting 50%.

Here I give you a view of top:

top - 23:17:09 up 45 days, 2:07, 4 users, load average: 20.32,
18.92, 13.80 Tasks: 338 total, 24 running, 111 sleeping, 0
stopped, 0 zombie %Cpu(s): 28.7 us, 2.5 sy, 0.0 ni, 0.0 id, 0.0
wa, 0.0 hi, 0.0 si, 68.7 st


If I read that right, it's about 70% "steal". The description for this 
is "the percentage of time spent in involuntary wait by the virtual 
CPU or CPUs while the hypervisor was servicing another virtual 
processor.".


So basically, the CPU is spent dealing with other peoples VMs on the 
same hardware. Welcome to the multi-tenant cloud.


In particular, I believe T series instances get a limited number of 
CPU "credits" per hours. My guess is you've hit this limit and are 
thus being throttled. T series are not intended for persistent 
workloads. Either way, this is probably a question better asked at the 
Amazon EC2 forums rather than PostgreSQL as you'll find more people 
who know the EC2 interactions there.

--
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/