Re: [PATCH weston 0/6] ivi-shell proposal

[nobuhiko_tanibata]

Hi Jason,

Thank you very much for feedback.
Michale & Nobuhiko,

First of all, thank you for the clarification and thank you for
sending this to the list and being willing to work with the FOSS
community to try and make a standard. I'm sorry that this reply is not
inline. I think that would get disorganized and more confusing but
I'll try to hit everything I saw.

The first distinction that needs to be made (and I think Pekka was
trying to hint at this) is what should be standardized. If you look at
the current [wayland core protocol][1] there is only one shell
protocol called wl_shell. I have proposed another which will probably
get called [wl_fullscreen_shell][2]. Both of these have something in
common: they are purely client-side. There is nothing whatsoever in
the standard about managing surfaces. I think that we should focus on
what you have designated ivi_client.

What about the controller? If you look in the [weston protocol
folder][3] you will find a number of different protocol files. Some of
these are for experimental extensions such as subsurfaces which have
not yet made it into wayland core. However, a number of them such as
desktop-shell, screenshooter, etc. will *never* be standardized in the
wayland core. These protocols are completely internal to weston and
are considered implementation details. The primary example is
desktop-shell. This protocol exists for the purpose of allowing the
out-of-process shell controller manage surfaces similar to what you
propose with ivi_shell. There are other shell plugins for weston
(hawaii & orbital) that each have their own shell plugin and can have
their own protocol for talking to an out-of-process controller.

How does this impact your proposed protocol? Unless you are convinced
that every single IVI system manufacturer will want to manage surfaces
the same way, the controller should be left as a private
implementation detail. You are free to do it out-of-process and talk
the wayland protocol to do so (desktop-shell does) but there is no
need to expose it as part of a standard protocol. By only
standardizing the client interface you leave app developers (GPS,
Media players, etc.) free to design their apps however they want and
you leave IVI system manufacturers free to handle those clients and
surfaces in whatever way they want.

Ok, now on to actual suggestions. >From this point forward, I am going
to completely ignore the controller side of things.

First, I would propose to follow the pattern of wl_shell and make two
interfaces for clients to talk to the compositor. For now, I am going
to call them wl_ivi_shell and wl_ivi_surface. We can come up with
different names if you'd like, but those seem reasonable. If we follow
the pattern of wl_shell, wl_ivi_shell will probably exist for the sole
purpose of creating wl_ivi_surface objects. This pattern is common in
the protocol (wl_shell, wl_subcompositor, wl_compositor, etc.).

The main question, then, becomes what to put in wl_ivi_surface. I'm
not 100% sure what you intend with some of this surface and layer
stuff, so I'm afraid I don't have a whole lot of specific suggestions
on that for now. I do, however have some general thoughts and
questions:

 First, I agree with Pekka that you can probably avoid the layers
thing by simply using subsurfaces.

I see. However, we have a use case that several application, different 
process share a layer. E.g. Navigation map and Route guidance are 
separated into other application. It may kind of grouping parent 
surfaces.

Second, Why are you specifying pixel formats in ivi_surface? Is the
compositor supposed to tell the client what format to render in?

Third, concerning the "visibility" flag. The wayland protocol as it
currently stands tries to avoid telling clients specifically whether
or not they are visible and where they are on screen. This is because,
when clients abuse this information, compositors lose the freedom to
throw surfaces around how they want. Instead of a visibility flag, the
wl_surface interface provides a "frame" callback that the clients can
use to know when was the last time they were drawn to the screen. A
client should throttle rendering based on these frame events. If the
surface is offscreen and the compositor wants the client to stop
rendering it simply stop sending it frame events and the client will
stop drawing.


I have two concerning to use "frame" for realizing invisible.
- To set invisible, application needs to call clear color by itself. I 
think it might be overhead for GPU. If we can realize it in shell, it 
simply skips it to be composite. Of course, application shall stop 
drawing as well.
- Invisible shall be conrolled by cetral controller due to safy 
reason.It shall be done in lower part as much as possible. Ideally, if 
we can allocate it to another physical plane, it may be best. If Display 
contoller doesn't support it, next is compositor.

Once again, thank you for mailing the list. I hope my thoughts above
are helpful and can clear a few things up.
 Thanks,

--Jason Ekstrand

 [1]:
http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
[2]
 [2]:
http://lists.freedesktop.org/archives/wayland-devel/2013-August/010720.html
[3]
 [3]: http://cgit.freedesktop.org/wayland/weston/tree/protocol [4]

On Mon, Sep 9, 2013 at 7:59 PM, Nobuhiko Tanibata
<nobuhiko_tanib...@xddp.denso.co.jp> wrote:

 ----- Original Message ----- From: "Pekka Paalanen"
<ppaala...@gmail.com>

To: "nobuhiko_tanibata" <nobuhiko_tanib...@localhost.xddp.denso.co.jp>
 Cc: "Nobuhiko Tanibata" <nobuhiko_tanib...@xddp.denso.co.jp>;
<securitych...@denso.co.jp>; <wayland-devel@lists.freedesktop.org>
 Sent: Sunday, September 08, 2013 6:31 PM

 Subject: Re: [PATCH weston 0/6] ivi-shell proposal

On Sun, 08 Sep 2013 00:13:55 +0900
 nobuhiko_tanibata <nobuhiko_tanib...@localhost.xddp.denso.co.jp>
wrote:

2013-09-06 19:16 に Pekka Paalanen さんは書きました:
 > On Fri, 6 Sep 2013 08:39:24 +0900
 > "Nobuhiko Tanibata" <nobuhiko_tanib...@xddp.denso.co.jp> wrote:
 >
 >> ----- Original Message -----
 >> From: "Pekka Paalanen" <ppaala...@gmail.com>
 >> To: "Nobuhiko Tanibata" <nobuhiko_tanib...@xddp.denso.co.jp>
 >> Cc: <wayland-devel@lists.freedesktop.org>;
<securitych...@denso.co.jp>
 >> Sent: Thursday, September 05, 2013 10:02 PM
 >> Subject: Re: [PATCH weston 0/6] ivi-shell proposal
 >>
 >>
 >> > On Wed, 4 Sep 2013 09:08:26 +0900
 >> > "Nobuhiko Tanibata" <nobuhiko_tanib...@xddp.denso.co.jp> wrote:
 >> >
 >> >> Hi,
 >> >>
 >> >> This series implements ivi-shell to fulfill use cases of
In-Vehicle
 >> >> Infotainment, IVI. Such use cases are well overviewed in a
project;
 >> >> Genivi IVI layer management.
 >> >> http://projects.genivi.org/ivi-layer-management/node/13 [5]
 >> >>
 >> >> A motivation of this series and basis idea are introduced by
Ossama
 >> >> at Automotive Linux Summit 2012 spring. The series implements
 >> >> ivi-shell part. Additionally, GENIVI LM Client Library at slide
20 >> >> is
 >> >> contributed to ivi-layer-management project to support
compatible
 >> >> interfaces for Genivi Layer management users.
 >> >>
http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
[6]
 >> >>
 >> >> Before I start implementation of ivi-shell, Core members of
Genivi
 >> >> IVI layer management defined draft of ivi-shell.xml to fulfill
 >> >> requirements of IVI layer management, inviting Kristian. The
series
 >> >> also includes the ivi-shell.xml with updates I faced in actual
 >> >> implementation.
 >> >>
 >> >> Please give me any suggestions.
 >> >
 >> > Hi,
 >> >
 >> > I have understood that the whole design has been cut deep into
stone >> > a
 >> > long time ago. What are you able to change now? I do not think
it is
 >> > worth commenting on things you can no longer change, so what
aspects >> > are
 >> > you looking suggestions for?
 >> >
 >>
 >> Hi,
 >>
 >> I would like to get feedback about that if somebody has a similar
 >> motivation
 >> to support ivi as well as desktop and tablet.
 >> So it is not a stone, just a proposal. If somebody has good idea,
I
 >> would
 >> like to use it or collaborate it.
 >
 > Ok, I just had the understanding that the layer manager simply has
to
 > be a separate process and not built into the compositor. If that is
 > not the case, then that is very good news indeed. Everything that
 > manages surfaces, layers, windows, or whatever belongs into the
 > compositor process, where they are much easier to implement and you
 > don't need to introduce interfaces and IPC which are later hard to
 > develop further and cause latencies. Roundtrips and synchronous
 > calls between processes can become a difficult bottle-neck, as X11
 > has taught us. Also having too many processes becomes a real mess
 > when trying to avoid deadlocks but still keep things coherent and
 > glitch-free (see X11 server vs. window manager vs. ...).
 > So I'm roughly on the same track as Andreas Pokorny.
 >
 > Weston should become the window manager and layer manager, while
 > weston backends deal with the hardware details of compositing. For
 > example, the Raspberry Pi backend of Weston forwards all
 > compositing to the VideoCore firmware, unlike any other backend
 > who actually render a composite themselves. However, the rpi
 > backend does not forward all surfaces to VideoCore all the time,
 > but only the visible ones as needed. And Weston is the only
 > component that *can* know what is visible at any time, since Weston
 > contains the complete scene graph. Weston's internal architecture
 > is also well-suited for *automatically* deciding how to use the
 > limited hardware resources like overlays efficiently, and fall back
 > as necessary, per each output frame. You cannot do that, if you
 > tell applications about specific hardware resources.

 Yes, my understind is the same as yours.

 Great. :-)

And I expects Weston backend
 you mentioned next paragraph is ideally released as reference from
SoC
 vender.
 For exmple, some SoC vendor supports 2D blit engine which can
effciently
 composite surfaces and relese weston backend as reference.

 Tbh, I doubt vendor's abilities in producing a proper Weston
 backend, and maintaining it as Weston changes. But yeah, something
 like that.

I suspect there might be some terminology differences here.
 > Something like what IVI calls a "compositor" is a "weston backend"
 > when talking about Weston and Wayland, and IVI layer manager is
 > actually a window manager which is just a shell plugin to Weston.
 > Or am I completely off?
 >

 Yes, you are correct. And I focus on shell part to realize ivi
 requirements.

 Shell is usually the part which is primarily used by applications,
not
 supporting components of a graphical environment (see wl_shell vs.
 desktop_shell protocol interfaces). I mean the public part of
 shell, like the wl_shell interface for desktop applications.

 Which brings me to another question: how likely is it that you
 actually want to support PC desktop applications unmodified
 directly on an IVI system? And I mean on the main compositor of an
 IVI system, which I would imagine to be quite a critical component.

 I think native IVI applications could be different enough to PC
desktop
 applications, that creating a new shell interface to *replace*
 wl_shell (or whatever shell interface we will be using in the
 future for PC desktop) would be a right choice.

 If you actually do want to support PC desktop applications, I could
 see you having another, nested compositor just for supporting PC
 applications, which could run on a more restricted environment and
 maybe access to a more complex GPU which would be too risky for
 the main compositor in IVI. A sort of "untrusted" domain.

 Well, Genivi probably has already designed all that, so I'm just
 reinventing the wheel badly here.

Have you tried to map your IVI concepts of surface/layer/display to
 > Wayland wl_surface, wl_subsurface, and wl_output? I don't really
 > see what kind of interfaces your applications (Wayland clients?)
 > expect to use.

 Yes. As you comment, some use case; visibility and crop/scaling is
not
 supported now.
 So I thought starting new set of protocal to cover ivi requiremnts
would
 be better.
 But I will re-consider them and mail it back.

 Right. I replied from purely FOSS point of view. You probably have
 time and money deadlines which you must meet while creating a
 self-standing product, and in that case, I do understand going with
 a big re-invention. I understand it, but I'm not happy about it,
 although if you can publish your ad hoc approach like you do here,
 you are contributing valuable experience to the community.
 Especially, if you say something about the shortcomings of the
 design and use experiences.

 I'm very happy to see this proposal on the mailing list, even when
 I do not agree with it.

When I look at the protocol in ivi-shell.xml, I get the feeling
that:
 > - Interfaces ivi_layer, ivi_controller_surface,
 > ivi_controller_layer, ivi_controller_screen, and ivi_controller
 > should be internal implementation details inside the weston
 > process, not protocol. Having these as interfaces looks like the
 > X architecture, where the X server process and window manager
 > process continuously struggle to keep each other up-to-date, and
 > carefully try to keep state in sync (and fail), which also makes
 > races and glitches practically unavoidable.

 I have basic question to the above; strugling. wl_subsurface supports
 set_poistion now.

 set_position for sub-surfaces is *always* relative to the parent
 surface. The sub-surface position is given as a point on the parent
 surface. You still have no control where on the output any surface
 will appear, because you cannot control where the root surface of a
 tree of sub-surfaces will appear.

 If a parent surface is moved on screen, all its sub-surfaces stay
 glued to it automatically without any client interaction.

I thought it implies positioning from a windowmanger is allowed on
 weston basic concept.

 >From a window manager, yes, but this assumes that the window manager
 is in-process with weston; the window manager must be a compositor
 plugin. Making it an external process will be a huge amount of
 trouble and performance loss.

 And the protocol you propose seems to be for an external window
 manager process, from my understanding.

Each other up-to-date may occurs from window manager and weston
internl
 decision.
 Or positioning of sub surfaces is out of scope of weston and it just
 composite them according to attributes.
 I may be wrong. Please let me know history.

 The first thing is that in Wayland core protocol, there is no
 global positioning system. There are no global coordinates in the
 protocol. All coordinates that clients deal with, are relative to
 some wl_surface. (This also allows the compositor to do arbitrary
 transformations on surfaces, because there is no need for clients
 to know about them, and so no need to express transformations in
 the protocol.)

 Sub-surfaces do not change that design.

 Another thing is that with sub-surface positioning, the information
 flows strictly into one direction, and we use wl_surface.commit
 request on the parent surface to synchronize everything. That means
 that a client can manipulate a whole tree of sub-surfaces, and
 *guarantee* an atomic, flicker-free, glitch-free update on screen.

 If one needed IPC between a compositor and a window manager, you
 would either risk visual glitches as compositor first draws one
 thing before the window manager says otherwise, or jerky compositor
 performance as it needs to wait for the window manager to respond
 before it can draw anything. I don't see any way around that.

- Interface ivi_client is just a reinvention of wl_compositor and
 > wl_subcompositor.
 > - Interface ivi_surface is a reinvention of wl_surface.
 >
 > Yes, I see there are some details to may want to control like
 > surface opacity, that the current Wayland protocols do not support,
 > but I don't think that replacing everything is a good way to start.
 >
 > It is also very hard to see how objects from all these interfaces
 > are created, and how (if?) they associate to any other protocol
 > objects.
 >
 > Btw. if you need support for surface scaling and cropping, there
 > have been discussion on the Wayland mailing list to bring a crop &
 > scale protocol extension to Wayland. It is actually necessary for
 > efficient video playback etc., so pushing that forward would be
 > nice.
 >

 Thank you. I will check.

 Search for "wl_scaler", that was the working name of a proposal.

After looking through the two links you gave, the ivi-shell.xml,
 > and what you have wrote in the emails, I still have no clue what is
 > the big picture here.
 >

 I will draw a pciture to explain them. I will mail it back later.

Hi,

 I am enclosing a pdf to show relationship compositor, surface, layer,
and controller.

 As Michael Schuldt says in reply, only a process will control
attribute of surfaces and layers like position, visivility, order, and
so on.
 The controller process is kind of central one to manage policy of
layout and status of surfaces and layers by taking account into
infomation in vehicle. For example, when gear position is rear, rear
view camera would be set to the top of order and visible. When I see
your comment,
 I think wl_subsurface can be applied to layer and parent can be
mapped to layer. Now I am considering it. However, I have one use case
for ivi to control attribute of wl_surface from outside, e.g set
invisible to TV screen when speed is e.g 10km/s for safty. it is not
welcome for users but I have to consider it. Each application can do
it but on the safty point of view, it shall be handle in cetranl
controller.

 Addtionally, As Michael Schuldt says in reply as well, for debugging
purpose, controlling wl_surface would be required.

 When I see my picture by myself, ivi-shell will be window manager in
side of weston. So it might not be conflict be concept of shell.

 Please give me your feedback.

 Best regards,
 Tanibata

Cool, thanks.

I saw the terms surface, layer, etc. in the IVI docs but I didn't
really get what they are used for.

- What processes are going to use which interfaces? It looks to me
like some interfaces are not meant for all Wayland clients, but
how is it supposed to work?

- What components are in a whole IVI system, from the point of
view
of Wayland protocol? What are the responsibilities of each
component and how are these distributed into processes?

- What does a typical IVI application do in terms of Wayland
protocol? Are you using wl_compositor at all? Or any other
Wayland core interfaces?

These are just few questions to get you oriented on what kind of
things puzzle me here. Obviously, I have never been in touch with
Genivi stuff before, and I would assume most here have not
either.

The protocol you propose seems to have many references to
"id_native" and "native content", what is all this "native" stuff
about? Or all the integer id's you seem to be sending back and
forth, why can't you use real protocol objects to refer to those?


The above is the first impression from someone, who does not know
anything about the IVI architecture, but is fairly familiar with
Wayland. Sorry if it came out harsh, but I feel like I totally
missed the whole background of this proposal: why design it like
that?


Thank you again for good feedback. They are very helpfull for me.

Thank you,
pq
_______________________________________________
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]

 _______________________________________________
 wayland-devel mailing list
 wayland-devel@lists.freedesktop.org
 http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]



Links:
------
[1] http://lists.freedesktop.org/mailman/listinfo/wayland-devel
[2] 
http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
[3] 
http://lists.freedesktop.org/archives/wayland-devel/2013-August/010720.html
[4] http://cgit.freedesktop.org/wayland/weston/tree/protocol
[5] http://projects.genivi.org/ivi-layer-management/node/13
[6] 
http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
_______________________________________________
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/wayland-devel