Re: [dev-servo] UI Workers
On Sun, Feb 22, 2015 at 3:45 AM, Robert O'Callahan wrote: > On Fri, Feb 20, 2015 at 9:11 PM, Jonas Sicking wrote: >> >> On Thu, Feb 19, 2015 at 6:27 PM, Robert O'Callahan >> wrote: >> > Should UIWorkers have access to the full Worker API? It seems like >> > there's >> > no reason not to give them that. >> >> There's two use-cases that I think argues against that. >> >> First off I'd like to enable saving a webpage for later viewing. Right >> now the best we can do is to save a pdf which means that all >> scrollable areas are lost. >> >> Saving a full page with all scripts is rarely going to work result in >> the user seeing the page they did when saving the page. >> >> What I'd like to be able to do is serialize the DOM, strip all script, >> save all linked stylesheets, images and media and change the >> references to them to use the correct relative URLs. >> >> However this would require also saving the UIWorker and enabling it to >> run in a reliable way. Otherwise the saved page won't have a working >> layout. >> >> Second, when printing, I'd like to enable the user to change page >> layout and margins and have that reflow the page and generate a >> correct layout. This means that we need to enable the UIWorker to work >> on the cloned document in a reliable way. >> >> I think both these usecases would work much better if we ensured that >> the UIWorker couldn't "interact with the outside world". > > Your use-cases already fail today because many Web pages use scroll event > handlers and JS custom layouts. UIWorkers won't make the problem any worse. I agree that it's not making the problem worse. However #1 is making the problem better, so unless we can address this, this becomes an argument for #1 over #2. > Furthermore, I think facilitating expressive performant mobile Web apps is > much more important than those use-cases, and preventing UIWorkers from > interacting with the outside world (including, I assume, the application > main thread) would severely limit their usefulness. > > Personally I think it's futile to try to save a complex app "for later > viewing" and expect anything useful to happen unless the whole app is > captured and able to operate offline. The lack of ability to save for later viewing is a big problem on mobile. The fact that native is so much better at retaining content to make it available later when the user is offline is one of the big reasons that the web is having a hard time competing with native. So I do think that this is an important problem to solve. / Jonas ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
A couple thoughts from the perspective of a web app dev who has struggled with this problem: I get excited when I hear a proven solution with well-understood scope like “animations and user interactions on the same thread”. I get nervous about ambitious and unknown solutions (async DOM). The web needs a pragmatic fix for animation and interaction and it needs it yesterday. On February 23, 2015 at 10:37:20 , Jonas Sicking (jo...@sicking.cc) wrote: The fact that native is so much better at retaining content to make it available later when the user is offline is one of the big reasons that the web is having a hard time competing with native. ServiceWorkers give us a powerful way to serialize state for offline persistence on a per-app basis. This is also essentially the approach native apps take, right? Your app can persist to a local DB, or not. You can persist via iCloud, or not. It’s up to you to serialize. Magic persistence for complex apps sounds mythical. I want to be wrong about this. --- Gordon Brander Sr Design Strategist Mozilla ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Tue, Feb 24, 2015 at 8:56 AM, Jonas Sicking wrote: > On Mon, Feb 23, 2015 at 10:56 AM, Gavin Sharp > wrote: > > What does it mean to "save your for later viewing"? > > In gmail it would mean saving the set of emails that you are currently > looking at. > > For facebook it would mean the news-feed content that's currently on > the screen, or the event invitation details that you are currently > looking at. > > That's what we would get if we serialized the current DOM+CSS+images > to disk without any additional smarts. And all of these seem useful > for users. > If we serialize the current DOM + CSS + images and remove the scripts, we will present something to the user which is little more than a screenshot. Users will immediately try to interact with it and discover that's broken. That's a bad experience. To present these in a non-broken way we must run page scripts. We should not try to restrict the functionality of Web APIs for the sake of avoiding this. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Tue, Feb 24, 2015 at 7:36 AM, Jonas Sicking wrote: > On Sun, Feb 22, 2015 at 3:45 AM, Robert O'Callahan > wrote: > > Your use-cases already fail today because many Web pages use scroll event > > handlers and JS custom layouts. UIWorkers won't make the problem any > worse. > > I agree that it's not making the problem worse. However #1 is making > the problem better, so unless we can address this, this becomes an > argument for #1 over #2. > We can still do #1 as well as #2. It's a matter of priorities, and I think enabling richer performant mobile apps is much higher priority than enabling more scroll effects in saved Web pages. > Furthermore, I think facilitating expressive performant mobile Web apps is > > much more important than those use-cases, and preventing UIWorkers from > > interacting with the outside world (including, I assume, the application > > main thread) would severely limit their usefulness. > > > > Personally I think it's futile to try to save a complex app "for later > > viewing" and expect anything useful to happen unless the whole app is > > captured and able to operate offline. > > The lack of ability to save for later viewing is a big problem on mobile. > > The fact that native is so much better at retaining content to make it > available later when the user is offline is one of the big reasons > that the web is having a hard time competing with native. > > So I do think that this is an important problem to solve. > Echoing Gordon and Gavin: saving the DOM+CSS is totally unlike the way native apps work and not an adequate solution to this problem. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Mon, Feb 23, 2015 at 12:07 PM, Robert O'Callahan wrote: > On Tue, Feb 24, 2015 at 8:56 AM, Jonas Sicking wrote: >> >> On Mon, Feb 23, 2015 at 10:56 AM, Gavin Sharp >> wrote: >> > What does it mean to "save your for later viewing"? >> >> In gmail it would mean saving the set of emails that you are currently >> looking at. >> >> For facebook it would mean the news-feed content that's currently on >> the screen, or the event invitation details that you are currently >> looking at. >> >> That's what we would get if we serialized the current DOM+CSS+images >> to disk without any additional smarts. And all of these seem useful >> for users. > > > If we serialize the current DOM + CSS + images and remove the scripts, we > will present something to the user which is little more than a screenshot. > Users will immediately try to interact with it and discover that's broken. > That's a bad experience. > > To present these in a non-broken way we must run page scripts. We should not > try to restrict the functionality of Web APIs for the sake of avoiding this. I think this is letting perfect get in the way of good. It's going to take many years before a majority of the websites that a given user uses to have made themselves "offlineable". I think waiting for that long is going to add a lot of risk to the web's chance of surviving the switch to mobile. / Jonas ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
It’s funny: I have come to the opposite conclusion for the same reason. The Good: getting 60fps interactions and animations in web apps using a proven approach (UI and interaction thread). The Ideal: also automatically serializing those apps for offline use. While I very much want the ideal to exist, I wouldn’t be willing to risk the “good” for it. Or perhaps it is possible to have it all. I don’t understand the tradeoffs deeply enough to know. If I had to prioritize, I would say that interactions and animations are the basis of competition for UI and will be as long as we’re using touch. We lag way behind there. With Service Workers, we now have parity in offline capability with native apps (serialize it yourself), even if practical usage lags behind. --- Gordon Brander Sr Design Strategist Mozilla On February 23, 2015 at 13:21:55 , Jonas Sicking (jo...@sicking.cc(mailto:jo...@sicking.cc)) wrote: > On Mon, Feb 23, 2015 at 12:07 PM, Robert O'Callahan > wrote: > > On Tue, Feb 24, 2015 at 8:56 AM, Jonas Sicking wrote: > >> > >> On Mon, Feb 23, 2015 at 10:56 AM, Gavin Sharp > >> wrote: > >> > What does it mean to "save your for later viewing"? > >> > >> In gmail it would mean saving the set of emails that you are currently > >> looking at. > >> > >> For facebook it would mean the news-feed content that's currently on > >> the screen, or the event invitation details that you are currently > >> looking at. > >> > >> That's what we would get if we serialized the current DOM+CSS+images > >> to disk without any additional smarts. And all of these seem useful > >> for users. > > > > > > If we serialize the current DOM + CSS + images and remove the scripts, we > > will present something to the user which is little more than a screenshot. > > Users will immediately try to interact with it and discover that's broken. > > That's a bad experience. > > > > To present these in a non-broken way we must run page scripts. We should not > > try to restrict the functionality of Web APIs for the sake of avoiding this. > > I think this is letting perfect get in the way of good. > > It's going to take many years before a majority of the websites that a > given user uses to have made themselves "offlineable". I think waiting > for that long is going to add a lot of risk to the web's chance of > surviving the switch to mobile. > > / Jonas > ___ > dev-platform mailing list > dev-platf...@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Tue, Feb 24, 2015 at 10:57 AM, Gordon Brander wrote: > It’s funny: I have come to the opposite conclusion for the same reason. > > The Good: getting 60fps interactions and animations in web apps using a > proven approach (UI and interaction thread). > The Ideal: also automatically serializing those apps for offline use. > I agree with this. Also, there is a way to get "save for later viewing" to work with complex apps: serialize all the application state --- DOM, CSS, JS heap, workers, etc --- and revive it later, possibly in a jail that blocks it from accessing the network when revived, but possibly not. A lot of work to implement, but not terribly difficult, and it would give you a great experience, without restricting (explicitly or implicitly) what Web apps can do. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Mon, Feb 23, 2015 at 3:02 PM, Robert O'Callahan wrote: > On Tue, Feb 24, 2015 at 10:57 AM, Gordon Brander > wrote: >> >> It's funny: I have come to the opposite conclusion for the same reason. >> >> The Good: getting 60fps interactions and animations in web apps using a >> proven approach (UI and interaction thread). >> The Ideal: also automatically serializing those apps for offline use. > > > I agree with this. > > Also, there is a way to get "save for later viewing" to work with complex > apps: serialize all the application state --- DOM, CSS, JS heap, workers, > etc --- and revive it later, possibly in a jail that blocks it from > accessing the network when revived, but possibly not. I think this would fall over more often than not. Most developers will not write their code to be resilient in the face of being suspended for extended periods of time. Upon reopening they would likely display error dialogs, or updated version of whatever was saved. In fact, I would argue that it's directly harmful to keep even some of the app logic working. It could cause things like overwriting server state with whatever state was serialized into the saved copy. The usecase here isn't to magically make apps available offline. That's unlikely to ever work well enough that it's worth doing without author cooperation. The use case here is the ability to save some *information* for later access. If I'm about to leave house to visit a friend, i'm interested in saving the directions for getting to that friend. I'm less interested in having an updated facebook feed with me. / Jonas ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
Le 24 févr. 2015 à 08:02, Robert O'Callahan a écrit : > Also, there is a way to get "save for later viewing" to work with complex > apps: serialize all the application state --- DOM, CSS, JS heap, workers, > etc --- and revive it later, possibly in a jail that blocks it from > accessing the network when revived, but possibly not. Some use cases for this: * The very useful scrapbook Add-on. https://addons.mozilla.org/en-US/firefox/addon/scrapbook/ * The crash or quit your browser and restart (with 100 tabs) and out of the network. We want to be able to see all the tabs content before the browser has been closed. * A "browser time machine", aka the possibility to save dated copies of the same site ala time machine backup, or web.archive.org or http://timetravel.mementoweb.org/about/ or google street view and it's timeline feature (top right these days). -- Karl Dubost, Mozilla http://www.la-grange.net/karl/moz ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Tue, Feb 24, 2015 at 12:09 PM, Jonas Sicking wrote: > I think this would fall over more often than not. > > Most developers will not write their code to be resilient in the face > of being suspended for extended periods of time. Upon reopening they > would likely display error dialogs, or updated version of whatever was > saved. > > In fact, I would argue that it's directly harmful to keep even some of > the app logic working. It could cause things like overwriting server > state with whatever state was serialized into the saved copy. > You missed the part about the jail. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Mon, Feb 23, 2015 at 4:01 PM, Robert O'Callahan wrote: > On Tue, Feb 24, 2015 at 12:09 PM, Jonas Sicking wrote: >> >> I think this would fall over more often than not. >> >> Most developers will not write their code to be resilient in the face >> of being suspended for extended periods of time. Upon reopening they >> would likely display error dialogs, or updated version of whatever was >> saved. >> >> In fact, I would argue that it's directly harmful to keep even some of >> the app logic working. It could cause things like overwriting server >> state with whatever state was serialized into the saved copy. > > You missed the part about the jail. Sure, the jail would prevent server overwrites. But there's still problems like code behaving strange due to being suspended so long, for example by removing content from the screen that is "out of date". Additionally the jail is more likely to cause error dialogs to appear due to failed network requests. I would also argue that keeping a lot of the page "working" by running scripts, is going to be even more confusing when that page is then unable to actually save anything due to the jail. Rather than nuking all the scripts and ensuring that the only thing the user can do is scroll around on the page. / Jonas ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
> 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. > > How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. This sounds great from Servo's point of view. I'd like to suggest that we allow browsers to expand the set of properties that can be set off the main thread beyond the set that are considered "compositor" or "accelerated" properties today. (This was my main issue with Google's "AnimationProxy": it exposed an API tailored only to those properties that could be handled without a round trip through layout in current browser architectures, but browser engines that have off-main-thread layout like Servo need not restrict themselves to those.) CSS transitions as implemented in my work-in-progress Servo branch go through a "back door" into layout that allows them to directly poke property values into the layout data structures without ever looking at the DOM (i.e. without going through the "front door" of CSS style recalc), which allows script to run concurrently with all CSS transitions. Since we went through such pains to make this "back door" work in Servo, it seems desirable to expose it directly to Web authors, and UIWorker sounds like as reasonable an approach as any. Patrick On Thu, Feb 19, 2015 at 6:27 PM, Robert O'Callahan wrote: > Last week in Sydney I spent a lot of time talking to Chrome devs about > different approaches for 60fps effects in Web pages. There are three > different kinds of approaches being discussed (so far): > 1) Apple's animation-timeline proposal, which lets CSS animations use > scroll position as an input instead of time. > 2) UIWorker: some kind of JS worker that receives callbacks during > composition; each callback can take inputs such as time and scroll > position(s) as inputs and can update certain CSS properties (e.g. > transforms, opacity) on elements that the compositor then uses. > 3) Provide a way for pages to turn off async scrolling and make everything > fast enough (and isolated enough) for pages to do 60fps updates from their > main thread. > > All of these approaches have problems. Approach #1 is much more limited in > its expressiveness than the alternatives. Approach #3 is more fragile and > less composable than the alternatives --- sharing your main thread with any > JS you don't control could cause jank. I like #2. It's strictly more > powerful than #1 without the downsides of #3. > > Obvious question: how do we stop UIWorkers janking the compositor? We could > give them a time budget (say 8ms). If a worker blows its budget, we notify > it by sending it an event, we give up on it and composite anyway, and we > run it separately from the compositor for a while. This requires an API > design that lets UIWorkers still work, with some lag, when the compositor > is not blocking on them, but that seems doable. > > Should UIWorkers have access to the full Worker API? It seems like there's > no reason not to give them that. > > How should we explain the CSS effects of UIWorkers? A promising idea is to > extend the Web Animations API to allow adding a new kind of animation > effect to DOM elements --- a UIWorker-controlled effect. Essentially the > UIWorker would then be responsible for computing the output of the timing > function in each frame. The UIWorker could then animate *any* CSS property, > though most property updates would require a round trip through main thread > layout/rendering before they get rendered. > > One good thing about UIWorkers is extensibility. We can imagine providing > touch input coordinates to UIWorkers to enable 60fps object dragging (with > arbitrary effects like resistance, snapping, etc). UIWorkers could render > to canvases: this would let you render VR with minimum latency, and let you > render to canvases used by CSS masking for 60fps dissolves and clipping > effects. If you really want to, you could go all Flipboard and render your > entire UI to a canvas in the compositor --- if you keep hitting your > deadlines. > > I like the idea of doing #2 before either #1 or #3. > > Rob > -- > oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo > owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo > osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo > owohooo > osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o > oioso >
