I think there is a way to grab big chunks of oprations, put them in a queue to be done asynchronously and that would be it.
My take would be that using proxies it would be easy to queue any long operation transparently. I've done that with EJBs in the past, I don't see the reason a QueingProxy couldn't be written. Cheers, Guillermo. On 25 feb, 17:02, Jeff Schnitzer <[email protected]> wrote: > I don't think the original poster had a requirement for synchronous > writes; he just didn't want to do the writes asynchronously because it > involved a lot more code. > > I'm also perfectly fine with asynchronous writes and a very lax > interpretation of consistency. I don't even mind writing extra code. > The thing I worry about is the feasibility of a heavy write load and > the total cost of it. > > Unfortunately I really can't describe in detail what I want to do (I > normally laugh at this kind of secrecy, but in this case it's > warranted). For the game mechanic I'm thinking about, the > average-case scenario is not very far from the worst-case scenario. > Just a little detail: > > * There is no requirement that all of a user's friends must be > playing the game or even have installed the app to receive points. > Welcome to the world of social gaming, you can play without even > without knowing it! > * There are *lots* of FB users that have > 1k friends. Probably > millions. More active FB users are likely to have more friends... and > more likely to use my app. > * Points can be assigned to multiple layers, so the # of updates is > (layers * friends). > * Tens of thousands of people play this "game". It could become > hundreds of thousands very soon. If I'm lucky, millions. > > I would love to implement this game mechanic, but I just can't. > Asynchronous or not, it's *way* too expensive on appengine. When it > comes time to implement this feature (and it's going to come, I can > see the winds blowing), I'm probably going to have to move my scoring > system out of appengine. Which is a bit ironic, because one of the > main advantages of appengine is scalability. > > I would *love* to see some sort of super-super-lax and > super-super-cheap consistency option for BigTable. Or even an > alternative key/value datastore that simply works like a persistent > version of memcached. Something that would let me sustain 10k > writes/sec without bankrupting me. > > Jeff > > On Thu, Feb 25, 2010 at 11:16 AM, Ikai L (Google) <[email protected]> wrote: > > > Jeff, point taken, but the original poster has been asking for three > > different requirements: > > - requirement to do all writes synchronously > > - sub-some-couple-hundred-millisecond writes > > - 12k entities being written > > > This just won't scale well if it's common. Messaging users can be done > > asynchronously, as can the portion crediting friends. I understand the > > argument that you may want to do this during the lifecycle of the request so > > the original user gets some kind of feedback backed by a strongly consistent > > datastore, but this just isn't done. Feedback is usually faked out > > optimistically, assuming that the writes will all be successful with some > > cached layer being the only part of the stack being updated inside the > > request. Thinking of worse case scenarios is a good thought exercise, but > > it's also a bit too optimistic to design a product assuming all of a Users' > > friends will play the game and engineer to meet that unrealistic > > expectation. What are the standard and slightly non-standard use cases? I'd > > probably look at a solution where I can store the data somewhere associated > > with the original user for any users not already in the datastore, then > > retrieve and generate a score for any of that User's friends on first > > access. Facebook's developer admin tool has some pretty good statistics such > > as bounce rate, block rate and invitation accept rate that can be used to > > tune this design. > > Slightly off topic, but we've been asked before if it was possible to > > provide different levels of datastore consistency. In some cases I can see > > the tradeoffs making sense. > > > On Wed, Feb 24, 2010 at 5:52 PM, Jeff Schnitzer <[email protected]> wrote: > > >> On Wed, Feb 24, 2010 at 1:06 PM, Ikai L (Google) <[email protected]> > >> wrote: > >> > My point wasn't necessarily that it wasn't possible. makePersistentAll > >> > does > >> > use a batch write, and there are definitely sites that can do 12,000+ > >> > writes > >> > a second (and well above that), but I don't know of any that will > >> > attempt to > >> > do that in a single request. While it's an interesting thought exercise > >> > to > >> > see if BigTable can do it through App Engine's interface (hint: it can, > >> > globally, easily), I can't think of a single use case for a site to need > >> > to > >> > do this all the time and with the sub-second requirement. I think it's > >> > reasonable to ask why this design exists and why the requirements exist > >> > and > >> > rethink one or the other. > > >> It does seem to be a pretty extreme case, but it's not all that far > >> fetched. It's possible for a Facebook user to have 5,000 friends. > >> Perhaps a user wants to message all 5k of them. > > >> I could actually use this right ability now. I would like to add a > >> game mechanic which, when you score some points, you also credit a > >> portion of that to all of a user's friends. Worst case scenario is a > >> 5,000 element read followed by a 5,000 element write. I'm probably > >> going to skip this mechanic for now because I can't afford it - even > >> with the average 200 or so friends. If I want it badly enough, I may > >> ultimately need to move my scoring system offsite. > > >> Jeff > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine for Java" group. > >> To post to this group, send email to > >> [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine-java?hl=en. > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > >http://googleappengine.blogspot.com|http://twitter.com/app_engine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
