Guillermo, Taskqueue items can only be 10K (http://code.google.com/appengine/docs/java/taskqueue/overview.html#Quotas_and_Limits). The basic idea is that if you have more data than that you put it into an entity (in the data-store) and have the task pull it out and process it. It might be that you can persist those 12K entities in a lot of large entities (that are still under 1mByte each), but that is a lot of work for something that will still probably fail. I guess it all depends where the cost is on the puts (indexing, raw writes by bytes, number of items).
And if your comeback is "memcache", well, I won't even start discussing using a non-persistent, volatile, store like that for temporary storage while you write them to the datastore... in batches using the taskqueue/cron/etc. Really, there needs to be something that can handle the write volume. On Thu, Feb 25, 2010 at 12:08 PM, Guillermo Schwarz <[email protected]> wrote: > I think there is a way to grab big chunks of oprations, put them in a > queue to be done asynchronously and that would be it. > > My take would be that using proxies it would be easy to queue any long > operation transparently. I've done that with EJBs in the past, I don't > see the reason a QueingProxy couldn't be written. > > Cheers, > Guillermo. > > On 25 feb, 17:02, Jeff Schnitzer <[email protected]> wrote: >> I don't think the original poster had a requirement for synchronous >> writes; he just didn't want to do the writes asynchronously because it >> involved a lot more code. >> >> I'm also perfectly fine with asynchronous writes and a very lax >> interpretation of consistency. I don't even mind writing extra code. >> The thing I worry about is the feasibility of a heavy write load and >> the total cost of it. >> >> Unfortunately I really can't describe in detail what I want to do (I >> normally laugh at this kind of secrecy, but in this case it's >> warranted). For the game mechanic I'm thinking about, the >> average-case scenario is not very far from the worst-case scenario. >> Just a little detail: >> >> * There is no requirement that all of a user's friends must be >> playing the game or even have installed the app to receive points. >> Welcome to the world of social gaming, you can play without even >> without knowing it! >> * There are *lots* of FB users that have > 1k friends. Probably >> millions. More active FB users are likely to have more friends... and >> more likely to use my app. >> * Points can be assigned to multiple layers, so the # of updates is >> (layers * friends). >> * Tens of thousands of people play this "game". It could become >> hundreds of thousands very soon. If I'm lucky, millions. >> >> I would love to implement this game mechanic, but I just can't. >> Asynchronous or not, it's *way* too expensive on appengine. When it >> comes time to implement this feature (and it's going to come, I can >> see the winds blowing), I'm probably going to have to move my scoring >> system out of appengine. Which is a bit ironic, because one of the >> main advantages of appengine is scalability. >> >> I would *love* to see some sort of super-super-lax and >> super-super-cheap consistency option for BigTable. Or even an >> alternative key/value datastore that simply works like a persistent >> version of memcached. Something that would let me sustain 10k >> writes/sec without bankrupting me. >> >> Jeff >> >> On Thu, Feb 25, 2010 at 11:16 AM, Ikai L (Google) <[email protected]> wrote: >> >> > Jeff, point taken, but the original poster has been asking for three >> > different requirements: >> > - requirement to do all writes synchronously >> > - sub-some-couple-hundred-millisecond writes >> > - 12k entities being written >> >> > This just won't scale well if it's common. Messaging users can be done >> > asynchronously, as can the portion crediting friends. I understand the >> > argument that you may want to do this during the lifecycle of the request >> > so >> > the original user gets some kind of feedback backed by a strongly >> > consistent >> > datastore, but this just isn't done. Feedback is usually faked out >> > optimistically, assuming that the writes will all be successful with some >> > cached layer being the only part of the stack being updated inside the >> > request. Thinking of worse case scenarios is a good thought exercise, but >> > it's also a bit too optimistic to design a product assuming all of a Users' >> > friends will play the game and engineer to meet that unrealistic >> > expectation. What are the standard and slightly non-standard use cases? I'd >> > probably look at a solution where I can store the data somewhere associated >> > with the original user for any users not already in the datastore, then >> > retrieve and generate a score for any of that User's friends on first >> > access. Facebook's developer admin tool has some pretty good statistics >> > such >> > as bounce rate, block rate and invitation accept rate that can be used to >> > tune this design. >> > Slightly off topic, but we've been asked before if it was possible to >> > provide different levels of datastore consistency. In some cases I can see >> > the tradeoffs making sense. >> >> > On Wed, Feb 24, 2010 at 5:52 PM, Jeff Schnitzer <[email protected]> >> > wrote: >> >> >> On Wed, Feb 24, 2010 at 1:06 PM, Ikai L (Google) <[email protected]> >> >> wrote: >> >> > My point wasn't necessarily that it wasn't possible. makePersistentAll >> >> > does >> >> > use a batch write, and there are definitely sites that can do 12,000+ >> >> > writes >> >> > a second (and well above that), but I don't know of any that will >> >> > attempt to >> >> > do that in a single request. While it's an interesting thought exercise >> >> > to >> >> > see if BigTable can do it through App Engine's interface (hint: it can, >> >> > globally, easily), I can't think of a single use case for a site to need >> >> > to >> >> > do this all the time and with the sub-second requirement. I think it's >> >> > reasonable to ask why this design exists and why the requirements exist >> >> > and >> >> > rethink one or the other. >> >> >> It does seem to be a pretty extreme case, but it's not all that far >> >> fetched. It's possible for a Facebook user to have 5,000 friends. >> >> Perhaps a user wants to message all 5k of them. >> >> >> I could actually use this right ability now. I would like to add a >> >> game mechanic which, when you score some points, you also credit a >> >> portion of that to all of a user's friends. Worst case scenario is a >> >> 5,000 element read followed by a 5,000 element write. I'm probably >> >> going to skip this mechanic for now because I can't afford it - even >> >> with the average 200 or so friends. If I want it badly enough, I may >> >> ultimately need to move my scoring system offsite. >> >> >> Jeff >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> >> "Google App Engine for Java" group. >> >> To post to this group, send email to >> >> [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >>http://groups.google.com/group/google-appengine-java?hl=en. >> >> > -- >> > Ikai Lan >> > Developer Programs Engineer, Google App Engine >> >http://googleappengine.blogspot.com|http://twitter.com/app_engine >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine for Java" group. >> > To post to this group, send email to >> > [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
