#33636: BulkProcessMixin on models.Model
-------------------------------------+-------------------------------------
Reporter: Myung Eui Yoon | Owner: Myung Eui
| Yoon
Type: New feature | Status: assigned
Component: Database layer | Version: 4.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk, model | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Myung Eui Yoon:
Old description:
> New Feature for convinient context manager for bulk create/update.
> Just inherit MoModelBulkProcessMixin on Model class.
>
> {{{
> class BulkyModel(models.Model, ModelBulkProcessMixin):
> id = models.BigAutoField(primary_key=True)
> name = models.CharField(max_length=10, null=False)
>
> names = [f"name-{num}" for num in range(100_100_000)]
>
> # Case : Raw bulk_create
> objs = [BulkyModel(name=name) for name in names]
> BulkyModel.objects.bulk_create(
> objs
> ) # We should maintain big array size and this leades to OOM
> error
>
> # Case : Chunked bulk_create
> objs = list()
> for name in names:
> obj = BulkyModel(name=name)
> objs.append(obj)
> if len(objs) > 10_1000:
> BulkyModel.objects.bulk_create(objs)
> objs.clear()
> if len(objs) > 0:
> BulkyModel.objects.bulk_create(objs)
> objs.clear()
>
> # Case : With ModelBulkProcessMixin
> with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
> for name in names:
> bulk.add(BulkyModel(name=name))
> }}}
>
> https://github.com/django/django/pull/15577
New description:
New Feature for convinient context manager for bulk create/update.
Just inherit MoModelBulkProcessMixin on Model class.
{{{
class BulkyModel(models.Model, ModelBulkProcessMixin):
id = models.BigAutoField(primary_key=True)
name = models.CharField(max_length=10, null=False)
names = [f"name-{num}" for num in range(100_100_000)]
# Case : Raw bulk_create
objs = [BulkyModel(name=name) for name in names]
BulkyModel.objects.bulk_create(
objs
) # We should maintain big array size and this leades to OOM
error
# Case : Chunked bulk_create
objs = list()
for name in names:
obj = BulkyModel(name=name)
objs.append(obj)
if len(objs) > 10_1000:
BulkyModel.objects.bulk_create(objs)
objs.clear()
if len(objs) > 0:
BulkyModel.objects.bulk_create(objs)
objs.clear()
# Case : With ModelBulkProcessMixin
with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
for name in names:
bulk.add(BulkyModel(name=name))
}}}
https://github.com/django/django/pull/15578
--
--
Ticket URL: <https://code.djangoproject.com/ticket/33636#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/01070180191ce02e-f3b918f4-116c-433f-a105-9dcb8eb16f5f-000000%40eu-central-1.amazonses.com.
