How to match scipy.spatial.distance results back to pandas df
I am trying to find points in the activity that are near points in segment. I get the results from distance() and what to identify the points in segment and activity that are within some value of each other .0001 for example. One idea I had was to append each array row (or column) to the corresponding row in the df after converting it to a dict so that I can lookup the point in the other df. Any suggestions, ideas? from scipy.spatial import distance a = distance.cdist(segment[['Latitude', 'Longitude']], activity[['Latitude', 'Longitude']], 'euclidean') b = a[a < .0001] b array([8.83911760e-05, 6.31347765e-05, 3.89486842e-05, 2.13775583e-05, 2.10950231e-05, 4.10487515e-05, 6.7000e-05, 9.10878697e-05, 7.61183289e-05, 9.90050504e-05, 7.88162420e-05, 5.90931468e-05, 4.50111097e-05, 4.97393205e-05, 6.78969808e-05, 8.52115016e-05, ... Thanks Vincent Davis 720-301-3003 *Want to get a hold of me?* *SMS: awesome.phone: ok...* *email: bad!* -- https://mail.python.org/mailman/listinfo/python-list
Re: A rule for your twitlist/mailing list
On 7/14/20 9:51 PM, Grant Edwards wrote: On 2020-07-15, Cameron Simpson wrote: On 14Jul2020 08:49, Nomen Nescio wrote: Is the mailing list for comp.lang.python still open? If you mean the python-list mailing list, yes. It is what I use, and it does not suffer from the spam you describe. Here: https://mail.python.org/mailman/listinfo/python-list It gateways with the newsgroup, but has much better spam qualities. And if you prefer to read the mailing list with an NNTP client, you can point your favorite newsreader at news.gmane.org and subscribe to gmane.comp.python.general -- Grant I think that should now be news.gmane.io, at least that's how I get comp.python. I think gmane.org shut down. Regards, Jim -- https://mail.python.org/mailman/listinfo/python-list
Re: frozendict: an experiment
On Wed, 15 Jul 2020 at 08:07, Inada Naoki wrote:
> I don't think so. The view objects are useful when we need a set-like
> operation. (e.g. `assert d.keys() == {"spam", "egg"}`)
Yes, but, instead of creating a view, you can create and cache the
pointer of a "real" object, that implements the dict view API.
For example, keys for a frozendict could be an "ordered" frozenset.
This "oset" could be a frozendict, which values are the keys and the
keys are the key hashes (as set).
On Wed, 15 Jul 2020 at 08:07, Inada Naoki wrote:
> There is no difference between mutable and immutable dicts.
There's a huge difference, and it's in the contract. As I said, if we
assume that a frozendict is immutable, we can optimize its speed.
Furthermore, currently no real functional programming can be done in
Python with dicts.
--
https://mail.python.org/mailman/listinfo/python-list
Re: frozendict: an experiment
On Thu, 16 Jul 2020 at 06:11, Inada Naoki wrote:
> On Thu, Jul 16, 2020 at 2:32 AM Marco Sulla
> wrote:
> > Yes, but, instead of creating a view, you can create and cache the
> > pointer of a "real" object, that implements the dict view API.
> > For example, keys for a frozendict could be an "ordered" frozenset.
> > This "oset" could be a frozendict
> I am not sure about what are you saying.
> Does it really solve the usage of dict views?
> How about my example? (`assert d.keys() == {"spam", "egg"}`)
Well, if the frozendict "views" will implement the full API of the
dict views, your assert and all the existing code will continue to
work.
The difference will be that views *could* be faster.
> > On Wed, 15 Jul 2020 at 08:07, Inada Naoki wrote:
> Oh, ! am talking about dict views. I meant there is no difference
> between "how dict view is useful for dict" and "how dict view is
> useful for frozendict".
>
> But I am still not sure about the optimizations and functional
> programming you are talking about.
> Please elaborate more, please?
About frozendict optimization
an immutable dict could be optimized in different ways in Python:
1. as I said, views can be replaced with "real", cached objects. This
will speed up subsequent calls to keys() etc.
2. as tuples and strings, small frozendicts can be interned in a
cache. The same cache can be used to return a cached frozendict
instead of recreate it, as tuples.
3. many python internals uses a mapping proxy to a dict, to avoid its
modification. A frozendict can be used instead.
4. dicts can use a split table or a combined table. This is useful for
maintaining insertion order upon deletion and insertion.
This is not required for frozendicts, since once created they can't
be modified. frozendicts can be all split or all combined dicts.
So in theory frozendicts could be created with the max useful size,
without resizing them, add the keys and values and then remove the
unused.
This could speed up creation.
About functional programming
Well, pure functional programming requires no side effects:
https://en.wikipedia.org/wiki/Functional_programming
Object mutability is a side effect. Pure functional programming uses
only immutable objects.
In Python, pure functional programming is impossible, since even
functions are (partially) mutable objects. But Python has a lot of
functional programming features:
https://docs.python.org/3/howto/functional.html
Every builtin mutable data type has its immutable counterpart now,
dict apart. And types.MappingProxyType is not usable in real world,
since it's really slow.
--
https://mail.python.org/mailman/listinfo/python-list
Re: A rule for your twitlist/mailing list
On 2020-07-15, Jim wrote: > On 7/14/20 9:51 PM, Grant Edwards wrote: >> On 2020-07-15, Cameron Simpson wrote: >>> On 14Jul2020 08:49, Nomen Nescio wrote: >> Is the mailing list for comp.lang.python still open? >>> >>> If you mean the python-list mailing list, yes. It is what I use, and it >>> does not suffer from the spam you describe. Here: >>> >>> https://mail.python.org/mailman/listinfo/python-list >>> >>> It gateways with the newsgroup, but has much better spam qualities. >> >> And if you prefer to read the mailing list with an NNTP client, you >> can point your favorite newsreader at news.gmane.org and subscribe to >> gmane.comp.python.general > > I think that should now be news.gmane.io, at least that's how I get > comp.python. I think gmane.org shut down. You're right. I was looking at an old entry in my slrn config file. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: frozendict: an experiment
On Fri, Jul 17, 2020 at 2:16 AM Marco Sulla
wrote:
>
> On Thu, 16 Jul 2020 at 06:11, Inada Naoki wrote:
> > On Thu, Jul 16, 2020 at 2:32 AM Marco Sulla
> > wrote:
> > > Yes, but, instead of creating a view, you can create and cache the
> > > pointer of a "real" object, that implements the dict view API.
> > > For example, keys for a frozendict could be an "ordered" frozenset.
> > > This "oset" could be a frozendict
> > I am not sure about what are you saying.
> > Does it really solve the usage of dict views?
> > How about my example? (`assert d.keys() == {"spam", "egg"}`)
>
> Well, if the frozendict "views" will implement the full API of the
> dict views, your assert and all the existing code will continue to
> work.
> The difference will be that views *could* be faster.
>
Then, there is no reason to not support the view for frozendict?
> > > On Wed, 15 Jul 2020 at 08:07, Inada Naoki wrote:
> About frozendict optimization
>
> an immutable dict could be optimized in different ways in Python:
> 1. as I said, views can be replaced with "real", cached objects. This
> will speed up subsequent calls to keys() etc.
I don't think it is an interesting optimization.
> 2. as tuples and strings, small frozendicts can be interned in a
> cache. The same cache can be used to return a cached frozendict
> instead of recreate it, as tuples.
I am not sure tuple is "internined" or just "merged". (I don't know precise
definition of the "interning").
Tuples are merged while compiling.
```
for a in ["foo", "bar"]: # compiler convert this list to tuple
...
for b in ("foo", "bar"): # and compiler merge same tuple
...
```
But when frozendicts are merged?
I think there is a very little chance.
> 3. many python internals uses a mapping proxy to a dict, to avoid its
> modification. A frozendict can be used instead.
Are they used frequently in performance critical path?
Could you point some concrete examples?
> 4. dicts can use a split table or a combined table. This is useful for
> maintaining insertion order upon deletion and insertion.
>This is not required for frozendicts, since once created they can't
> be modified. frozendicts can be all split or all combined dicts
Key-sharing dict vs regular dict are not relating to insertion order.
Key-sharing dict is implemented before insertion order keeping dict.
And when I implemented order preserving dict, key-sharing dict is very
difficult to maintain.
I even wanted to drop key-sharing dict support.
I'm OK to all combined dict for frozen dict. But I think key-sharing is still
interesting optimization for frozen dict. And supporting key-sharing dict
is much easier for frozendict than for mutable dict.
>
> About functional programming
>
> Well, pure functional programming requires no side effects:
> https://en.wikipedia.org/wiki/Functional_programming
> Object mutability is a side effect. Pure functional programming uses
> only immutable objects.
> In Python, pure functional programming is impossible, since even
> functions are (partially) mutable objects. But Python has a lot of
> functional programming features:
> https://docs.python.org/3/howto/functional.html
> Every builtin mutable data type has its immutable counterpart now,
> dict apart. And types.MappingProxyType is not usable in real world,
> since it's really slow.
I feel this motivation is too theorical. And theorically, I don't think Python
should add bultin type for pure functional programming. Python should
focus only Pytohnic programming.
Would you provide some concrete examples how frozendict can improve
Python code significantly?
--
Inada Naoki
--
https://mail.python.org/mailman/listinfo/python-list
excel (multiple sheets) to yml file for each sheet
Hi, I have excel file with multiple sheets and need output as yml file for each sheet. could someone help me with python code? following is an example: aep sheet: aepaep_description infra_vlan test_AEP test aepno aeps_to_domain sheet: aep domaindomain_type test_AEPtest_PHY_DOMphys test1_AEP test1_PHY_DOM l3dom scription should output two files: aeps-vars.yml: aeps: - aep: test_AEP aep_description: test aep infra_vlan: no aeps_to_domain-vars.yml: aeps_to_domain: - aep: test_AEP domain: test_PHY_DOM domain_type: phys - aep: test_AEP domain: test_L3O_DOM domain_type: l3dom -- https://mail.python.org/mailman/listinfo/python-list
SCons 4.0.1 Released
A new SCons release, 4.0.1, is now available on the SCons download page: https://scons.org/pages/download.html Here is a summary of the changes since 4.0.1: NEW FUNCTIONALITY - Added Environment() variable TEMPFILEDIR which allows setting the directory which temp files createdby TEMPFILEMUNGE are created in. DEPRECATED FUNCTIONALITY - N/A CHANGED/ENHANCED EXISTING FUNCTIONALITY - N/A FIXES - Fix fortran tools to set SHFORTRAN variables to $FORTRAN, similarly SHF77, SHF90, SHF95, SHF03 and SHF08 will default to the variables $F77, $F90, $F95, $F03 and $F08 respectively. If you were depending on changing the value of FORTRAN (or $F[0-9][0-9]) having no effect on the value of SHFORTRAN, this change will break that. The values of FORTRAN, F77, F90, F95, F03, F08 and SHFORTRAN, SHF77 (etc.) now are not overridden in generate if alredy set by the user. - Fix subprocess execution of 'lslpp' on AIX to produce text standard i/o. - Re-do the fix for suncxx tool (Oracle Studio compiler) now that only Python 3 is supported, to avoid decoding errors. IMPROVEMENTS - N/A PACKAGING - N/A DOCUMENTATION - N/A DEVELOPMENT - N/A Thanks to the following contributors listed below for their contributions to this release. git shortlog --no-merges -ns 4.0.0..HEAD 12 William Deegan 2 Rob Boehne 1 Clemens Tolboom -- https://mail.python.org/mailman/listinfo/python-list
