Dawid,
First, I have tried the patch against git head 5.0 branch with no
apparent change in behavior for me with two Merrill Lynch investment
accounts and one Bank of America credit card. I use PAYEEID for Payee,
so I didn't expect any changes. Let me know if I missed something.
Responses to direct questions inline below.
On 1/24/20 10:45 PM, Dawid Wrobel wrote:
Thank you both.
Jack, correct me if I am wrong, but isn't the matched Payee
replacement behavior you described typical across the board for all of
the plugins? From what I noticed so far, Payee matching happens
independently in
MyMoneyStatementReader::processTransactionEntry() after the importing
plugin returns a list of MyMoneyStamtents.
I do believe you are correct. My problem is clearly different from the
one you were solving, although at least somewhat related, in terms of
which OFX fields end up in which KMM Transaction fields.
However, I do agree that the current matching is somewhat limiting,
since it indeed replaces the original transaction's payee field. And
it is related to a different limitation I noticed so far: I wish it
was possible to specify Regex rules for Payee matching using any
field, not just the actual /Payee/.
I had thought it matched against whichever field you told KMM to use for
Payee.
Then, depending on the match, user should be allowed to decide which
of the matches is to be used for which field. A similar option exists
now in OFX Direct Connect plugin, but it is limited to Payee and Memo
matching only.
Ah - this is what I'm using, so it's why I though the above. However, I
don't think I've noticed any different behavior when importing an OFX
file. Do I need to test better?
This is useful and it would be even better if it was possible to:
* make this work for all importers, by defining these matching rules
per Payee in /Payees/ view, not separately in plugin's settings.
* use any of the fields, not just /Payee/ and /Memo/.
o Specifically, each of the importing plugins should be allowed
to add an arbitrary list of key:value fields to
MyMoneyStatement object
o These could then be used for a more advanced Payee matching, e.g.:
+ Match 1: key: /MEMO/, match method: /Regex/, match
value:///(<Memo>Online transfer.*/)
+ Match 2: key: /NAME/, match method: /Regex/: match value:
/Payee: (<Payee>.*)/
+ ...
+ Match n: key:/ANY/, match method: /Exact/, match value:
/SOMESPECIFICVALUE/
* You could use the named groups (<Payee>, <Memo>)to force the
assignment of the matches to actual UI fields, /or skip assignment
altogether/ and default to current behavior on a second thought,
this is probably too complicated to an average Joe, so a more
explicit UI for assignment could be used instead.
o ideally, it should be possible to compose Payee and Memo from
more than one match, e.g. /Payee: "\1 \2"/. Believe it or not,
but this is how JPMorgan CHASE bank's QFX file looks like:
<STMTTRN>
<TRNTYPE>DEBIT
<DTPOSTED>20200110120000[0:GMT]
<TRNAMT>-5000.00
<FITID>202001100
<NAME>*Online Realtime Transfer to Citi*
<MEMO>*bank Checking 1234 transaction#*
</STMTTRN>
* it would probably also be useful to store the original key:value
metadata for future inspection
Ouch. And I though I had issues with Merril Lynch. :-)
I do understand this may sound somewhat complicated, but almost
identical functionality already exists in OFX Direct Connect plugin,
so it would be moved to where I think it makes more sense and be used
by QIF- and CSV-imported transactions, too.
I'll have to leave that to Thomas, but my need still sounds a bit
orthogonal, and is just that if the Payee used for the imported
transaction is found by matching (anything other then being exactly the
same as whatever imported field is used) the keep the original field as
part of the memo, so as not to lose the actual original payee. (In my
case, for example, I have a payee "Misc Restaurants" and every
restaurant I eat at more than once or twice is listed on the matching tab.
From the UI perspective, the way I see it, the current "List of
matching names" option's view could be modified to make it more
detailed – especially that it already supports regex matching which
one cannot really guess from the UI itself. While I don't have a
mockup handy, I can see how this can be designed in a clean way. With
proper user manual updates, it would really make transaction matching
a super convenient and powerful tool.
You'll have to discuss the code with Thomas, but I take care of the
manual, so that's actually easy for me to update at the same time as the
code. (I know there are still outstanding edits to bring the manual
fully up to date with current 5.0.x release.)
PS. Jack, I believe you replied outside of mailing list by mistake, so
I CC-ed kmymoney-devel again.
It was intentional, because I believe it was only a place holder until I
could test. It's fine that the discussion here Is back on the list. (I
also email oddities not worth getting into here, but I wasn't sure I
would be able to reply to the list until I got home from being away for
the weekend. Things worked out for my laptop, fortunately.)
Jack
On Fri, Jan 24, 2020 at 12:41 PM Jack <ostrof...@users.sourceforge.net
<mailto:ostrof...@users.sourceforge.net>> wrote:
I'll be glad to check. I might get time over the weekend,
otherwise it
will be early next week.
By the way - I have also noticed that the Payee field (used for
Payee)
is NOT copied into imported transactions, so if I match many
different
restaurants into the Misc Restaurant payee, I don't know which on it
actually was. For now, I've been manually adding it back when I
confirm imported transactions against either the receipt or the
statement when I reconcile. I'm not sure if that's addressed in this
patch, but I'll find out soon enough.
Jack
On 2020.01.24 10:13, Thomas Baumgart wrote:
> Dawid,
>
> thanks for taking the time to improve KMyMoney by providing a
patch.
> The 'messy' code is probably due to historic growth ;)
>
> In fact, the OFX part is not my area of expertise because none
of the
> banks here uses/supports it. But the code looks like we could
use it
> to improve the functionality.
>
> @Jack: can you check if the patch works with your OFX downloads?
>
> Regards
>
> Thomas
>
> On Freitag, 24. Januar 2020 05:57:25 CET Dawid Wrobel wrote:
>
> > Hello,
> >
> >
> > The US Citibank's OFX account statement uses NAME and MEMO fields
> only.
> > Even worse, Payee is saved in MEMO. When importing using OFX
> plugin, one
> > has an option to choose preferred field for Payee. However,
> choosing MEMO
> > there solves the problem only partially, since the NAME field gets
> > discarded despite carrying useful transaction description
> information that
> > could be otherwise used as Memo.
> >
> >
> > I looked into the code and noticed that it is quite messy and
> indeed does
> > not consider non-empty NAME field at all, so I rewrote that
section
> of the
> > code.
> >
> >
> > I am building a "matrix" of preference for each of the Payee,
Memo
> and
> > Name, based on the user's choice for Payee. The importing
logic then
> > evaluates each transaction by:
> >
> > - finding the first valid (non-empty) OFX field from the
preferred
> list for
> > Payee;
> >
> > - looking up the corresponding OFX field for Memo preference. If
> the OFX
> > field is not valid, the corresponding field of the Name
preference
> list is
> > used instead.
> >
> >
> > This opportunistic approach guarantees that no information gets
> ignored
> > whenever it can be assigned to either Payee or Memo fields. It
also
> handles
> > the investment transactions more cleanly.
> >
> >
> > STEPS TO REPRODUCE
> >
> > 1. Prepare an OFX file with the following transaction:
> >
> >
> > <STMTTRN>
> >
> > <TRNTYPE>CREDIT
> >
> > <DTPOSTED>20200115120000
> >
> > <TRNAMT>1366.31
> >
> > <FITID>202001110001
> >
> > <NAME>ACH Electronic Credit
> >
> > <MEMO>EMPLOYER
> >
> > </STMTTRN>
> >
> >
> > 2. Attempt to import, set the preferred Payee field to MEMO
> >
> >
> > OBSERVED RESULT
> >
> > NAME gets ignored and the MEMO gets used twice instead
> >
> >
> > EXPECTED RESULT
> >
> > Since MEMO is already used for Payee and PAYEEID is not
provided by
> the
> > OFX, NAME field should be used to populate Memo information.
> >
> >
> > ADDITIONAL INFORMATION
> >
> > I haven't coded in C++/QT for over 6 years, so my apologies if
the
> code is
> > not up to modern standards.
> >
> >
> > Patch attached and also checked-in to my forked repo here:
> >
>
https://github.com/wrobelda/kmymoney/commit/8064ced6da1c4acea582618dbc7777b915124f16
> >
> >
> > If required, I can try and push it through Phabricator instead.
> >
>
> --
>
> Regards
>
> Thomas Baumgart
>
> https://www.signal.org/ Signal, the better WhatsApp
> -------------------------------------------------------------
> Embedded Linux: because you can't do control-alt-delete on a
> pacemaker.
> -------------------------------------------------------------
>
