Re: [R] stfrtime function not returning proper results through sqldf package in R

[Jeff Newmiller]

Yes, some of my info was outdated and misleading. :)

However, after I RTFM [1] it becomes clear that the strftime function 
being invoked in the SQL statement has completely different parameters 
(both ordering and interpretation) and behavior than the strftime function 
in R, and that is the primary issue affecting the original question.

library(sqldf)
Loading required package: gsubfn
Loading required package: proto
Loading required package: RSQLite
Loading required package: DBI
Sys.setenv( TZ="GMT" )
log <- data.frame( DateC = c( "2013/05/25", "2013/05/28", 
"2013/05/31","2013/06/01"
+                             , "2013/06/02", "2013/06/05", "2013/06/07" )
+                  , Quantity = c( 9, 1, 15, 4, 5, 17, 18 )
+                  , stringsAsFactors = FALSE
+                  )
log$Date <- as.Date( log$DateC, format="%Y/%m/%d" )
log$DateS <- as.character( log$Date, format="%Y-%m-%d" )
str( log )
'data.frame':   7 obs. of  4 variables:
 $ DateC   : chr  "2013/05/25" "2013/05/28" "2013/05/31" "2013/06/01" ...
 $ Quantity: num  9 1 15 4 5 17 18
 $ Date    : Date, format: "2013-05-25" "2013-05-28" ...
 $ DateS   : chr  "2013-05-25" "2013-05-28" "2013-05-31" "2013-06-01" ...
# SQLite treatment of standard date string format converts to 
# astronomical Julian Day
sqldf( "select strftime( '%J', '1970-01-01' )" )
Loading required package: tcltk
  strftime( '%J', '1970-01-01' )
1                      2440587.5
# SQLite strftime function can understand strings or julian day
# SQLite does not know 'Date' column is being stored by sqldf as days 
# since 1970-01-01
sqldf( "select Date, Date AS DateN, strftime( '%Y-%m-%d', Date+2440587.5 ) AS DateJC, Date+2440587.5 AS DateJN from 
log", method=c( "Date", "numeric", "character", "numeric" ) )
        Date DateN     DateJC  DateJN
1 2013-05-25 15850 2013-05-25 2456438
2 2013-05-28 15853 2013-05-28 2456441
3 2013-05-31 15856 2013-05-31 2456444
4 2013-06-01 15857 2013-06-01 2456445
5 2013-06-02 15858 2013-06-02 2456446
6 2013-06-05 15861 2013-06-05 2456449
7 2013-06-07 15863 2013-06-07 2456451
# So, to group by month using month computed in SQLite, have to convert 
# to JD
sqldf( "select strftime( '%m', Date+2440587.5 ) AS Month, sum(Quantity) AS 
SumOfQuantity from log group by Month" )
  Month SumOfQuantity
1    05            25
2    06            44

# alternatively, figure out the month in R first
log$Month <- strftime( log$Date, format="%m" )
sqldf( "select Month, sum(Quantity) AS SumOfQuantity from log group by Month" )
  Month SumOfQuantity
1    05            25
2    06            44

---
[1] https://www.sqlite.org/lang_datefunc.html

On Fri, 16 Sep 2016, Gabor Grothendieck wrote:

To be precise it's SQLite that does not have date and time data types.
If you use an sqldf backend such as H2 that does have such types then
sqldf will pass them as such.  In the case of R's "Date" class such
objects are passed to SQLite as numbers since that is what SQLite can
understand but they are passed as dates to H2 (and other backends if
they have a date type).

Also note that after sqldf passes a date to SQLite as a number then
when SQLite passes it back to sqldf then sqldf  knows that it was
originally a date (due to the column name being the same) and coerces
it to Date class again.  In the example below sqldf passed the number
of days to 2000-01-01 since the UNIX epoch to SQLite.  SQLite then
processed it and passed it back as a number again. Then sqldf realized
that it was originally a Date because the column name is still d and
the original column d was of "Date" class and so coerces the number
from SQLite to Date.  There are a limited number of circumstances
where this heuristic works but they are sufficient that it's often
transparent even though SQLite has no date and time types.

library(sqldf)
DF <- data.frame(d = as.Date("2000-01-01"))
sqldf("select d+1 as d from DF") # return next day
          d
1 2000-01-02

At the same time iif you really need to do serious date processing on
the SQL side it's much easier with an sqldf backend such as H2 that
actually supports date and time types and no heuristic is needed by
sqldf and no user workarounds on the R side are needed.


On Fri, Sep 16, 2016 at 10:15 AM, Jeff Newmiller
<jdnew...@dcn.davis.ca.us> wrote:
SQLite only understands certain fundamental data types, and neither Date nor 
POSIXct types are among them. They get stored as their internal numeric 
representations.

The internal numeric representations of Date and POSIXct are incompatible. You 
are sending Dates to SQLite and trying to then interpret it as POSIXct by 
handing that numeric to strftime.

Note that within R the Date and POSIXct types are made sort-of compatible by 
internal checking of class attributes that are not stored in SQLite. They are 
still only sort-of compatible because Date has no concept of time zone and 
always assumes GMT rather than local time when being converted.

I recommend retrieving the stored Date value as a Date value into R so that 
strftime can recognize how to interpret it. If you need to handle time as well 
as date you may find that converting to character first before converting to 
POSIXct with an appropriate time zone behaves with least surprises.
--
Sent from my phone. Please excuse my brevity.

On September 16, 2016 6:23:48 AM PDT, PIKAL Petr <petr.pi...@precheza.cz> wrote:
Hi Peter

The devil is in detail

Data from OP had different format and was transferred to Date object by
as.Date, which results in incorrect values (and NA if not transferred)
df <- data.frame(Date =
c("2013/05/25","2013/05/28","2013/05/31","2013/06/01","2013/06/02",
"2013/06/05","2013/06/07"), Quantity = c(9,1,15,4,5,17,18))
df$Date<-as.Date(df$Date)
cbind(df, sqldf("select strftime( '%m', Date) from df"))

Data formatted according to your example transferred to Date object by
as data, again incorrect result
df2 <- data.frame(Date =
c("2013-05-25","2013-05-28","2013-05-31","2013-06-01","2013-06-02",
"2013-06-05","2013-06-07"), Quantity = c(9,1,15,4,5,17,18))
df2$Date<-as.Date(df2$Date)
cbind(df2, sqldf("select strftime( '%m', Date) from df2"))

Data formatted according to your example but **not** changed to Dates,
correct result
df3 <- data.frame(Date =
c("2013-05-25","2013-05-28","2013-05-31","2013-06-01","2013-06-02",
"2013-06-05","2013-06-07"), Quantity = c(9,1,15,4,5,17,18))
cbind(df3, sqldf("select strftime( '%m', Date) from df3"))

so sqldf is a bit peculiar about required input values and does not
know how to handle Date objects.

Cheers
Petr


-----Original Message-----
From: peter dalgaard [mailto:pda...@gmail.com]
Sent: Friday, September 16, 2016 2:45 PM
To: PIKAL Petr <petr.pi...@precheza.cz>
Cc: Manohar Reddy <manu.redd...@gmail.com>; R-help <r-help@r-
project.org>
Subject: Re: [R] stfrtime function not returning proper results
through sqldf
package in R

Presumably, sqldf does not know about Date object so passes an
integer that
gets interpreted as who knows what...

This seems to work:

df <- data.frame(date=as.character(Sys.Date()+seq(0,180,,10)))
cbind(df, sqldf("select strftime( '%m', date) from df"))
         date strftime( '%m', date)
1  2016-09-16                    09
2  2016-10-06                    10
3  2016-10-26                    10
4  2016-11-15                    11
5  2016-12-05                    12
6  2016-12-25                    12
7  2017-01-14                    01
8  2017-02-03                    02
9  2017-02-23                    02
10 2017-03-15                    03

-pd



On 16 Sep 2016, at 13:48 , PIKAL Petr <petr.pi...@precheza.cz> wrote:

Hi

Most probably there is some syntactic suger how to correctly
formulate sql
query.

sqldf('select Date, strftime("Date", "%m") from log')
       Date strftime("Date", "%m")
1 2013-05-25                   <NA>
2 2013-05-28                   <NA>
3 2013-05-31                   <NA>
4 2013-06-01                   <NA>
5 2013-06-02                   <NA>
6 2013-06-05                   <NA>
7 2013-06-07                   <NA>
sqldf('select Date, format("Date", "%m") from log')
Error in sqliteSendQuery(con, statement, bind.data) :
 error in statement: no such function: format


format(sqldf("select Date from log"), "%m")

This one however works.

Cheers
Petr

From: Manohar Reddy [mailto:manu.redd...@gmail.com]
Sent: Friday, September 16, 2016 11:55 AM
To: PIKAL Petr <petr.pi...@precheza.cz>
Cc: R-help <r-help@r-project.org>
Subject: Re: [R] stfrtime function not returning proper results
through sqldf package in R


Hi Petr,

Thanks, For me also it?s working fine when I directly used that
function  but
when I call strftime function through sqldf package it?s returning NA
values
(PFA) ,but my requirement is I need to do that only sqldf as I?m
writing some
?T sql ? queries against on the dataset.

Manu.

On Fri, Sep 16, 2016 at 12:56 PM, PIKAL Petr
<petr.pi...@precheza.cz<mailto:petr.pi...@precheza.cz>> wrote:
Hi

Hm
format(log$Date, "%m")
[1] "05" "05" "05" "06" "06" "06" "06"
strftime(log$Date, "%m")
[1] "05" "05" "05" "06" "06" "06" "06"


works for me.

Just a blind guess, is an object date somwhere in your environment?

Cheers
Petr


-----Original Message-----
From: R-help

[mailto:r-help-boun...@r-project.org<mailto:r-help-bounces@r-project.
org>] On Behalf Of Manohar Reddy
Sent: Friday, September 16, 2016 6:36 AM
To: R-help <r-help@r-project.org<mailto:r-help@r-project.org>>
Subject: [R] stfrtime function not returning proper results
through
sqldf package in R

Hi ,



 I have data something looks like below (or PFA), but when I?m
extracting month  using *strftime*  function through *sqldf*
library
,it?s returning below results but it?s not returning exact results
,it supposed to return 05,05,05,06,06,06.Can anyone please guide
me
how to do that with
*strftime* function.



Thanks in advance.



Quiries :


library(scales)

# load data:
log <- data.frame(Date =

c("2013/05/25","2013/05/28","2013/05/31","2013/06/01","2013/06/02","2
013
/06/05","2013/06/07"),
 Quantity = c(9,1,15,4,5,17,18))


# convert date variable from factor to date format:
log$Date <- as.Date(log$Date,
 "%Y/%m/%d") # tabulate all the options here
str(log)







Manu.

________________________________
Tento e-mail a jak?koliv k n?mu p?ipojen? dokumenty jsou d?v?rn? a
jsou
ur?eny pouze jeho adres?t?m.
Jestli?e jste obdr?el(a) tento e-mail omylem, informujte laskav?
neprodlen? jeho odes?latele. Obsah tohoto emailu i s p??lohami a jeho
kopie
vyma?te ze sv?ho syst?mu.
Nejste-li zam??len?m adres?tem tohoto emailu, nejste opr?vn?ni
tento
email jakkoliv u??vat, roz?i?ovat, kop?rovat ?i zve?ej?ovat.
Odes?latel e-mailu neodpov?d? za eventu?ln? ?kodu zp?sobenou
modifikacemi ?i zpo?d?n?m p?enosu e-mailu.

V p??pad?, ?e je tento e-mail sou??st? obchodn?ho jedn?n?:
- vyhrazuje si odes?latel pr?vo ukon?it kdykoliv jedn?n? o uzav?en?
smlouvy,
a to z jak?hokoliv d?vodu i bez uveden? d?vodu.
- a obsahuje-li nab?dku, je adres?t opr?vn?n nab?dku bezodkladn?
p?ijmout; Odes?latel tohoto e-mailu (nab?dky) vylu?uje p?ijet?
nab?dky ze
strany p??jemce s dodatkem ?i odchylkou.
- trv? odes?latel na tom, ?e p??slu?n? smlouva je uzav?ena teprve
v?slovn?m
dosa?en?m shody na v?ech jej?ch n?le?itostech.
- odes?latel tohoto emailu informuje, ?e nen? opr?vn?n uzav?rat za
spole?nost ??dn? smlouvy s v?jimkou p??pad?, kdy k tomu byl p?semn?
zmocn?n nebo p?semn? pov??en a takov? pov??en? nebo pln? moc byly
adres?tovi tohoto emailu p??padn? osob?, kterou adres?t zastupuje,
p?edlo?eny nebo jejich existence je adres?tovi ?i osob? j?m
zastoupen?
zn?m?.

This e-mail and any documents attached to it may be confidential
and are
intended only for its intended recipients.
If you received this e-mail by mistake, please immediately inform
its
sender. Delete the contents of this e-mail with all attachments and
its copies
from your system.
If you are not the intended recipient of this e-mail, you are not
authorized
to use, disseminate, copy or disclose this e-mail in any manner.
The sender of this e-mail shall not be liable for any possible
damage caused
by modifications of the e-mail or by delay with transfer of the
email.

In case that this e-mail forms part of business dealings:
- the sender reserves the right to end negotiations about entering
into a
contract in any time, for any reason, and without stating any
reasoning.
- if the e-mail contains an offer, the recipient is entitled to
immediately
accept such offer; The sender of this e-mail (offer) excludes any
acceptance
of the offer on the part of the recipient containing any amendment or
variation.
- the sender insists on that the respective contract is concluded
only upon
an express mutual agreement on all its aspects.
- the sender of this e-mail informs that he/she is not authorized
to enter
into any contracts on behalf of the company except for cases in which
he/she
is expressly authorized to do so in writing, and such authorization
or power of
attorney is submitted to the recipient or the person represented by
the
recipient, or the existence of such authorization is known to the
recipient of
the person represented by the recipient.



--


Thanks,
Manohar Reddy P
+91-9705302062.

________________________________
Tento e-mail a jak?koliv k n?mu p?ipojen? dokumenty jsou d?v?rn? a
jsou
ur?eny pouze jeho adres?t?m.
Jestli?e jste obdr?el(a) tento e-mail omylem, informujte laskav?
neprodlen? jeho odes?latele. Obsah tohoto emailu i s p??lohami a jeho
kopie
vyma?te ze sv?ho syst?mu.
Nejste-li zam??len?m adres?tem tohoto emailu, nejste opr?vn?ni
tento
email jakkoliv u??vat, roz?i?ovat, kop?rovat ?i zve?ej?ovat.
Odes?latel e-mailu neodpov?d? za eventu?ln? ?kodu zp?sobenou
modifikacemi ?i zpo?d?n?m p?enosu e-mailu.

V p??pad?, ?e je tento e-mail sou??st? obchodn?ho jedn?n?:
- vyhrazuje si odes?latel pr?vo ukon?it kdykoliv jedn?n? o uzav?en?
smlouvy,
a to z jak?hokoliv d?vodu i bez uveden? d?vodu.
- a obsahuje-li nab?dku, je adres?t opr?vn?n nab?dku bezodkladn?
p?ijmout; Odes?latel tohoto e-mailu (nab?dky) vylu?uje p?ijet?
nab?dky ze
strany p??jemce s dodatkem ?i odchylkou.
- trv? odes?latel na tom, ?e p??slu?n? smlouva je uzav?ena teprve
v?slovn?m
dosa?en?m shody na v?ech jej?ch n?le?itostech.
- odes?latel tohoto emailu informuje, ?e nen? opr?vn?n uzav?rat za
spole?nost ??dn? smlouvy s v?jimkou p??pad?, kdy k tomu byl p?semn?
zmocn?n nebo p?semn? pov??en a takov? pov??en? nebo pln? moc byly
adres?tovi tohoto emailu p??padn? osob?, kterou adres?t zastupuje,
p?edlo?eny nebo jejich existence je adres?tovi ?i osob? j?m
zastoupen?
zn?m?.

This e-mail and any documents attached to it may be confidential
and are
intended only for its intended recipients.
If you received this e-mail by mistake, please immediately inform
its
sender. Delete the contents of this e-mail with all attachments and
its copies
from your system.
If you are not the intended recipient of this e-mail, you are not
authorized
to use, disseminate, copy or disclose this e-mail in any manner.
The sender of this e-mail shall not be liable for any possible
damage caused
by modifications of the e-mail or by delay with transfer of the
email.

In case that this e-mail forms part of business dealings:
- the sender reserves the right to end negotiations about entering
into a
contract in any time, for any reason, and without stating any
reasoning.
- if the e-mail contains an offer, the recipient is entitled to
immediately
accept such offer; The sender of this e-mail (offer) excludes any
acceptance
of the offer on the part of the recipient containing any amendment or
variation.
- the sender insists on that the respective contract is concluded
only upon
an express mutual agreement on all its aspects.
- the sender of this e-mail informs that he/she is not authorized
to enter
into any contracts on behalf of the company except for cases in which
he/she
is expressly authorized to do so in writing, and such authorization
or power of
attorney is submitted to the recipient or the person represented by
the
recipient, or the existence of such authorization is known to the
recipient of
the person represented by the recipient.

    [[alternative HTML version deleted]]

______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

--
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School Solbjerg Plads 3,
2000
Frederiksberg, Denmark
Phone: (+45)38153501
Office: A 4.23
Email: pd....@cbs.dk  Priv: pda...@gmail.com


________________________________
Tento e-mail a jak?koliv k n?mu p?ipojen? dokumenty jsou d?v?rn? a jsou
ur?eny pouze jeho adres?t?m.
Jestli?e jste obdr?el(a) tento e-mail omylem, informujte laskav?
neprodlen? jeho odes?latele. Obsah tohoto emailu i s p??lohami a jeho
kopie vyma?te ze sv?ho syst?mu.
Nejste-li zam??len?m adres?tem tohoto emailu, nejste opr?vn?ni tento
email jakkoliv u??vat, roz?i?ovat, kop?rovat ?i zve?ej?ovat.
Odes?latel e-mailu neodpov?d? za eventu?ln? ?kodu zp?sobenou
modifikacemi ?i zpo?d?n?m p?enosu e-mailu.

V p??pad?, ?e je tento e-mail sou??st? obchodn?ho jedn?n?:
- vyhrazuje si odes?latel pr?vo ukon?it kdykoliv jedn?n? o uzav?en?
smlouvy, a to z jak?hokoliv d?vodu i bez uveden? d?vodu.
- a obsahuje-li nab?dku, je adres?t opr?vn?n nab?dku bezodkladn?
p?ijmout; Odes?latel tohoto e-mailu (nab?dky) vylu?uje p?ijet? nab?dky
ze strany p??jemce s dodatkem ?i odchylkou.
- trv? odes?latel na tom, ?e p??slu?n? smlouva je uzav?ena teprve
v?slovn?m dosa?en?m shody na v?ech jej?ch n?le?itostech.
- odes?latel tohoto emailu informuje, ?e nen? opr?vn?n uzav?rat za
spole?nost ??dn? smlouvy s v?jimkou p??pad?, kdy k tomu byl p?semn?
zmocn?n nebo p?semn? pov??en a takov? pov??en? nebo pln? moc byly
adres?tovi tohoto emailu p??padn? osob?, kterou adres?t zastupuje,
p?edlo?eny nebo jejich existence je adres?tovi ?i osob? j?m zastoupen?
zn?m?.

This e-mail and any documents attached to it may be confidential and
are intended only for its intended recipients.
If you received this e-mail by mistake, please immediately inform its
sender. Delete the contents of this e-mail with all attachments and its
copies from your system.
If you are not the intended recipient of this e-mail, you are not
authorized to use, disseminate, copy or disclose this e-mail in any
manner.
The sender of this e-mail shall not be liable for any possible damage
caused by modifications of the e-mail or by delay with transfer of the
email.

In case that this e-mail forms part of business dealings:
- the sender reserves the right to end negotiations about entering into
a contract in any time, for any reason, and without stating any
reasoning.
- if the e-mail contains an offer, the recipient is entitled to
immediately accept such offer; The sender of this e-mail (offer)
excludes any acceptance of the offer on the part of the recipient
containing any amendment or variation.
- the sender insists on that the respective contract is concluded only
upon an express mutual agreement on all its aspects.
- the sender of this e-mail informs that he/she is not authorized to
enter into any contracts on behalf of the company except for cases in
which he/she is expressly authorized to do so in writing, and such
authorization or power of attorney is submitted to the recipient or the
person represented by the recipient, or the existence of such
authorization is known to the recipient of the person represented by
the recipient.
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.



--
Statistics & Software Consulting
GKX Group, GKX Associates Inc.
tel: 1-877-GKX-GROUP
email: ggrothendieck at gmail.com


---------------------------------------------------------------------------
Jeff Newmiller                        The     .....       .....  Go Live...
DCN:<jdnew...@dcn.davis.ca.us>        Basics: ##.#.       ##.#.  Live Go...
                                      Live:   OO#.. Dead: OO#..  Playing
Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
/Software/Embedded Controllers)               .OO#.       .OO#.  rocks...1k

______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.