Re: [R] Possible causes of unexpected behavior
Please confirm that when you do the manual load and check that f(v*) matches the result from qsub() it succeeds for cases #1,#2 but only fails for #3. On Fri, Mar 4, 2022 at 10:06 AM Arthur Fendrich wrote: > Dear all, > > I am currently having a weird problem with a large-scale optimization > routine. It would be nice to know if any of you have already gone through > something similar, and how you solved it. > > I apologize in advance for not providing an example, but I think the > non-reproducibility of the error is maybe a key point of this problem. > > Simplest possible description of the problem: I have two functions: g(X) > and f(v). > g(X) does: > i) inputs a large matrix X; > ii) derives four other matrices from X (I'll call them A, B, C and D) then > saves to disk for debugging purposes; > > Then, f(v) does: > iii) loads A, B, C, D from disk > iv) calculates the log-likelihood, which vary according to a vector of > parameters, v. > > My goal application is quite big (X is a 4x4 matrix), so I created > the following versions to test and run the codes/math/parallelization: > #1) A simulated example with X being 100x100 > #2) A degraded version of the goal application, with X being 4000x4000 > #3) The goal application, with X being 4x4 > > When I use qsub to submit the job, using the exact same code and processing > cluster, #1 and #2 run flawlessly, so no problem. These results tell me > that the codes/math/parallelization are fine. > > For application #3, it converges to a vector v*. However, when I manually > load A, B, C and D from disk and calculate f(v*), then the value I get is > completely different. > For example: > - qsub job says v* = c(0, 1, 2, 3) is a minimum with f(v*) = 1. > - when I manually load A, B, C, D from disk and calculate f(v*) on the > exact same machine with the same libraries and environment variables, I get > f(v*) = 1000. > > This is a very confusing behavior. In theory the size of X should not > affect my problem, but it seems that things get unstable as the dimension > grows. The main issue for debugging is that g(X) for simulation #3 takes > two hours to run, and I am completely lost on how I could find the causes > of the problem. Would you have any general advices? > > Thank you very much in advance for literally any suggestions you might > have! > > Best regards, > Arthur > > [[alternative HTML version deleted]] > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Looking for package for data generation for classification and regression
On Fri, Mar 4, 2022 at 8:07 AM Ranjan Maitra wrote: > > > I am in need of generating artificial data for machine learning > > classification and regression analysis. What I am looking for is > > something similar to Python sklearn.datasets.make_classification and > > sklearn.datasets.make_regression: > > > > https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html > > > > https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_regression.html > > > > I have searched CRAN for something similar, but found nothing. Could > > someone please help me with this? > > Not sure if this helps, but at least for classification and clustering, there > is the MixSim package on CRAN which provides classification datasets > according to an overall overlap measure. Thanks, Ranjan, that is also quite helpful, since clustering is also a topic of the course! Paul __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Problem installing Rcmdr on version 4.1.2...
On Fri, 4 Mar 2022 08:23:43 -0500 Brian Lunergan wrote: > Running R 4.1.2 on Linux Mint 19.3. > g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include > -I'/home/brian/R/x86_64-pc-linux-gnu-library/4.1/testthat/include' > -fpic -g -O2 -fdebug-prefix-map=/build/r-base-J7pprH/r-base-4.1.2=. > -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time > -D_FORTIFY_SOURCE=2 -g -c test-runner.cpp -o test-runner.o > g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions > -Wl,-z,relro -o nloptr.so init_nloptr.o nloptr.o test-C-API.o > test-runner.o -llapack -lblas -lgfortran -lm -lquadmath -Lnlopt/lib > -lnlopt -L/usr/lib/R/lib -lR > /usr/bin/ld: cannot find -lnlopt > collect2: error: ld returned 1 exit status Typically, when an R package wraps a third-party library, you need a development version of it installed in order to install that package from source. If you're running R from the Linux Mint repos, try to install r-cran-nloptr from Linux Mint repositories. If you don't, or have some trouble installing the package, install the libnlopt-dev Linux Mint package before trying to install the nloptr R package. -- Best regards, Ivan __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] copying huge strings via clipboard?
Hi,
This is on Windows10 via the R gui . I, admittedly inadvisably, tried
to create a new character object by first copying a 1-million character
string (including lead and trail "'" chars) to the clipboard and then,
in the console,
>> foo <-
and hitting "paste"
What I found is that, around 5000 characters, a newline ( "\n") char
showed up. Is this something that the Windows Clipboard does, or
something odd about pasting into a command in R?
Postscript: using
>> bar <- readChar('thefile.txt',1e6)
the import works perfectly.
--
Carl Witthoft
personal: c...@witthoft.com
The Witthoft Group, Consulting
https://witthoftgroup.weebly.com/
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] linear mixed model using lmer
Thanks Jeff for reminding me that the attachment is removed. I put it in my
google drive if anyone wants to test the data
(https://drive.google.com/file/d/1lgVZVLHeecp9a_sFxEPeg6353O-qXZhM/view?usp=sharing)
I'll try the mixed model mailing list as well.
John
On Friday, March 4, 2022, 04:56:20 PM PST, Jeff Newmiller
wrote:
a) There is a mailing list for that:
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
b) Read the Posting Guide, as most attachment types are removed to avoid
propagating worms/viruses. (None seen upon receipt of this email.)
On March 4, 2022 4:41:57 PM PST, array chip via R-help
wrote:
>Dear all, I have this simple dataset to measure the yeild of a crop collected
>in 2 batches (attached). when I ran a simple inear mixed model using lmer to
>estimate within-batch and between-batch variability, the between-batch
>variability is 0. The run showed that data is singular. Does anyone know why
>the data is singular and what's the reason for 0 variability? is it because
>the dataset only has 2 batches?
>> daty<-read.table("datx.txt",sep='\t',header=T,row.names=NULL)
>> library(lme4)> lmer(yield~1+(1|batch),daty)
>boundary (singular) fit: see ?isSingular
>Linear mixed model fit by REML ['lmerMod']
>Formula: yield ~ 1 + (1 | batch)
> Data: daty
>REML criterion at convergence: 115.6358
>Random effects:
> Groups Name Std.Dev.
> batch (Intercept) 0.000
> Residual 2.789
>Number of obs: 24, groups: batch, 2
>Fixed Effects:
>(Intercept)
> 5.788
>
>Thanks!
>John
--
Sent from my phone. Please excuse my brevity.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] linear mixed model using lmer
Do you really think a variance from a sample size of 2 makes any sense?
Bert Gunter
"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Fri, Mar 4, 2022 at 5:06 PM array chip via R-help
wrote:
>
> Thanks Jeff for reminding me that the attachment is removed. I put it in my
> google drive if anyone wants to test the data
> (https://drive.google.com/file/d/1lgVZVLHeecp9a_sFxEPeg6353O-qXZhM/view?usp=sharing)
> I'll try the mixed model mailing list as well.
> John
> On Friday, March 4, 2022, 04:56:20 PM PST, Jeff Newmiller
> wrote:
>
> a) There is a mailing list for that:
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
> b) Read the Posting Guide, as most attachment types are removed to avoid
> propagating worms/viruses. (None seen upon receipt of this email.)
>
> On March 4, 2022 4:41:57 PM PST, array chip via R-help
> wrote:
> >Dear all, I have this simple dataset to measure the yeild of a crop
> >collected in 2 batches (attached). when I ran a simple inear mixed model
> >using lmer to estimate within-batch and between-batch variability, the
> >between-batch variability is 0. The run showed that data is singular. Does
> >anyone know why the data is singular and what's the reason for 0
> >variability? is it because the dataset only has 2 batches?
> >> daty<-read.table("datx.txt",sep='\t',header=T,row.names=NULL)
> >> library(lme4)> lmer(yield~1+(1|batch),daty)
> >boundary (singular) fit: see ?isSingular
> >Linear mixed model fit by REML ['lmerMod']
> >Formula: yield ~ 1 + (1 | batch)
> > Data: daty
> >REML criterion at convergence: 115.6358
> >Random effects:
> > Groups NameStd.Dev.
> > batch(Intercept) 0.000
> > Residual 2.789
> >Number of obs: 24, groups: batch, 2
> >Fixed Effects:
> >(Intercept)
> > 5.788
> >
> >Thanks!
> >John
> --
> Sent from my phone. Please excuse my brevity.
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
