Renamed repository: https://github.com/AleXoundOS/haskell-stack-mirror-script
On Saturday, March 4, 2017 at 11:01:24 PM UTC+3, Alexander T wrote: > > Hi again! > > I tried your commit that skips downloading if file is present and it works. > However, imho this is still a suboptimal solution. > > This is what I came up with. A script that downloads every url from stack > setup > yaml and all packages from hackage index. Here is the repository: > https://github.com/AleXoundOS/haskell-dev-mirror-script . In other words, > you > get everything you need to develop software in Haskell except remote docs. > > After everything is downloaded it uses 20GiB of space and takes about 15 > minutes to verify files integrity. Not that much. Tbh, I was afraid it > will take > more space and time. > > Downloaded files are placed into proper subdirectories in "mirror" > directory > that is ready to be served by http server. Clients need to get generated > config.yaml with all the paths adjusted to the structure of "mirror" > directory. > And finally, to bypass the well-known stack bug ignoring "setup-info" > field, > clients need the generated stack setup yaml for "stack setup" operations. > > On Sunday, February 19, 2017 at 6:27:56 PM UTC+3, Neil Mitchell wrote: >> >> > The name of github repository "offline-stack" is a bit misleading. And >> > repository descriptions "Install Stack without internet access", "demo >> of >> > using Haskell Stack without internet access" confuse further. >> >> Sorry, I'll tweak the text. It's meant to be a demo of the steps you >> need to install stack without internet access - not a ready rolled >> solution. In particular, it isn't the latter because: >> >> 1) If your machine has full internet access then this solution isn't >> required. You have to have some way to initially get the files onto >> the machine, and that will vary by location, so there's no chance of >> doing something standard. >> >> 2) Most places have existing file serving servers, and in my case I >> just reused one of them. No point firing up a new Haskell server just >> to serve up a few files, when putting it on an Apache/IIS server is >> probably easier. >> >> > I thought that the http-redirect server is able to run offline, i.e. >> without >> > internet access at all once all the required files were downloaded by >> > passing requests through proxy. >> >> You could easily do that - in fact I've just pushed a 1 line change to >> do that - >> https://github.com/ndmitchell/offline-stack/commit/86c6ff60dc7df9e20363c1e4f4cbcd79a6543d2f >> >> (entirely untested). >> >> > Thank you for all your work in this direction. >> > So, do you think it is possible to turn this idea into an offline >> server? Do >> > you see any pitfalls building up such system for production? Like, does >> the >> > theory of idea need major adjustments in your opinion? >> >> In my case, step 1 doesn't work, and step 2 is more hassle than >> reusing existing solutions. However, if you can get around step 1 >> (which it seems like you can), and step 2 is no big deal, then it >> should work. >> >> Thanks, Neil >> >> > >> > On Sunday, February 19, 2017 at 5:46:49 PM UTC+3, Neil Mitchell wrote: >> >> >> >> The fact that you can't do "stack new" is known, and raised at >> >> https://github.com/commercialhaskell/stack/issues/2804. >> >> >> >> Note that the http-redirect tool I wrote _always_ goes to the >> >> internet, it never serves up mirrored content. If you want to run on a >> >> server which doesn't have internet access you'll need to modify that >> >> code, or spin up another type of server to serve up the content. The >> >> purpose of http-redirect is to robustly test my configuration steps - >> >> I didn't write it with the idea of production use in mind. >> >> >> >> Thanks, Neil >> >> >> >> >> >> On Sun, Feb 19, 2017 at 11:57 AM, Alexander T <[email protected]> >> wrote: >> >> > Ah, also forgot to clarify that the nightly-2017-02-02.yaml file was >> >> > already >> >> > previously retrieved by the http-redirect server and actually exists >> in >> >> > it's >> >> > mirror directory (with filename renamed according to server's >> >> > conventions). >> >> > >> >> > Server side >> >> > $ stat >> >> > >> >> > >> mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml >> >> >> > File: >> >> > >> >> > >> mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml >> >> >> > Size: 5377585 Blocks: 10504 IO Block: 4096 regular >> file >> >> > Device: 2dh/45d Inode: 5819 Links: 1 >> >> > Access: (0644/-rw-r--r--) Uid: ( 1004/ alex) Gid: ( 100/ >> users) >> >> > Access: 2017-02-17 05:02:18.946927356 +0300 >> >> > Modify: 2017-02-17 05:02:18.936927354 +0300 >> >> > Change: 2017-02-19 03:30:53.923441888 +0300 >> >> > Birth: - >> >> > >> >> > On Sunday, February 19, 2017 at 4:29:49 AM UTC+3, Alexander T wrote: >> >> >> >> >> >> Yes I did. >> >> >> >> >> >> Server side >> >> >> $ env | grep STACK_ROOT >> >> >> STACK_ROOT=/data/srv/stack_root >> >> >> $ pwd >> >> >> /data/srv >> >> >> $ ls >> >> >> http-redirect mirror stack_root >> >> >> $ stack path | grep config >> >> >> config-location: /data/srv/stack_root/global-project/stack.yaml >> >> >> $ cat /data/srv/stack_root/config.yaml >> >> >> urls: >> >> >> latest-snapshot: >> >> >> >> http://localhost:3000/https/www.stackage.org/download/snapshots.json >> >> >> lts-build-plans: >> >> >> >> >> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/ >> >> >> >> nightly-build-plans: >> >> >> >> >> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/ >> >> >> >> >> >> >> package-indices: >> >> >> - name: Hackage >> >> >> download-prefix: >> >> >> >> >> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/ >> >> >> http: >> >> >> >> >> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz >> >> >> >> >> >> >> >> >> >> Client side >> >> >> $ env | grep STACK_ROOT >> >> >> $ cat ~/.stack/config.yaml >> >> >> urls: >> >> >> latest-snapshot: >> >> >> >> http://localhost:3000/https/www.stackage.org/download/snapshots.json >> >> >> lts-build-plans: >> >> >> >> >> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/ >> >> >> >> nightly-build-plans: >> >> >> >> >> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/ >> >> >> >> >> >> >> package-indices: >> >> >> - name: Hackage >> >> >> download-prefix: >> >> >> >> >> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/ >> >> >> http: >> >> >> >> >> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz >> >> >> >> >> >> >> When I try to run "stack path" on client side without internet >> >> >> connection >> >> >> on both sides >> >> >> $ stack path >> >> >> Downloading nightly-2017-02-02 build plan ...RedownloadFailed >> Request { >> >> >> host = "localhost" >> >> >> port = 3000 >> >> >> secure = False >> >> >> requestHeaders = [] >> >> >> path = >> >> >> >> >> >> "/https/ >> raw.githubusercontent.com/fpco/stackage-nightly/master//nightly-2017-02-02.yaml" >> >> >> >> >> queryString = "" >> >> >> method = "GET" >> >> >> proxy = Nothing >> >> >> rawBody = False >> >> >> redirectCount = 10 >> >> >> responseTimeout = ResponseTimeoutDefault >> >> >> requestVersion = HTTP/1.1 >> >> >> } >> >> >> "/home/alex/.stack/build-plan/nightly-2017-02-02.yaml" (Response >> >> >> {responseStatus = Status {statusCode = 500, statusMessage = >> "Internal >> >> >> Server >> >> >> Error"}, responseVersion = HTTP/1.0, responseHeaders = >> [("Date","Sun, >> >> >> 19 Feb >> >> >> 2017 01:23:48 >> >> >> GMT"),("Server","Warp/3.2.11"),("Content-Type","text/plain; >> >> >> charset=utf-8")], responseBody = (), responseCookieJar = CJ {expose >> = >> >> >> []}, >> >> >> responseClose' = ResponseClose}) >> >> >> >> >> >> At the same time http-redirect server throws: >> >> >> HttpExceptionRequest Request { >> >> >> host = "raw.githubusercontent.com" >> >> >> port = 443 >> >> >> secure = True >> >> >> requestHeaders = [] >> >> >> path = >> >> >> "/fpco/stackage-nightly/master//nightly-2017-02-02.yaml" >> >> >> queryString = "" >> >> >> method = "GET" >> >> >> proxy = Nothing >> >> >> rawBody = False >> >> >> redirectCount = 10 >> >> >> responseTimeout = ResponseTimeoutDefault >> >> >> requestVersion = HTTP/1.1 >> >> >> } >> >> >> (ConnectionFailure getAddrInfo: does not exist (Name or service >> not >> >> >> known)) >> >> >> >> >> >> On Friday, February 17, 2017 at 9:48:46 AM UTC+3, Neil Mitchell >> wrote: >> >> >>> >> >> >>> > Thank you. I tried this, http-redirect server builds fine and >> runs. >> >> >>> > This is >> >> >>> > what I figured out. >> >> >>> >> >> >>> > Once I did "stack setup", "stack build" through the proxy (with >> >> >>> > corresponding config.yaml), I can reproduce it on the same >> system >> >> >>> > without >> >> >>> > internet connection. >> >> >>> > But if I take a fresh system with empty global stack root (e.g. >> >> >>> > ~/.stack on >> >> >>> > GNU/Linux) except solely config.yaml, it immediately fails. >> >> >>> > http-redirect >> >> >>> > throws this: >> >> >>> >> >> >>> Did you copy the config.yaml from the offline-stack repo into >> >> >>> STACK_ROOT? Is your STACK_ROOT actually where you think it is? >> (use >> >> >>> "stack path" to see). >> >> >>> >> >> >>> > "stack new" fails in both cases with: >> >> >>> > Here I can see that this request is not processed by the >> >> >>> > http-redirect, >> >> >>> > it >> >> >>> > keeps silent when stack throws this exception. >> >> >>> > >> >> >>> > Maybe I missed something? >> >> >>> >> >> >>> stack new isn't supported - I imagine you have to have another >> >> >>> redirect in config.yaml. Pull requests welcome! I've also raised a >> >> >>> ticket at https://github.com/ndmitchell/offline-stack/issues/1. >> >> >>> >> >> >>> Thanks, Neil >> >> >>> >> >> >>> > >> >> >>> > On Wednesday, February 15, 2017 at 11:04:36 PM UTC+3, Neil >> Mitchell >> >> >>> > wrote: >> >> >>> >> >> >> >>> >> > Please, can you describe all operations in detail. >> >> >>> >> >> >> >>> >> Describing is fragile. I've created a github project that >> describes >> >> >>> >> it. See the README at >> >> >>> >> https://github.com/ndmitchell/offline-stack#readme, plus the >> code >> >> >>> >> which executes it at >> >> >>> >> >> >> >>> >> >> https://github.com/ndmitchell/offline-stack/blob/master/appveyor.yml. >> >> >>> >> I test those steps on Appveyor, so it's guaranteed to work/ >> >> >>> >> >> >> >>> >> > The whole idea of using haskell with stack/cabal offline is >> >> >>> >> > crucial >> >> >>> >> > for >> >> >>> >> > work >> >> >>> >> > at many companies. >> >> >>> >> > First of all it's not clear how to obtain all the data needed >> for >> >> >>> >> > stack >> >> >>> >> > and >> >> >>> >> > cabal to serve locally. >> >> >>> >> >> >> >>> >> Agreed! Hopefully everything is now 100% clear. >> >> >>> >> >> >> >>> >> > >> >> >>> >> > On Friday, February 10, 2017 at 10:39:51 AM UTC+3, Neil >> Mitchell >> >> >>> >> > wrote: >> >> >>> >> >> >> >> >>> >> >> Hi David, >> >> >>> >> >> >> >> >>> >> >> Its not that bad, now the steps are known, so go ahead and >> email >> >> >>> >> >> me >> >> >>> >> >> if >> >> >>> >> >> you >> >> >>> >> >> get stuck. Steps are: >> >> >>> >> >> >> >> >>> >> >> * create stack root >> >> >>> >> >> >> >> >>> >> >> * copy config.yaml to the root >> >> >>> >> >> >> >> >>> >> >> * run stack setup with the setup yaml flag >> >> >>> >> >> >> >> >>> >> >> Figuring our these steps took weeks :) >> >> >>> >> >> >> >> >>> >> >> Thanks, Neil >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> On Thu, 9 Feb 2017 at 11:47 pm, David Sicilia >> >> >>> >> >> <[email protected]> >> >> >>> >> >> wrote: >> >> >>> >> >>> >> >> >>> >> >>> Hi Neil, >> >> >>> >> >>> >> >> >>> >> >>> Thank you for raising the ticket, I just gave it a +1, if >> that >> >> >>> >> >>> helps >> >> >>> >> >>> :-) >> >> >>> >> >>> >> >> >>> >> >>> So would you recommend that I go ahead and try it right >> now, >> >> >>> >> >>> or is it complicated enough that I should just wait for >> that >> >> >>> >> >>> change >> >> >>> >> >>> to be implemented? I guess what I mean is, after you >> realized >> >> >>> >> >>> which config files to change, was it straightforward from >> >> >>> >> >>> there? >> >> >>> >> >>> >> >> >>> >> >>> David >> >> >>> >> >>> >> >> >>> >> >>> On Saturday, January 21, 2017 at 11:27:22 AM UTC-5, David >> >> >>> >> >>> Sicilia >> >> >>> >> >>> wrote: >> >> >>> >> >>>> >> >> >>> >> >>>> Hi there, >> >> >>> >> >>>> >> >> >>> >> >>>> >> >> >>> >> >>>> I am behind a firewall with strict rules about downloading >> >> >>> >> >>>> software >> >> >>> >> >>>> from >> >> >>> >> >>>> the internet, >> >> >>> >> >>>> >> >> >>> >> >>>> so I would like to know if it is possible to download an >> >> >>> >> >>>> entire >> >> >>> >> >>>> stackage >> >> >>> >> >>>> LTS snapshot >> >> >>> >> >>>> >> >> >>> >> >>>> in one go and then serve it on a local server, to which we >> >> >>> >> >>>> could >> >> >>> >> >>>> then >> >> >>> >> >>>> point the stack tool. >> >> >>> >> >>>> >> >> >>> >> >>>> >> >> >>> >> >>>> By "snapshot" I would be referring to the source code for >> all >> >> >>> >> >>>> packages >> >> >>> >> >>>> because >> >> >>> >> >>>> >> >> >>> >> >>>> we'd need to draw from that local server to build on both >> >> >>> >> >>>> Linux >> >> >>> >> >>>> and >> >> >>> >> >>>> Windows. >> >> >>> >> >>>> >> >> >>> >> >>>> >> >> >>> >> >>>> For example, perhaps we'd be able to go to a server within >> the >> >> >>> >> >>>> firewall >> >> >>> >> >>>> and run >> >> >>> >> >>>> >> >> >>> >> >>>> "stack new", then edit the yaml file to point it to a >> local >> >> >>> >> >>>> stackage >> >> >>> >> >>>> server with given >> >> >>> >> >>>> >> >> >>> >> >>>> resolver number, then it would just behave normally from >> then >> >> >>> >> >>>> on, >> >> >>> >> >>>> except >> >> >>> >> >>>> always >> >> >>> >> >>>> >> >> >>> >> >>>> downloading packages from the local server. >> >> >>> >> >>>> >> >> >>> >> >>>> >> >> >>> >> >>>> What about compilers? I know that stack also downloads the >> >> >>> >> >>>> compilers, >> >> >>> >> >>>> so >> >> >>> >> >>>> I guess >> >> >>> >> >>>> >> >> >>> >> >>>> those would have to be included in the downloaded >> >> >>> >> >>>> snapshot-package >> >> >>> >> >>>> for >> >> >>> >> >>>> various >> >> >>> >> >>>> >> >> >>> >> >>>> platforms? >> >> >>> >> >>>> >> >> >>> >> >>>> >> >> >>> >> >>>> Any guidance would be appreciated, because we will not be >> able >> >> >>> >> >>>> to >> >> >>> >> >>>> make >> >> >>> >> >>>> use of >> >> >>> >> >>>> >> >> >>> >> >>>> stack/stackage without something like this! >> >> >>> >> >>>> >> >> >>> >> >>>> >> >> >>> >> >>>> Thank you >> >> >>> >> >>>> >> >> >>> >> >>>> David >> >> >>> >> >>> >> >> >>> >> >>> -- >> >> >>> >> >>> You received this message because you are subscribed to the >> >> >>> >> >>> Google >> >> >>> >> >>> Groups >> >> >>> >> >>> "haskell-stack" group. >> >> >>> >> >>> To unsubscribe from this group and stop receiving emails >> from >> >> >>> >> >>> it, >> >> >>> >> >>> send >> >> >>> >> >>> an >> >> >>> >> >>> email to [email protected]. >> >> >>> >> >>> To post to this group, send email to >> >> >>> >> >>> [email protected]. >> >> >>> >> >>> To view this discussion on the web visit >> >> >>> >> >>> >> >> >>> >> >>> >> >> >>> >> >>> >> >> >>> >> >>> >> https://groups.google.com/d/msgid/haskell-stack/35616520-204d-41f6-b1c9-e37ee1570473%40googlegroups.com. >> >> >> >> >>> >> >>> For more options, visit https://groups.google.com/d/optout. >> >> >> >>> >> > >> >> >>> >> > -- >> >> >>> >> > You received this message because you are subscribed to the >> >> >>> >> > Google >> >> >>> >> > Groups >> >> >>> >> > "haskell-stack" group. >> >> >>> >> > To unsubscribe from this group and stop receiving emails from >> it, >> >> >>> >> > send >> >> >>> >> > an >> >> >>> >> > email to [email protected]. >> >> >>> >> > To post to this group, send email to >> [email protected]. >> >> >>> >> > To view this discussion on the web visit >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > >> https://groups.google.com/d/msgid/haskell-stack/094da0e4-6eab-4c10-8842-4d3c37b3ad78%40googlegroups.com. >> >> >> >> >>> >> > >> >> >>> >> > For more options, visit https://groups.google.com/d/optout. >> >> >>> > >> >> >>> > -- >> >> >>> > You received this message because you are subscribed to the >> Google >> >> >>> > Groups >> >> >>> > "haskell-stack" group. >> >> >>> > To unsubscribe from this group and stop receiving emails from >> it, >> >> >>> > send >> >> >>> > an >> >> >>> > email to [email protected]. >> >> >>> > To post to this group, send email to [email protected]. >> >> >> >>> > To view this discussion on the web visit >> >> >>> > >> >> >>> > >> >> >>> > >> https://groups.google.com/d/msgid/haskell-stack/674862b8-2d46-4c6e-ac93-0b1cdfde0ec5%40googlegroups.com. >> >> >> >> >>> > >> >> >>> > For more options, visit https://groups.google.com/d/optout. >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "haskell-stack" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> send >> >> > an >> >> > email to [email protected]. >> >> > To post to this group, send email to [email protected]. >> >> > To view this discussion on the web visit >> >> > >> >> > >> https://groups.google.com/d/msgid/haskell-stack/17cc8023-090c-4cf8-982e-c8e743df65ed%40googlegroups.com. >> >> >> >> > >> >> > For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "haskell-stack" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected]. >> > To post to this group, send email to [email protected]. >> > To view this discussion on the web visit >> > >> https://groups.google.com/d/msgid/haskell-stack/c2b35b88-e1f7-4a87-adf1-0ac423a4d53a%40googlegroups.com. >> >> >> > >> > For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "haskell-stack" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/34ac3637-269f-4193-b8e6-1c003e757427%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
