Thanks, I've linked to that from my offline-stack repo. On Mon, Mar 6, 2017 at 8:35 PM, Alexander T <[email protected]> wrote: > Renamed repository: > https://github.com/AleXoundOS/haskell-stack-mirror-script > > > On Saturday, March 4, 2017 at 11:01:24 PM UTC+3, Alexander T wrote: >> >> Hi again! >> >> I tried your commit that skips downloading if file is present and it >> works. >> However, imho this is still a suboptimal solution. >> >> This is what I came up with. A script that downloads every url from stack >> setup >> yaml and all packages from hackage index. Here is the repository: >> https://github.com/AleXoundOS/haskell-dev-mirror-script . In other words, >> you >> get everything you need to develop software in Haskell except remote docs. >> >> After everything is downloaded it uses 20GiB of space and takes about 15 >> minutes to verify files integrity. Not that much. Tbh, I was afraid it >> will take >> more space and time. >> >> Downloaded files are placed into proper subdirectories in "mirror" >> directory >> that is ready to be served by http server. Clients need to get generated >> config.yaml with all the paths adjusted to the structure of "mirror" >> directory. >> And finally, to bypass the well-known stack bug ignoring "setup-info" >> field, >> clients need the generated stack setup yaml for "stack setup" operations. >> >> On Sunday, February 19, 2017 at 6:27:56 PM UTC+3, Neil Mitchell wrote: >>> >>> > The name of github repository "offline-stack" is a bit misleading. And >>> > repository descriptions "Install Stack without internet access", "demo >>> > of >>> > using Haskell Stack without internet access" confuse further. >>> >>> Sorry, I'll tweak the text. It's meant to be a demo of the steps you >>> need to install stack without internet access - not a ready rolled >>> solution. In particular, it isn't the latter because: >>> >>> 1) If your machine has full internet access then this solution isn't >>> required. You have to have some way to initially get the files onto >>> the machine, and that will vary by location, so there's no chance of >>> doing something standard. >>> >>> 2) Most places have existing file serving servers, and in my case I >>> just reused one of them. No point firing up a new Haskell server just >>> to serve up a few files, when putting it on an Apache/IIS server is >>> probably easier. >>> >>> > I thought that the http-redirect server is able to run offline, i.e. >>> > without >>> > internet access at all once all the required files were downloaded by >>> > passing requests through proxy. >>> >>> You could easily do that - in fact I've just pushed a 1 line change to >>> do that - >>> https://github.com/ndmitchell/offline-stack/commit/86c6ff60dc7df9e20363c1e4f4cbcd79a6543d2f >>> (entirely untested). >>> >>> > Thank you for all your work in this direction. >>> > So, do you think it is possible to turn this idea into an offline >>> > server? Do >>> > you see any pitfalls building up such system for production? Like, does >>> > the >>> > theory of idea need major adjustments in your opinion? >>> >>> In my case, step 1 doesn't work, and step 2 is more hassle than >>> reusing existing solutions. However, if you can get around step 1 >>> (which it seems like you can), and step 2 is no big deal, then it >>> should work. >>> >>> Thanks, Neil >>> >>> > >>> > On Sunday, February 19, 2017 at 5:46:49 PM UTC+3, Neil Mitchell wrote: >>> >> >>> >> The fact that you can't do "stack new" is known, and raised at >>> >> https://github.com/commercialhaskell/stack/issues/2804. >>> >> >>> >> Note that the http-redirect tool I wrote _always_ goes to the >>> >> internet, it never serves up mirrored content. If you want to run on a >>> >> server which doesn't have internet access you'll need to modify that >>> >> code, or spin up another type of server to serve up the content. The >>> >> purpose of http-redirect is to robustly test my configuration steps - >>> >> I didn't write it with the idea of production use in mind. >>> >> >>> >> Thanks, Neil >>> >> >>> >> >>> >> On Sun, Feb 19, 2017 at 11:57 AM, Alexander T <[email protected]> >>> >> wrote: >>> >> > Ah, also forgot to clarify that the nightly-2017-02-02.yaml file was >>> >> > already >>> >> > previously retrieved by the http-redirect server and actually exists >>> >> > in >>> >> > it's >>> >> > mirror directory (with filename renamed according to server's >>> >> > conventions). >>> >> > >>> >> > Server side >>> >> > $ stat >>> >> > >>> >> > >>> >> > mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml >>> >> > File: >>> >> > >>> >> > >>> >> > mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml >>> >> > Size: 5377585 Blocks: 10504 IO Block: 4096 regular >>> >> > file >>> >> > Device: 2dh/45d Inode: 5819 Links: 1 >>> >> > Access: (0644/-rw-r--r--) Uid: ( 1004/ alex) Gid: ( 100/ >>> >> > users) >>> >> > Access: 2017-02-17 05:02:18.946927356 +0300 >>> >> > Modify: 2017-02-17 05:02:18.936927354 +0300 >>> >> > Change: 2017-02-19 03:30:53.923441888 +0300 >>> >> > Birth: - >>> >> > >>> >> > On Sunday, February 19, 2017 at 4:29:49 AM UTC+3, Alexander T wrote: >>> >> >> >>> >> >> Yes I did. >>> >> >> >>> >> >> Server side >>> >> >> $ env | grep STACK_ROOT >>> >> >> STACK_ROOT=/data/srv/stack_root >>> >> >> $ pwd >>> >> >> /data/srv >>> >> >> $ ls >>> >> >> http-redirect mirror stack_root >>> >> >> $ stack path | grep config >>> >> >> config-location: /data/srv/stack_root/global-project/stack.yaml >>> >> >> $ cat /data/srv/stack_root/config.yaml >>> >> >> urls: >>> >> >> latest-snapshot: >>> >> >> >>> >> >> http://localhost:3000/https/www.stackage.org/download/snapshots.json >>> >> >> lts-build-plans: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/ >>> >> >> nightly-build-plans: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/ >>> >> >> >>> >> >> package-indices: >>> >> >> - name: Hackage >>> >> >> download-prefix: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/ >>> >> >> http: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz >>> >> >> >>> >> >> >>> >> >> Client side >>> >> >> $ env | grep STACK_ROOT >>> >> >> $ cat ~/.stack/config.yaml >>> >> >> urls: >>> >> >> latest-snapshot: >>> >> >> >>> >> >> http://localhost:3000/https/www.stackage.org/download/snapshots.json >>> >> >> lts-build-plans: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/ >>> >> >> nightly-build-plans: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/ >>> >> >> >>> >> >> package-indices: >>> >> >> - name: Hackage >>> >> >> download-prefix: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/ >>> >> >> http: >>> >> >> >>> >> >> >>> >> >> http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz >>> >> >> >>> >> >> When I try to run "stack path" on client side without internet >>> >> >> connection >>> >> >> on both sides >>> >> >> $ stack path >>> >> >> Downloading nightly-2017-02-02 build plan ...RedownloadFailed >>> >> >> Request { >>> >> >> host = "localhost" >>> >> >> port = 3000 >>> >> >> secure = False >>> >> >> requestHeaders = [] >>> >> >> path = >>> >> >> >>> >> >> >>> >> >> "/https/raw.githubusercontent.com/fpco/stackage-nightly/master//nightly-2017-02-02.yaml" >>> >> >> queryString = "" >>> >> >> method = "GET" >>> >> >> proxy = Nothing >>> >> >> rawBody = False >>> >> >> redirectCount = 10 >>> >> >> responseTimeout = ResponseTimeoutDefault >>> >> >> requestVersion = HTTP/1.1 >>> >> >> } >>> >> >> "/home/alex/.stack/build-plan/nightly-2017-02-02.yaml" (Response >>> >> >> {responseStatus = Status {statusCode = 500, statusMessage = >>> >> >> "Internal >>> >> >> Server >>> >> >> Error"}, responseVersion = HTTP/1.0, responseHeaders = >>> >> >> [("Date","Sun, >>> >> >> 19 Feb >>> >> >> 2017 01:23:48 >>> >> >> GMT"),("Server","Warp/3.2.11"),("Content-Type","text/plain; >>> >> >> charset=utf-8")], responseBody = (), responseCookieJar = CJ {expose >>> >> >> = >>> >> >> []}, >>> >> >> responseClose' = ResponseClose}) >>> >> >> >>> >> >> At the same time http-redirect server throws: >>> >> >> HttpExceptionRequest Request { >>> >> >> host = "raw.githubusercontent.com" >>> >> >> port = 443 >>> >> >> secure = True >>> >> >> requestHeaders = [] >>> >> >> path = >>> >> >> "/fpco/stackage-nightly/master//nightly-2017-02-02.yaml" >>> >> >> queryString = "" >>> >> >> method = "GET" >>> >> >> proxy = Nothing >>> >> >> rawBody = False >>> >> >> redirectCount = 10 >>> >> >> responseTimeout = ResponseTimeoutDefault >>> >> >> requestVersion = HTTP/1.1 >>> >> >> } >>> >> >> (ConnectionFailure getAddrInfo: does not exist (Name or service >>> >> >> not >>> >> >> known)) >>> >> >> >>> >> >> On Friday, February 17, 2017 at 9:48:46 AM UTC+3, Neil Mitchell >>> >> >> wrote: >>> >> >>> >>> >> >>> > Thank you. I tried this, http-redirect server builds fine and >>> >> >>> > runs. >>> >> >>> > This is >>> >> >>> > what I figured out. >>> >> >>> >>> >> >>> > Once I did "stack setup", "stack build" through the proxy (with >>> >> >>> > corresponding config.yaml), I can reproduce it on the same >>> >> >>> > system >>> >> >>> > without >>> >> >>> > internet connection. >>> >> >>> > But if I take a fresh system with empty global stack root (e.g. >>> >> >>> > ~/.stack on >>> >> >>> > GNU/Linux) except solely config.yaml, it immediately fails. >>> >> >>> > http-redirect >>> >> >>> > throws this: >>> >> >>> >>> >> >>> Did you copy the config.yaml from the offline-stack repo into >>> >> >>> STACK_ROOT? Is your STACK_ROOT actually where you think it is? >>> >> >>> (use >>> >> >>> "stack path" to see). >>> >> >>> >>> >> >>> > "stack new" fails in both cases with: >>> >> >>> > Here I can see that this request is not processed by the >>> >> >>> > http-redirect, >>> >> >>> > it >>> >> >>> > keeps silent when stack throws this exception. >>> >> >>> > >>> >> >>> > Maybe I missed something? >>> >> >>> >>> >> >>> stack new isn't supported - I imagine you have to have another >>> >> >>> redirect in config.yaml. Pull requests welcome! I've also raised a >>> >> >>> ticket at https://github.com/ndmitchell/offline-stack/issues/1. >>> >> >>> >>> >> >>> Thanks, Neil >>> >> >>> >>> >> >>> > >>> >> >>> > On Wednesday, February 15, 2017 at 11:04:36 PM UTC+3, Neil >>> >> >>> > Mitchell >>> >> >>> > wrote: >>> >> >>> >> >>> >> >>> >> > Please, can you describe all operations in detail. >>> >> >>> >> >>> >> >>> >> Describing is fragile. I've created a github project that >>> >> >>> >> describes >>> >> >>> >> it. See the README at >>> >> >>> >> https://github.com/ndmitchell/offline-stack#readme, plus the >>> >> >>> >> code >>> >> >>> >> which executes it at >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> https://github.com/ndmitchell/offline-stack/blob/master/appveyor.yml. >>> >> >>> >> I test those steps on Appveyor, so it's guaranteed to work/ >>> >> >>> >> >>> >> >>> >> > The whole idea of using haskell with stack/cabal offline is >>> >> >>> >> > crucial >>> >> >>> >> > for >>> >> >>> >> > work >>> >> >>> >> > at many companies. >>> >> >>> >> > First of all it's not clear how to obtain all the data needed >>> >> >>> >> > for >>> >> >>> >> > stack >>> >> >>> >> > and >>> >> >>> >> > cabal to serve locally. >>> >> >>> >> >>> >> >>> >> Agreed! Hopefully everything is now 100% clear. >>> >> >>> >> >>> >> >>> >> > >>> >> >>> >> > On Friday, February 10, 2017 at 10:39:51 AM UTC+3, Neil >>> >> >>> >> > Mitchell >>> >> >>> >> > wrote: >>> >> >>> >> >> >>> >> >>> >> >> Hi David, >>> >> >>> >> >> >>> >> >>> >> >> Its not that bad, now the steps are known, so go ahead and >>> >> >>> >> >> email >>> >> >>> >> >> me >>> >> >>> >> >> if >>> >> >>> >> >> you >>> >> >>> >> >> get stuck. Steps are: >>> >> >>> >> >> >>> >> >>> >> >> * create stack root >>> >> >>> >> >> >>> >> >>> >> >> * copy config.yaml to the root >>> >> >>> >> >> >>> >> >>> >> >> * run stack setup with the setup yaml flag >>> >> >>> >> >> >>> >> >>> >> >> Figuring our these steps took weeks :) >>> >> >>> >> >> >>> >> >>> >> >> Thanks, Neil >>> >> >>> >> >> >>> >> >>> >> >> >>> >> >>> >> >> >>> >> >>> >> >> On Thu, 9 Feb 2017 at 11:47 pm, David Sicilia >>> >> >>> >> >> <[email protected]> >>> >> >>> >> >> wrote: >>> >> >>> >> >>> >>> >> >>> >> >>> Hi Neil, >>> >> >>> >> >>> >>> >> >>> >> >>> Thank you for raising the ticket, I just gave it a +1, if >>> >> >>> >> >>> that >>> >> >>> >> >>> helps >>> >> >>> >> >>> :-) >>> >> >>> >> >>> >>> >> >>> >> >>> So would you recommend that I go ahead and try it right >>> >> >>> >> >>> now, >>> >> >>> >> >>> or is it complicated enough that I should just wait for >>> >> >>> >> >>> that >>> >> >>> >> >>> change >>> >> >>> >> >>> to be implemented? I guess what I mean is, after you >>> >> >>> >> >>> realized >>> >> >>> >> >>> which config files to change, was it straightforward from >>> >> >>> >> >>> there? >>> >> >>> >> >>> >>> >> >>> >> >>> David >>> >> >>> >> >>> >>> >> >>> >> >>> On Saturday, January 21, 2017 at 11:27:22 AM UTC-5, David >>> >> >>> >> >>> Sicilia >>> >> >>> >> >>> wrote: >>> >> >>> >> >>>> >>> >> >>> >> >>>> Hi there, >>> >> >>> >> >>>> >>> >> >>> >> >>>> >>> >> >>> >> >>>> I am behind a firewall with strict rules about downloading >>> >> >>> >> >>>> software >>> >> >>> >> >>>> from >>> >> >>> >> >>>> the internet, >>> >> >>> >> >>>> >>> >> >>> >> >>>> so I would like to know if it is possible to download an >>> >> >>> >> >>>> entire >>> >> >>> >> >>>> stackage >>> >> >>> >> >>>> LTS snapshot >>> >> >>> >> >>>> >>> >> >>> >> >>>> in one go and then serve it on a local server, to which we >>> >> >>> >> >>>> could >>> >> >>> >> >>>> then >>> >> >>> >> >>>> point the stack tool. >>> >> >>> >> >>>> >>> >> >>> >> >>>> >>> >> >>> >> >>>> By "snapshot" I would be referring to the source code for >>> >> >>> >> >>>> all >>> >> >>> >> >>>> packages >>> >> >>> >> >>>> because >>> >> >>> >> >>>> >>> >> >>> >> >>>> we'd need to draw from that local server to build on both >>> >> >>> >> >>>> Linux >>> >> >>> >> >>>> and >>> >> >>> >> >>>> Windows. >>> >> >>> >> >>>> >>> >> >>> >> >>>> >>> >> >>> >> >>>> For example, perhaps we'd be able to go to a server within >>> >> >>> >> >>>> the >>> >> >>> >> >>>> firewall >>> >> >>> >> >>>> and run >>> >> >>> >> >>>> >>> >> >>> >> >>>> "stack new", then edit the yaml file to point it to a >>> >> >>> >> >>>> local >>> >> >>> >> >>>> stackage >>> >> >>> >> >>>> server with given >>> >> >>> >> >>>> >>> >> >>> >> >>>> resolver number, then it would just behave normally from >>> >> >>> >> >>>> then >>> >> >>> >> >>>> on, >>> >> >>> >> >>>> except >>> >> >>> >> >>>> always >>> >> >>> >> >>>> >>> >> >>> >> >>>> downloading packages from the local server. >>> >> >>> >> >>>> >>> >> >>> >> >>>> >>> >> >>> >> >>>> What about compilers? I know that stack also downloads the >>> >> >>> >> >>>> compilers, >>> >> >>> >> >>>> so >>> >> >>> >> >>>> I guess >>> >> >>> >> >>>> >>> >> >>> >> >>>> those would have to be included in the downloaded >>> >> >>> >> >>>> snapshot-package >>> >> >>> >> >>>> for >>> >> >>> >> >>>> various >>> >> >>> >> >>>> >>> >> >>> >> >>>> platforms? >>> >> >>> >> >>>> >>> >> >>> >> >>>> >>> >> >>> >> >>>> Any guidance would be appreciated, because we will not be >>> >> >>> >> >>>> able >>> >> >>> >> >>>> to >>> >> >>> >> >>>> make >>> >> >>> >> >>>> use of >>> >> >>> >> >>>> >>> >> >>> >> >>>> stack/stackage without something like this! >>> >> >>> >> >>>> >>> >> >>> >> >>>> >>> >> >>> >> >>>> Thank you >>> >> >>> >> >>>> >>> >> >>> >> >>>> David >>> >> >>> >> >>> >>> >> >>> >> >>> -- >>> >> >>> >> >>> You received this message because you are subscribed to the >>> >> >>> >> >>> Google >>> >> >>> >> >>> Groups >>> >> >>> >> >>> "haskell-stack" group. >>> >> >>> >> >>> To unsubscribe from this group and stop receiving emails >>> >> >>> >> >>> from >>> >> >>> >> >>> it, >>> >> >>> >> >>> send >>> >> >>> >> >>> an >>> >> >>> >> >>> email to [email protected]. >>> >> >>> >> >>> To post to this group, send email to >>> >> >>> >> >>> [email protected]. >>> >> >>> >> >>> To view this discussion on the web visit >>> >> >>> >> >>> >>> >> >>> >> >>> >>> >> >>> >> >>> >>> >> >>> >> >>> >>> >> >>> >> >>> https://groups.google.com/d/msgid/haskell-stack/35616520-204d-41f6-b1c9-e37ee1570473%40googlegroups.com. >>> >> >>> >> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >>> >> > >>> >> >>> >> > -- >>> >> >>> >> > You received this message because you are subscribed to the >>> >> >>> >> > Google >>> >> >>> >> > Groups >>> >> >>> >> > "haskell-stack" group. >>> >> >>> >> > To unsubscribe from this group and stop receiving emails from >>> >> >>> >> > it, >>> >> >>> >> > send >>> >> >>> >> > an >>> >> >>> >> > email to [email protected]. >>> >> >>> >> > To post to this group, send email to >>> >> >>> >> > [email protected]. >>> >> >>> >> > To view this discussion on the web visit >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > https://groups.google.com/d/msgid/haskell-stack/094da0e4-6eab-4c10-8842-4d3c37b3ad78%40googlegroups.com. >>> >> >>> >> > >>> >> >>> >> > For more options, visit https://groups.google.com/d/optout. >>> >> >>> > >>> >> >>> > -- >>> >> >>> > You received this message because you are subscribed to the >>> >> >>> > Google >>> >> >>> > Groups >>> >> >>> > "haskell-stack" group. >>> >> >>> > To unsubscribe from this group and stop receiving emails from >>> >> >>> > it, >>> >> >>> > send >>> >> >>> > an >>> >> >>> > email to [email protected]. >>> >> >>> > To post to this group, send email to [email protected]. >>> >> >>> > To view this discussion on the web visit >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > https://groups.google.com/d/msgid/haskell-stack/674862b8-2d46-4c6e-ac93-0b1cdfde0ec5%40googlegroups.com. >>> >> >>> > >>> >> >>> > For more options, visit https://groups.google.com/d/optout. >>> >> > >>> >> > -- >>> >> > You received this message because you are subscribed to the Google >>> >> > Groups >>> >> > "haskell-stack" group. >>> >> > To unsubscribe from this group and stop receiving emails from it, >>> >> > send >>> >> > an >>> >> > email to [email protected]. >>> >> > To post to this group, send email to [email protected]. >>> >> > To view this discussion on the web visit >>> >> > >>> >> > >>> >> > https://groups.google.com/d/msgid/haskell-stack/17cc8023-090c-4cf8-982e-c8e743df65ed%40googlegroups.com. >>> >> > >>> >> > For more options, visit https://groups.google.com/d/optout. >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "haskell-stack" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> > an >>> > email to [email protected]. >>> > To post to this group, send email to [email protected]. >>> > To view this discussion on the web visit >>> > >>> > https://groups.google.com/d/msgid/haskell-stack/c2b35b88-e1f7-4a87-adf1-0ac423a4d53a%40googlegroups.com. >>> > >>> > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "haskell-stack" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/haskell-stack/34ac3637-269f-4193-b8e6-1c003e757427%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout.
-- You received this message because you are subscribed to the Google Groups "haskell-stack" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAKcFpm%2BEF5tbLdaoDiUX-K8WpdaenDF5HLsq5x0FHSCdU0j%2BEw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
