Hi Casperites, Just some corrections from my previous email below - got the numpy version update incorrect:
Okay, so it turns out that creating the virtual environment for python using "virtualen -p python3 <name_of_env>" does not work. This is what you need to do in order to create a successful virtual environment and get your designs to build. I tested it on a brand new virtual environment and it works. Thanks to Clifford van Wyk (Peralex), Morag Brown and Jack Hickish for their assistance. The docs will definitely need to be updated - we will add this to the agenda for the CASPER meeting. The following needs to be done if you want to generate a proper virtual environment that will build in the toolflow - Kaj, I recommend the below way: 1) Create the virtual environment: "python3 -m venv <some-venv-name>". The old way of using virtualenv -p python 3 <some-venv-name> just doesn't work. Thanks to Jack and Morag for pointing this out to me. 2) Activate the environment: "source <some-venv-name>/bin/activate" 3) Go to the mlib_devel directory and edit the requirements.txt file. It should have "numpy<1.19" (thanks for your sleuth work, Jack!). Now save the file. 4) Go to mlib_devel directory and type exactly: "pip install -r requirements.txt". This will install without error or issues. if you check the site-packages in the virtual environment you will see what I mean - all the python packages will be installed properly in your virtual environment. 5) You will need to edit line 32 of castro.py (located in the same folder as mlib_devel/jasper_library), so that it is "c = yaml.load(fh, Loader=yaml.Loader) . Refer to https://github.com/yaml/pyyaml/issues/266 <https://github.com/yaml/pyyaml/issues/266>for more info on this version change issue. I tested the build with my original working virtualenv and the newly generated virtualenv and it works for both. Don't worry, I will be committing this change once I have done a bit more investigation - maybe adding versions in requirements.txt is not a bad idea for future support. You can now start your matlab session and build your designs :). Hooray! NB: I am now going to commit and do a pull request for casper-astro to review. Kind regards, Adam Isaacson South African Radio Astronomy Observatory (SARAO) Hardware Manager Cell: (+27) 825639602 Tel: (+27) 215067300 email: [email protected] On Thu, Jan 28, 2021 at 9:41 PM Adam Isaacson <[email protected]> wrote: > Hi Casperites, > > Okay, so it turns out that creating the virtual environment for python > using "virtualen -p python3 <name_of_env>" does not work. This is what you > need to do in order to create a successful virtual environment and get your > designs to build. I tested it on a brand new virtual environment and it > works. Thanks to Clifford van Wyk (Peralex), Morag Brown and Jack Hickish > for their assistance. The docs will definitely need to be updated - we will > add this to the agenda for the CASPER meeting. > > The following needs to be done if you want to generate a proper virtual > environment that will build in the toolflow - Kaj, I recommend the below > way: > > 1) Create the virtual environment: "python3 -m venv <some-venv-name>". The > old way of using virtualenv -p python 3 <some-venv-name> just doesn't work. > Thanks to Jack and Morag for pointing this out to me. > 2) Activate the environment: "source <some-venv-name>/bin/activate" > 3) Go to the mlib_devel directory and edit the requirements.txt file. It > should have "numpy<1.9" (thanks for your sleuth work, Jack!). Now save > the file. > 3) Go to mlib_devel directory and type exactly: "pip install -r > requirements.txt". This will install without error or issues. if you check > the site-packages in the virtual environment you will see what I mean - all > the python packages will be installed properly in your virtual environment. > 4) You will need to edit line 32 of castro.py (located in the same folder > as mlib_devel/jasper_library), so that it is "c = yaml.load(fh, > Loader=yaml.Loader) . Refer to https://github.com/yaml/pyyaml/issues/266 > <https://github.com/yaml/pyyaml/issues/266>for more info on this version > change issue. I tested the build with my original working virtualenv and > the newly generated virtualenv and it works for both. Don't worry, I will > be committing this change once I have done a bit more investigation - maybe > adding versions in requirements.txt is not a bad idea for future support. > > You can now start your matlab session and build your designs :). Hooray! > > Kind regards, > > Adam Isaacson > South African Radio Astronomy Observatory (SARAO) > Hardware Manager > Cell: (+27) 825639602 > Tel: (+27) 215067300 > email: [email protected] > > > > On Thu, Jan 28, 2021 at 9:55 AM Adam Isaacson <[email protected]> wrote: > >> Hi David, >> >> Yes, it was a bit of a rabbit hole exercise, but we (between Kaj and >> Peralex) worked our way through it and now debugging other issues :). >> >> You raise a valid point. There is no reason we shouldn't port to Ubuntu >> 18.04LTS or higher. It makes sense with what Kaj and Peralex are >> experiencing. >> >> Please note, my team (SARAO) is baselining the SKARAB work at Ubuntu >> 16.04LTS. We are coming to the end of the MeerKAT development for the F and >> X engines. I do believe there are those that have got the toolflow to work >> on Ubuntu 18.04LTS. I know Wesley New has struggled to get the SKARAB >> toolflow to work using Ubuntu 18.04LTS. I haven't tried it on my side yet. >> >> My team's focus this year is to update the toolflow to work with Vitis >> and target those Alveo boards. This will definitely involve moving to a >> newer version of Ubuntu. For now that is likely to be Ubuntu 18.04LTS, >> Vitis 2020.2 and Vivado 2020.2. >> >> I think this is something we should discuss at the next CASPER meeting on >> the 11th Feb. We should allocate a resource for this, but SARAO is focusing >> on other areas for now. >> >> Thanks for your email :). >> >> Kind regards, >> >> Adam Isaacson >> South African Radio Astronomy Observatory (SARAO) >> Hardware Manager >> Cell: (+27) 825639602 >> Tel: (+27) 215067300 >> email: [email protected] >> >> >> >> On Thu, Jan 28, 2021 at 12:09 AM David MacMahon <[email protected]> >> wrote: >> >>> Wow, thanks for these truly awesome forensics, Adam! It sounds like you >>> went down a rabbit hole and lived to tell the tale. I'm sure many of us >>> will benefit from these details. >>> >>> Sorry if this is a FAQ, but what are the prospects for moving the tool >>> flow beyond Ubuntu 16.04? That release is really starting to show its age, >>> especially since its stock Python3 is only 3.5. (Though to be fair, Python >>> is at least partly to blame due to its poor track record on version >>> compatibility. :P) >>> >>> Cheers, >>> Dave >>> >>> On Jan 27, 2021, at 13:00, Adam Isaacson <[email protected]> wrote: >>> >>> Hi Kaj, >>> >>> I am including the CASPER community in this email thread as it applies >>> to everyone. >>> >>> Interesting, so I have run into another person with the same virtualenv >>> install issue that you encountered as shown in red below. I have been >>> helping him debug too on a new machine and I am pleased to say that his >>> virtualenv is now working. It seems like the new Ubuntu 16.04LTS installs >>> come with a newer version of virtualenv which is not compatible with python >>> 3.5 - fun, fun, fun. This definitely applies to all Casperites who are >>> installing the toolflow on their new machines. >>> >>> kjwiik@casperx:~/work$ virtualenv -p python3 casper_venv >>> Traceback (most recent call last): >>> File "/usr/local/bin/virtualenv", line 7, in <module> >>> from virtualenv.__main__ import run_with_catch >>> File >>> "/home/kjwiik/.local/lib/python3.5/site-packages/virtualenv/__init__.py", >>> line 3, in <module> >>> from .run import cli_run, session_via_cli >>> File >>> "/home/kjwiik/.local/lib/python3.5/site-packages/virtualenv/run/__init__.py", >>> line 13, in <module> >>> from .plugin.activators import ActivationSelector >>> File >>> "/home/kjwiik/.local/lib/python3.5/site-packages/virtualenv/run/plugin/activators.py", >>> line 6, in <module> >>> from .base import ComponentBuilder >>> File >>> "/home/kjwiik/.local/lib/python3.5/site-packages/virtualenv/run/plugin/base.py", >>> line 9, in <module> >>> from importlib_metadata import entry_points >>> File >>> "/home/kjwiik/.local/lib/python3.5/site-packages/importlib_metadata/__init__.py", >>> line 88 >>> dist: Optional['Distribution'] = None >>> ^ >>> SyntaxError: invalid syntax >>> kjwiik@casperx:~/work$ >>> >>> I want you to check the following versions for me: >>> >>> 1) virtualenv -> type "virtualenv --version" at the prompt. You should >>> get 16.7.5. If not then you will need to uninstall the virtualenv by >>> typing: "pip3 uninstall virtualenv" and then we will need to install the >>> 16.7.5 version. To install an exact version then type: "pip3 install >>> virtualenv==16.7.5". The use of "sudo" with "-H" arguments may or may >>> not be needed. The installs will guide you. >>> >>> 2) What is your Ubuntu 16.04LTS version? -> type "lsb_release -a". I am >>> using Ubuntu 16.04.7 LTS, Xenial. >>> >>> 3) What is your pip3 version? -> type "pip3 --version". I am using pip >>> 8.1.1. >>> >>> 4) What is your python version? -> type: "python --version". I am using >>> python version 2.7.12 >>> >>> 5) What is your python3 version? -> type: "python3 --version". I am >>> using python3 version 3.5.2. >>> >>> Please make sure all these versions are the same before continuing. I >>> suspect that the latest Ubuntu 16.04LTS installs have made some upgrades >>> that are incompatible. >>> >>> Once these versions are as above then try the following step to create >>> the virtual env: >>> >>> 1) Type: "virtualenv -p python3 <name_of_virtual_env>". This should >>> create a folder with the same name as "name_of_virtual_env". There >>> should be no issues with the install. It should complete without issues. >>> 2) To activate the virtual environment -> type: "source >>> <path_to_folder>/<name_of_virtual_env>/bin/activate >>> 3) Once activated then you can deactivate it by typing "deactivate". >>> 4) If the virtual environment is activated then if you type "python" you >>> should get version 3.5.2 >>> 5) If the virtual environment is deactivated then if you type "python" >>> you should get 2.7.12. >>> >>> I think if you can achieve this then you are ready to install the >>> toolflow by following these steps: >>> >>> 1) >>> https://casper-toolflow.readthedocs.io/en/latest/src/Installing-the-Toolflow.html# >>> 2) >>> https://casper-toolflow.readthedocs.io/en/latest/src/How-to-install-Matlab.html >>> 3) >>> https://casper-toolflow.readthedocs.io/en/latest/src/How-to-install-Xilinx-Vivado.html >>> 4) >>> https://casper-toolflow.readthedocs.io/en/latest/src/Configuring-the-Toolflow.html >>> 5) >>> https://casper-toolflow.readthedocs.io/en/latest/src/Running-the-Toolflow.html >>> 6) https://casper-toolflow.readthedocs.io/projects/casperfpga/en/latest/ >>> (This >>> is how to install casperfpga) >>> >>> @Jonathon Kocz <[email protected]> : we will need to update the docs to >>> reflect this new information. I will wait until both parties have working >>> systems before updating the ReadtheDocs. One of the parties is using a >>> proxy and has some additional information on how to install with a proxy >>> that will be useful for other casperites. >>> >>> I hope this helps! >>> >>> Kind regards, >>> >>> Adam Isaacson >>> South African Radio Astronomy Observatory (SARAO) >>> Hardware Manager >>> Cell: (+27) 825639602 >>> Tel: (+27) 215067300 >>> email: [email protected] >>> >>> >>> >>> On Sun, Jan 24, 2021 at 8:09 PM Adam Isaacson <[email protected]> >>> wrote: >>> >>>> Hi Kaj, >>>> >>>> Are you running a python virtual environment using python 3.5? Once you >>>> start the virtual environment and type python you should see version 3.5.2 >>>> or something like that. It doesn't look like it. Here is some documentation >>>> that should be helpful - make sure you follow these steps: >>>> >>>> 1) How to configure the toollflow: >>>> https://casper-toolflow.readthedocs.io/en/latest/src/Configuring-the-Toolflow.html >>>> 2) Running the toolflow: >>>> https://casper-toolflow.readthedocs.io/en/latest/src/Running-the-Toolflow.html >>>> >>>> Does your startsg.local file look like this: >>>> >>>> export XILINX_PATH=/opt/Xilinx/Vivado/2019.1export >>>> MATLAB_PATH=/usr/local/MATLAB/R2018aexport PLATFORM=lin64export >>>> JASPER_BACKEND=vivado >>>> # over-ride the MATLAB libexpat version with the OS's one.# Using >>>> LD_PRELOAD=${LD_PRELOAD}:"..." rather than just LD_PRELOAD="..."# ensures >>>> that we preserve any other settings already configuredexport >>>> LD_PRELOAD=${LD_PRELOAD}:"/usr/lib/x86_64-linux-gnu/libexpat.so" >>>> # Activate a custom python environment on loadexport >>>> CASPER_PYTHON_VENV_ON_START=/home/user/work/casper_venv >>>> >>>> >>>> You must run everything from the virtual environment in order to work. >>>> The last line in the startsg.local will automatically start your virtual >>>> environment session, but you can also do it manually e.g. source >>>> ~/venv/bin/activate will give you the following "(venv) >>>> aisaacson@adam-cm:~/work/git_work/ska-sa/2019_1/mlib_devel$". The red >>>> (venv) tells you that you are in the virtual python environment. >>>> Remember you also need to install requirements.txt while the virtual >>>> environment is running. >>>> >>>> Then all you really need to do is run "./startsg" at the prompt and it >>>> should open up Matlab where you can build your slx file by typing in >>>> "jasper" or "jasper_frontend. if you use "jasper_frontend" then it will >>>> just compile sysgen and the simulink frontend. You can then open another >>>> terminal session and under "mlib_devel" type "source startsg >>>> startsg.local". This will make sure everything is in your path. You can >>>> then copy and paste the exec_flow.py command from the Matlab command window >>>> and then it should build the middle end and backend. Remember to start the >>>> virtual environment for this terminal window as well. This should all be in >>>> the above docs, but if you are still struggling then please send us a step >>>> by step terminal screen capture, so we can see if you are doing everything >>>> correctly, thanks. >>>> >>>> I hope this helps! >>>> >>>> Kind regards, >>>> >>>> Adam Isaacson >>>> South African Radio Astronomy Observatory (SARAO) >>>> Hardware Manager >>>> Cell: (+27) 825639602 >>>> Tel: (+27) 215067300 >>>> email: [email protected] >>>> >>>> >>>> >>>> On Sun, Jan 24, 2021 at 7:15 PM Kaj Wiik <[email protected]> wrote: >>>> >>>>> Hi! >>>>> >>>>> A followup... >>>>> >>>>> I was able to start (but not run, see below...) the toolflow with the >>>>> following modifications: >>>>> >>>>> - to run jasper_frontend, I had to install python-yaml. I tried first >>>>> to >>>>> add python3-yaml but it seems that the toolflow is mixing python2 and >>>>> python3? >>>>> >>>>> - I added 'export LM_LICENSE_FILE=/opt/Xilinx/Xilinx.lic' and >>>>> 'export MLIB_DEVEL_PATH=~/mlib_devel' to startsg.local >>>>> >>>>> When matlab gave a command line and I run it, I got the following error >>>>> >>>>> message that I do not (yet!) understand: >>>>> >>>>> kjwiik@casperx:~/tmp$ source ../mlib_devel/startsg.local >>>>> kjwiik@casperx:~/tmp$ /usr/bin/python >>>>> /home/kjwiik/mlib_devel/jasper_library/exec_flow.py -m >>>>> /home/kjwiik/mlib_devel/jasper_library/test_models/test_zcu111.slx >>>>> --middleware --backend --software >>>>> Starting compile >>>>> Starting Toolflow! >>>>> Frontend is simulink >>>>> Setting compile directory: >>>>> /home/kjwiik/mlib_devel/jasper_library/test_models/test_zcu111 >>>>> /home/kjwiik/mlib_devel/jasper_library/platforms/zcu111.yaml >>>>> {'backend_target': 'vivado', 'name': 'zcu111', 'fpga': >>>>> 'xczu28dr-ffvg1517-2-e', 'mmbus_architecture': 'AXI4-Lite', 'sources': >>>>> >>>>> [], 'cfgbvs': 'GND', 'provides': ['sys_clk'], 'config_voltage': 1.8, >>>>> 'pins': {'clk_100_p': {'loc': 'AM15', 'iostd': 'LVDS'}, 'clk_100_n': >>>>> {'loc': 'AN15', 'iostd': 'LVDS'}, 'led': {'loc': ['AR13', 'AP13', >>>>> 'AR16', 'AP16', 'AP15', 'AN16', 'AN17', 'AV15'], 'iostd': 'LVCMOS18'}}, >>>>> >>>>> 'mmbus_base_address': 2684354560, 'mmbus_address_alignment': 4, >>>>> 'manufacturer': 'Xilinx', 'constraints': []} >>>>> sw_reg >>>>> Traceback (most recent call last): >>>>> File "/home/kjwiik/mlib_devel/jasper_library/exec_flow.py", line >>>>> 197, >>>>> in <module> >>>>> tf.gen_periph_objs() >>>>> File "/home/kjwiik/mlib_devel/jasper_library/toolflow.py", line 363, >>>>> >>>>> in gen_periph_objs >>>>> self.peripherals[pk], self.plat)) >>>>> File >>>>> "/home/kjwiik/mlib_devel/jasper_library/yellow_blocks/yellow_block.py", >>>>> >>>>> line 65, in make_block >>>>> return cls(blk,platform,hdl_root=hdl_root) >>>>> File >>>>> "/home/kjwiik/mlib_devel/jasper_library/yellow_blocks/yellow_block.py", >>>>> >>>>> line 92, in __init__ >>>>> self.hdl_root = os.getenv('HDL_ROOT').rstrip('/') #: The base >>>>> directory from which source file's locations are specified >>>>> AttributeError: 'NoneType' object has no attribute 'rstrip' >>>>> ----------------------------------------------------------------------- >>>>> >>>>> Could this be again a python version problem? >>>>> >>>>> Thanks, >>>>> Kaj >>>>> >>>>> On 1/22/21 6:50 PM, Kaj Wiik wrote: >>>>> > Hi! >>>>> > >>>>> > I tried a bit different route: I installed python3-numpy and >>>>> > python3-setuptools-git Ubuntu packages first and commented out numpy >>>>> >>>>> > from requirements.txt. At least the installation went fine using >>>>> these >>>>> > workarounds. >>>>> > >>>>> > About ZCU111 Slack channel, I am very interested (and my colleague >>>>> > [email protected] also), could you please send us an invitation? >>>>> > >>>>> > Many thanks for your kind help! >>>>> > >>>>> > Kaj >>>>> > >>>>> > On 1/20/21 10:22 PM, Dan Werthimer wrote: >>>>> >> >>>>> >> hi kaj, >>>>> >> >>>>> >> regarding your interest in ZCU111: >>>>> >> there's a casper slack channel on RFSOC that you might find useful. >>>>> >> wei liu recently developed a casper ADC yellow block for the ZCU111. >>>>> >> >>>>> >> best wishes, >>>>> >> >>>>> >> dan >>>>> >> >>>>> >> >>>>> >> >>>>> >> Dan Werthimer >>>>> >> Marilyn and Watson Alberts Chair >>>>> >> Astronomy Dept and Space Sciences Lab >>>>> >> University of California, Berkeley >>>>> >> >>>>> >> >>>>> >> On Wed, Jan 20, 2021 at 12:17 PM Kaj Wiik <[email protected] >>>>> >> <mailto:[email protected]>> wrote: >>>>> >> >>>>> >> >>>>> >> >>>>> >> On 20/01/2021 18:49, Jack Hickish wrote: >>>>> >> >>>>> >> > I've been using Ubuntu 18.04 LTS without issues, at least >>>>> with >>>>> >> the boards which use Vivado 2019. I'm not sure what versions of >>>>> >> python the OS came with, but I'm currently running the toolflow >>>>> in a >>>>> >> python 3.6.9 virtual env. >>>>> >> >>>>> >> I first tried with Ubuntu 18.04 LTS but I got stuck with a >>>>> problem >>>>> >> with Matlab and some system library versions. >>>>> >> >>>>> >> I think I'll change the requirements.txt file to read >>>>> "numpy<1.19" >>>>> >> and try with that first. >>>>> >> >>>>> >> I am very interested of all information and experiences with >>>>> Casper >>>>> >> on ZCU111. Thanks to Adam for pointing out of a mlib_devel >>>>> ZCU111 >>>>> >> branch, I haven't noticed that! >>>>> >> >>>>> >> Thanks! >>>>> >> >>>>> >> Kaj >>>>> >> >>>>> >> -- You received this message because you are subscribed to >>>>> the >>>>> >> Google >>>>> >> Groups "[email protected] >>>>> >> <mailto:[email protected]>" group. >>>>> >> To unsubscribe from this group and stop receiving emails from >>>>> it, >>>>> >> send an email to [email protected] >>>>> >> <mailto:casper%[email protected]>. >>>>> >> To view this discussion on the web visit >>>>> >> >>>>> >> >>>>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/020a6f84-ded9-49cc-2d4a-00a766e743be%40utu.fi >>>>> . >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> You received this message because you are subscribed to the Google >>>>> >> Groups "[email protected]" group. >>>>> >> To unsubscribe from this group and stop receiving emails from it, >>>>> send >>>>> >> an email to [email protected] >>>>> >> <mailto:[email protected]>. >>>>> >> To view this discussion on the web visit >>>>> >> >>>>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAGHS_vHQ1bTha%3DFjCe0sGddd3Gt6GrYkh8dgz_eKDWXebf87iQ%40mail.gmail.com >>>>> >>>>> >> < >>>>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAGHS_vHQ1bTha%3DFjCe0sGddd3Gt6GrYkh8dgz_eKDWXebf87iQ%40mail.gmail.com?utm_medium=email&utm_source=footer >>>>> >. >>>>> >> >>>>> > >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "[email protected]" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/335be9c2-39ca-d9f9-fb8f-960076bfffa1%40utu.fi >>>>> . >>>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "[email protected]" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CADTJ%3DnGgmuxdVex74bZPBOn0kpwxMO4rPrUyGcu_PPHvgvU8XA%40mail.gmail.com >>> <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CADTJ%3DnGgmuxdVex74bZPBOn0kpwxMO4rPrUyGcu_PPHvgvU8XA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "[email protected]" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/34A86FAB-52C2-4923-83A3-B14BD83A8F2A%40berkeley.edu >>> <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/34A86FAB-52C2-4923-83A3-B14BD83A8F2A%40berkeley.edu?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CADTJ%3DnF1vkkDJ-WcDk%2BkavUThZM4o3DDq7Cju%2B%3D5vu%3DiFWS%3D3A%40mail.gmail.com.
