How To Create A Workspace

A quick summary of How To Create A Workspace

If you are developing multiple plugins at the same time a "workspace" using Docker and Docker Compose can make the process easier. Creating a workspace this way also generates a PyCharm project.
The basic idea is to use the parent directory of the plugins, so that they can share a common npm installation and docker setup.

Setting up a workspace from exisiting product

The generator contains a special command to setup a workspace based on a Phovea product definition. This will clone all the repositories, download data packages, install dependencies, and build docker for you. By convention, product definition repositories are ending with the _product suffix.

yo phovea:setup-workspace Caleydo/example_product

Attention!

Replace the product definition (“Caleydo/example_product”) with a real one.


For example, setting up the workspace for TaCo:

yo phovea:setup-workspace taco_product

The first part of the command (yo) is the Yeoman command-line tool. The second part tells Yeoman that it should use the “phovea” generator and execute “setup-workspace” sub-generator. The third part is the product definition from which the workspace should be created. A prompt will appear asking if the dependencies should be cloned via SSH or HTTPS. Cloning via HTTPS should work everywhere, even behind a proxy, but it requires to authenticate via username and password to access private git repositories / plugins. When cloning via SSH authentication is handled automatically with private and public keys, however when being behind a proxy this method of cloning may not work. See also the documentation on GitHub.

This command will create a new folder, which is named after the product without the _product suffix (in this case example). Inside this directory all needed plugins will be cloned. Cloning private repositories requires having the correct access rights and authentication either with SSH or username and password. After cloning all needed repositories their dependencies are collected and merged into the corresponding file inside the root directory of the workspace (for more information see the Folder structure section). The generator then executes the command npm install to install the node modules. Finally the docker containers will be downloaded and installed. The corresponding command to achieve this is docker-compose build. Building the containers can fail when being behind a proxy server because Docker does not know anything about the proxy settings. If this is the case, please refer to Running Docker behind a Proxy Server. When the build fails it can be restarted using the following command inside the workspace:

docker-compose build

The workspace can now be opened using PyCharm.

Create a new workspace without existing product

  1. Create a directory where you want to have all the sources:
mkdir workspace
cd workspace
  1. Optional: Clone a repository, you want to change
git clone https://github.com/phovea/phovea_server.git
# AND/OR clone any other repos that you will need.
  1. Resolve phovea dependencies and create workspace
yo phovea:resolve

yo phovea:workspace
npm install
docker-compose up

To add other Phovea plugins, halt Docker by hitting ctrl-C, and then:

# For example, if you want to support interactions with a SQL database:
git clone git@github.com:phovea/phovea_data_sql.git
yo phovea:resolve

yo phovea:workspace
npm install
docker-compose build api
docker-compose up

Folder structure

┣━ Root folder (e.g. example)
┃    ┣━ _backup/
┃    ┣━ _data/
┃    ┣━ ordino/
┃    ┣━ phovea_server/
┃    ┣━ phovea_clue/
┃    ┣━ …
┃    ┣━ phovea.js
┃    ┣━ phovea_registry.js
┃    ┣━ package.json
┃    ┣━ requirements.txt
┃    ┣━ docker-compose.yml
┃    ┣━ docker-compose-patch.yml
┃    ┣━ forEach.sh
┃    ┣━ forEach.cmd
┃    ┣━ docker-backup.sh
┃    ┣━ docker-backup.cmd
┃    ┣━ …

The folders inside the workspace root are plugins. An exception to this are folders starting with an underscore (_), which are not included in the workspace by default. These folders are not plugins but the place to put in database dumps (_backup) to be restored or other files, which should be served by the web server (_data). The created workspace contains some plugins by default, which are needed for the specified product. Many of the files (e.g., package.json, requirements.txt, docker-compose.yml, ...) are automatically generated. For example, each plugin maintains its own package.json file, but when installing the node modules for the workspace a package.json file, which combines the dependencies of all plugins, is needed. After changing one of these files the workspace needs to be updated (see Updating the workspace) such that the global workspace files include the latest changes. This step is done by default when creating the workspace.


Attention!

The folders beginning with an underscore (_) are not part of the default workspace.


Configure Workspace Files

If a proxy server needs to be configured, change the respective proxy settings inside the docker-compose-patch.yml file:

...
environment:
http_proxy=yourproxylocation
https_proxy=yourproxylocation
...

After configuring the proxy server the workspace needs to be updated, such that the proxy settings are copied into the docker-compose.yml file:

yo phovea:update

Then restart the build process:

docker-compose build

Workspace update

./forEach git pull

yo phovea:update

Restoring a database dump

The workspace contains a helper script for both Windows and Linux to restore a database dump. Place the dump under _backup and execute the following command:

./docker-backup restore

Updating the workspace

A workspace’s auto-generated files are created using the corresponding files inside each plugin. So for example the package.json file is created by taking the package.json file of each plugin and combining them. It is important to update the workspace whenever one of these files (e.g. package.json, requirements.txt, ...) changes inside a plugin, because otherwise the changes will not be available in the compiled application.

The workspace contains the forEach.sh shell script, which is used to update all plugins to the latest version. It will run through all plugins and execute git pull on the current branch.

./forEach git pull

To update the workspace’s auto-generated files, the Phovea’s Yeoman generator can be used again:

yo phovea:update

This command will check which files have changed and ask if they should be replaced.

Running the application

To compile and run the application inside a workspace, the Docker containers need to be started first because they contain the backend. The frontend application can be started using the NPM start script.

docker-compose up -d
npm start

The first command will start docker in the background (-d flag) and the current terminal window is free to execute the NPM start script. The logs from the docker containers can be printed by executing the following command:
docker-compose logs

Alternatively, docker can also be started in the foreground, where it shows the logs instantly. In this case another terminal window has to be used to start the application via the NPM start script.

docker-compose up

Usually, the application runs on localhost:8080, but in case port 8080 is already occupied the server will use the next open port (e.g., 8081). The information on which port the application is running can be seen at the beginning of the output of the start script.

Shutting the application down

First shut down the NPM start script by clicking on the terminal window that runs the script and press CTRL+C. If docker-compose is running in the foreground, it can be stopped using the same steps, otherwise it can be stopped using:

docker-compose stop

If the Docker Toolbox is used, the virtual machine needs to be stopped by executing:

docker-machine stop default