Skip to main content
Skip table of contents

Backend (middleware) development

[]

Warning Backend development and frontend development are mutually exclusive and should not be done at the same time.

Local development setup

Requirements

HCMS CSK local installation

As a first step before the actual development, the application needs to be run locally (usually on a developer's machine) in a docker-compose setup.

Developer machine

  1. Node 16

  2. yarn

    • install by npm install -g yarn

  3. A native compilation toolchain, required by some npm libraries. On OS X, this is a default part of the Xcode development environment and/or standard OS.

    • C and C++ compiler

    • GNU Make

    • python (version 3)

  4. GNU Make

  5. GNU cp on Mac OS X

    • Homebrew package coreutils or gcp

    • optional;

A testing environment for deployments

Also, prior to any customization, we strongly recommend to prepare an environment that fully imitates the future productive environment with the correct future domain and always to install a fully working application on it for testing purposes. This ensures that all customizations will work as expected and nothing is broken due to an incorrect server setup.

Steps

Deploy frontend files

Warning This setup is not meant for the frontend development. Please do not try to do both frontend and backend development at once! Running middleware without frontend files (mainly .js but also others) is possible, but will lead to limited functionality since one of the main purposes of the middleware is to server them.

  1. Run make -f deployment/docker/Makefile dist

    Alternatively, you can run the following step in the Dockerfile. It takes the hcms-client image and extracts the frontend files:

    BASH 
    
rm -rf dist
docker build -t hcms-client:local -f deployment/docker/Dockerfile .
mkdir dist
docker run --rm   hcms-client:local  tar zcf - . | (cd dist && tar zxvf - )

  2. Check that the following sources exist and note the full path to them:

    • dist/schemas | the directory with all schemas from all modules

    • dist/web | the directory with all static files to be served (all the JavaScript files, webpacked and minified)

    • dist/module-infos.json | the file with all modules and correct paths to JavaScript files

      • This file is very important, because filenames contain hash code!

Deploy backend

The following commands deploy the middleware and the HTTP server locally.

  1. cd backend

  2. yarn install

  3. yarn workspaces run build

    • Note that before compiling TypeScript, some files are generated. For this reason, this step must be run at least once.

  4. make sure that local-portal-data exists

    • mkdir local-portal-data

    • note that it is in .gitignore, so it does not pollute git workspace

    • the local-portal-data/portal.json file will contain portal configuration; it can be safely deleted and everything will return to "factory defaults"

  5. create config.json

    • use config.local.json as a template, filling (or removing) all the TODO: values

      • The only exception is the mail-server configuration (mailConfig). The application will run without it, with some limitations.

    • set webDir and moduleInfosFile to full paths to frontend build results

      • this is actually optional; remove both these properties to run without any frontend

    • set availableModules to list of available modules, or make sure it's not set at all (note that this behavior can change in future)

    • everything inside portal property is optional if you already have full portal definition (local-portal-data/portal.json)

      • usually, though, the portal.json contains placeholder of crucial information (domain, smtp server, hcms url) - removing these from config.json means that the result will be invalid (unresolved placeholders)

    • note that the config.json is in .gitignore, so creating it does not modify the git repository

  6. When using the local docker-compose setup: set the environment variable GLOBAL_AGENT_HTTP_PROXY (to use the http proxy) and NODE_TLS_REJECT_UNAUTHORIZED (to accept self-signed certificate)

    • export GLOBAL_AGENT_HTTP_PROXY=http://localhost:3128

    • export NODE_TLS_REJECT_UNAUTHORIZED=0

  7. run the backend

    • yarn run start

    • or directly node core/build/Server.js

    • or just directly run Server.js from IDE (or even Server.ts, if the IDE is able to handle typescript)

Thelocal-portal-data/portal.json file is created on the first run as this is actually a standard behaviour of the standalone variant.

Hooks module

After you deployed the backend, the hooks will be still missing. Even if added (as explained below), they won't be invoked as they would use the localhost URLs, but it is not possible to use localhost inside the Docker container in the current setup.

For this reason, a complete local testing of hooks is very tricky, if at all possible. It requires HCMS to be run locally too, and without docker, but then that eliminates all advantages of the Docker Compose setup.

It is, however, possible to develop changes, cover them by unit tests and even fully test them by crafting HTTP requests manually.

  1. cd backend/hooks

  2. yarn install

  3. yarn run build and yarn run test

  4. add ../hooks/build/MediaPortalHooks.js into the scripts array in backend/core/hooks/config.json

    JSON 
     "scripts": ["../build/ExpressAdapter.js", "../hooks/build/MediaPortalHooks.js"]

  5. run the backend using yarn run start

    • all the /hook/* endpoints will be now available

What you can customize and where you should do it

Below you will find an information about the functonalities that are customizable and where you can add your custom code in the respective source code.

Schema development

The most important (and usually the first) part of development is creating new schemas and changing existing ones. This is covered by a separate document.

Portal Configuration, multi-domain application

This document describes the concepts of portal configuration and domain lookup, and the reason why the code is quite complicated.

In most cases, there is only one single DNS domain and none of this is actually necessary in production. Even in that case, however, the local development needs to support alternative domains (localhost:3000, localhost:3390).

Main startup script and configuration

What it does

Main entrypoint of the whole application is the backend/core/src/Server.ts, which:

  1. loads the configuration file config.json and the environment variable CONFIG and merges them together

    • the CONFIG variable is optional, but the config.json is not

  2. initializes the HTTP server (express), in particular:

    • serving of static files (mostly compiled javascript), with special support of compressed javascript

    • cookie support

    • trust of reverse proxy headers like X-Forwarded-For: set to always enabled

      • except X-Forwarded-Host, which is disabled by default (it is not set by many reverse proxies, leaving a huge security hole when enabled)

    • automated addition of such security headers as Strict-Transport-Security, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy, etc.

    • support for request bodies: either JSON or raw bytes

      • limited to 10 MB (note: real data upload never goes through nodejs middleware!)

    • CORS headers handling, if enabled in configuration

    • /health.txt endpoint used as a basic health-check

  3. runs all the scripts specified by the configuration property scripts in the config.json

    • normally, this is the core ExpressAdapter.ts and the webhook handler Hook.ts

The configuration file is parsed as a generic any type, but it should conform to the MainConfiguration interface in Types.ts. Standard configuration properties are also described in detail here.

What can be customized

  1. Headers: you can add more security headers. Check the official browser documentation for available headers, e.g., https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#security.

  2. Request bodies: you can add support for other types (multipart to handle direct form submit, XML, or YAML) or change a limit for the request body size.

REST API

The script ExpressAdapter.ts serves as the main entrypoint to the portal core itself. The factory function creates HTTP request handles. factory is based upon express.Router() method of Express.js. Please consult the official Express.js documentation for more information. This is one of the places to add a completely new API. The default four ones, however, should be sufficient in most cases. See RestApi.ts. Most of the ExpressAdapter.ts's code is actually handling a very special edge case: an application running on several domains. This special case, however, is not officially supported because it is quite hard to configure it correctly and safely.

The best place is marked by special comment in the ExpressAdapter.ts file, search for CUSTOM API HERE.

Class MediaApi in the MiddlewareApi.ts script can server as a template for any new API implementations. It can be also directly enhanced if a new image- or video- related endpoint is needed.

The function makeJwtValidator in the JwtValidation.ts script is a convenient way to ensure that the API segment is accessible only for the logged-in users. The function can be simply added before any real request handler (the usual express pattern). As a side effect, it also adds the decoded token (ie all its claims) to the request object, so the handler can use it for some fine grained access control.

Development cycle and link to frontend

It is also strongly recommended setting up an automatic deployment pipeline (CI/CD) for deploying custom code, in accordance with the established best practices. Although the backend development is usually a smaller part of the whole project effort.

  1. Change the code (TypeScript) and/or schema (the HCMS schema file).

  2. Compile and run the node application

  3. Check that the schemas are all applied without errors.

    • In case of error, fix the schema and restart the application.

  4. Check that the application works

    • Invoke http://localhost:3000/data/config.json

  5. Test the new/modified REST endpoint (if any).

  6. Build the new docker image of the whole application and deploy it to the dev environment, if any.

    • Ideally, this is done by a CI/CD server.

  7. Check that changes are also available on the dev environment.

  8. When ready, deploy the new image to the testing/production environment according to any agreed process.

Warning: All frontend developers will need the new image too. They can either build it locally (inefficient) or just docker pull it from a shared repository (much more efficient). Please communicate your changes accordingly!

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close