Introduction

As an analyst, PM or developer you'll need a basic understanding of some specific IT technologies before you can start developing apps. It possible to skip specific sections if you already possess the knowledge in question.

Why are you working at ABB

Assuming you are a part of the intended audience of this guide it will be your job to support the development of IT systems for ABB. ABB is an agency of the Flemish government associated with the policy domain of 'Kanselarij, Bestuur, Buitenlandse Zaken en Justitie'. A department led by a minister is associated with this policy domain as well as ABB and other agencies. ABB, like all agencies, must fulfill it's duty to the Flemish people by supporting the execution of policy. In particular ABB must enforce policies associated with the provision of means promoting social cohesion; especially in situations where people are very diverse. This also includes facilitating cohesion between local governments and citizens.

A mayor way to achieve this goal of policy execution is though information technology. And that's were you come in.

In 2018 the Flemish government issued a decree specifying that local governments have to share their information with the public. Without going into detail: local governments need to share information about local decisions, regulations and many other things electronically in the format of linked data. This is why ABB develops applications to help local governments adhere to this decree. ABB also does many other things but making these applications is the principal task for development teams of which you are a part.

Will you do your part to help fulfill ABB's obligation to the Flemish population?

To make sure you have the knowledge to do a good job you need to learn the topics of this guide.

High level glossary

You might want to print this list because you will be returning back to it. Ideally you should study this and be able to explain all of the concepts in it. It will help you significantly with your work.

Organizational

• Vlaamse overheid (abbrev. 'VO', a.k.a. 'Flemish government'): The government of the Flemish region and the flemish community. Belgium has three regions: Brussels, Flanders and Wallonia. It also has three communities: Dutch, French and German. There is a federal government and each region as well as each community has its own government. This yields a total of 7 governments. The institutions of the Flemish region and the Flemish community fused into one: the Flemish government. This means that the Flemish government concerns itself with matters of both the region (w.r.t. territory) and the community (w.r.t. culture and education). It's headed by a prime minister and accountable to the Flemish parliament and the public administration. Because of the merger the total number of governments in Belgium is 6.

• Beleid (en. 'Policy'): In a political context, policy refers to the plans, strategies, principles, and actions that a government or political entity employs to address societal issues, achieve goals, and guide decision-making. It encompasses the formulation, implementation, and evaluation of measures or regulations aimed at shaping social, economic, environmental, or other aspects of public life.

• Beleidsdomein (en. 'Policy area): A policy area of the Flemish government is associated with a department and one or more agencies. It is associated with a major topic such as education or energy.

• Departement (en. 'department'): The department takes care of the preparation of policy proposals and support of the policies. It operates on the direct supervision of a minister.

• Agentschap (en. 'agency'): An agency is responsible for policy execution and has more autonomy compared to a department. ABB is an agency.

• Gewest (en. 'region'): Belgium consist of three regions: Flanders, Brussels and Wallonia. They have a parlement and a government. It's authority concerns affairs related to territory.

• Gemeenschap (en. 'community'): Belgium consists of three communities: Dutch, French and German. They have a parlement and a government. It's authority concerns affairs related to language and culture.

• ABB (full. 'Agentschap Binnenlands bestuur'): ABB is an agency of the Flemish government. It's associated policy area is 'Kanselarij, Bestuur, Buitenlandse Zaken en Justitie'. The department is called 'Department kanselarij en buitenlandse zaken'. It responsibilities are related to the provision of means which promote social cohesion; especially in situations where people are very diverse. This also includes facilitating cohesion between local governments and citizens. More info.

• LBLOD (full. 'Lokale Besluiten als geLinkte Open Data'): This is an ecosystem of open (as in 'open source') standards and systems related to information provision and sharing. This information, or 'data' is expressed as linked data. The software systems ABB provides for local governments and for the general public to fulfill its mission are a part of this ecosystem. More info

• Digiteam: This is the name if the ICT services department within ABB.

• Mandaat and mandataris (en. 'Mandate' and 'delegate'): A mandate is an official assignment that a delegate performs in service of Flemish citizens and on behalf of the government.

• Overheidsdienst (short. FOD): General term for a service provided by the government. It also refers to an older term called 'ministries'. It's only relevant and unambiguous in the context of the federal government. Not the flemish one.

• Bestuurseenheid (en. 'administrative unit'): Specific organization in Flanders which has some relation to government.

• District: Cities and provinces may consist of multiple districts. City districts are cities or urban areas that constitute a separate administrative level instead of forming a district together with other municipalities. Smaller towns around such a city or urban area may be part of a city district. Provincial distrits are purely administration related and are relevant in the election of provincial delegates.

• Provincie (en. 'province'): Administrative level associated with a part of the territory in Belgium. It has its own local government.

• Gemeente (en. 'municipality'): Lowest administrative level in Belgium associated with one or more clustered area's populated by people. It has its own local government.

• Stad (en. 'city'): Municipality in Belgium with a large number of inhabitants and which has been granted city status by royal decree.

• Loket: Software application, specifically a web app, provided to local administrative units (e.g. municipal governments) so they can fulfill their obligation to the Flemish public. Especially w.r.t. the decree stating that Flemish institutions need to publish information in both human readable and electronic format.

• Organisatie portaal (abbrev. 'OP'): Another web app developed by ABB. This particular system is used by collaborators within ABB itself in roder to manage records associated with organizations (administrative units).

• IGS (full. 'Intergemeentelijke samenwerking' en. 'cross municipal collaboration') TODO

• AGB (full. 'Autonoom gemeente bedrijf' en. 'autonomous municipal company') TODO

• APB (full. 'Autonoom provincie bedrijf' en. 'autonomous provincial company') TODO

High level technical

General knowledge applied to ABB. In my experience many people could do with a refresh of some general concepts before they dive into the docs. If you feel like skipping this section answer me this: Can you really explain how internet works? You kind of need to know this stuff before you start reasoning about and/or building internet applications.

• App: A general term for a computer program with a user facing component. Because ABB publishes apps using the internet 'app' really means 'web app'. A web app consists of a user facing part using web technology and associated systems running on a (web) server. Because the front-end part looks and feels like a desktop application the name 'app' is more suitable as opposed to a website. This is an example of an app ABB has recently released: Lokaal beslist.

• Website: A website consists of HTML documents accessible using the browser. They may contain interactive features facilitated by a programming language called Javascript. Users generally interact with a website by clicking links to other pages. Each time a link is clicked another page is loaded. A web app (cf. 'App') is a little different than a website because in that case the Javascript functionality is used to make the website behave more than a desktop app (i.e. it comes with its own runtime). Meaning that clicking elements often do not cause a reload of the page. The spectrum between websites and apps is somewhat fuzzy. But in general one might say that wikipedia is a website and an online game in the browser is a web app. Some internet services have properties of both. The protocol used is to deliver websites on the request of a browser is TCP/IP. The application layer protocol on top of TCP is called HTTP.

• HTTP (full. 'Hyper Text Transfer Protocol'): Application layer protocol based on text on top of TCP/IP. Therefore it has encoding. The default encoding is 'ISO-8859-1' (extended ASCII) but can be changed using a header. It's important to remember that this is a text based protocol so sending binary data over HTTP will require a binary encoding (such as base64). ABB uses JSON as a serialization format (not XML) for sending structured data over HTTP. Triples are sent using turtle, n-triples or n-quads. Naturally web pages are transferred using HTML. HTTP has a request format and a response format; both of which have headers. These headers inform the server (request) and the browser (response).

• Server: A computer program which listens to incoming TCP requests and sends answers. A server is not a physical machine.

• Web server: Server which implements the HTTP protocol

• Server computer: A computer specialized for running server programs. It has to run continuously and therefore must be highly reliable. In many cases a single server computer may run multiple web servers at the same time. In other cases a physical computer may run a hypervisor as an OS and run multiple virtual machines which in turn run multiple processes. A special case is docker (cf. 'docker').

• OS (full. Operating system): Computer software which manages compute resources such as memory and CPU time. Because of an OS multiple processes can run concurrently and reliably. In ABB's case the (virtual) machines which host the web services always use Linux as an OS. The versions of Linux do not come with a GUI but are controlled using a command terminal instead. You'll have to connect to these machines using SSH and be knowledgeable of the necessary commands which install software, start processes and many other things. Without basic knowledge of the linux operating system you will not be able to manage server computers effectively and/or deploy (new) ABB apps.

• Virtual machine (short. VM): A compute resource that uses software instead of a physical computer to run programs and deploy apps. One or more virtual “guest” machines run on a physical “host” machine. Each virtual machine runs its own operating system and functions separately from the other VMs, even when they are all running on the same host. This means that, for example, a virtual MacOS virtual machine can run on a physical host PC with linux. As a user you may not even know if the machine you connect with is physical or virtual because you'll only interact with the OS (Linux in our case).

• Docker: This is a set of products that provide 'OS level virtualization'. The 'docker engine' is a process that manages software packaged in containers. Each container behaves like a virtual machine in some way but it shares the kernel of the host machine running docker engine and is therefore more lightweight. Containers are useful because they package an OS, the process(es) you are interested in as well as any dependencies you might need. Thanks to docker you can run ubuntu linux hosting a node server in one container and alpine linux running an Elixir application at the same time. Docker allows ABB to package (parts of) apps more efficiently because the developers don't have to worry about installing any dependencies (specifically software libraries) on the host machine. Each ABB app uses docker to run a multitude of containers on a server computer.

• Docker image: This is a definition of all the necessary software and files, including the OS, that makes up a container. You can download docker images from repositories such as Docker hub. ABB publishes many different types of images on docker hub. From these images containers can be started.

• Docker container: A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. A container is associated with a docker image which provides everything the container needs to run. On a web server computer managed by ABB you will find many docker containers running at the same time. Try docker ps in the terminal.

• Docker compose: This is another product provided by docker which lets you manage a collection of containers. Thanks to docker compose you can start, stop and configure many containers all at the same time. The most important file which configures docker compose is called the docker compose file. This file gets interpreted and its instructions are followed in order to configure and start containers in the intended way. Any ABB app can be associated with a (substantial) docker compose file defining many different types of containers. All of these containers are associated with a single project name (for management) and a virtual network (so they can communicate with each other using IP).

• Micro service: This is a term used for a process, often packaged in a docker container, that has a limited functionality and responsibility and is therefore considered 'small'. The term micro service is always used when multiple micro services operating together make up a higher order more complex system. The latter is said to have a microservice architecture. ABB's apps are microservice based systems and consists of many micro services packaged in containers which communicate with each other. The system of microservices has one 'endpoint' to connect with and acts as one single complex web server. The communication between microservices are hidden from the outside world.

• Host: A network host is a computer or other device connected to a computer network. A host may work as a server offering information resources, services, and applications to users or other hosts on the network. Hosts are assigned at least one network address. In ABB hosts are always 'IP hosts' with an IP address. A 'host name' is assigned using DHCP and is especially relevant for CORS (Cross-origin resource sharing). The browser will set an origin header on the request and the server will send a 'Access-Control-Allow-Origin' header back. Based on the latter the browser can decide to process the response or not. 'Host' and 'hostname' tend to get confused with each other. Sometimes people mean 'hostname' when they say/write 'host'.

• TLS, SSL and HTTPS: TLS means 'transport layer security' and can be regarded as an extra layer in the protocol stack providing encryption. SSL 'secure socket layer' is the name of an older and deprecated version of TLS. When adding TLS to HTTP the resulting protocol stack is called 'HTTPS' (HTTP secure). TLS will work for any protocol based on TCP/IP, not only text based protocols. TLS includes a handshake phase in the communication in which asymmetric encryption is used (public and private keys) to establish a shared secret (shared key). The handshake phase requires that the server provide a digital certificate (proving its authenticity) as well as its public key. ABB apps use a proxy server packaged in a microservice to handle HTTPS. This server uses letsencrypt to get certificates. The proxy server forwards the request without encryption to the ABB microservice based app. You'll find that ABB servers have a least two docker compose networks: one for the ABB app and one for letsencrypt. These networks need to be linked together so the proxy can forward to the ABB endpoint.

• Programming language: General term for human readable language defining computer instructions. Programming languages can be either imperative (text defines HOW the computer should operate to get a result) or declarative (text defines WHAT computer should output). Functional programming languages are declarative and they allow the user to write programs using pure functions without describing the process flow. Elixir is a functional language for example and Javascript is multi-paradigm allowing for both imperative and 'functional' code.

• Compiler: Program that converts programming language into instructions for a processor. This processor may be the actual processor (e.g. an Intel Xeon server processor) or a virtual one. The Java virtual machine or BEAM (Erlang virtual machine) are examples of virtual machines (not to be confused with an OS level VM). ABB uses Elixir in some microservices and this needs to be compiled to bytecode for the BEAM engine. A docker container and docker file takes care of all this for us. One of the most famous compiled languages are C and C++ but ABB does not use these languages.

• Interpreter: Program that reads programming language as a text string and executes instructions directly on the processor. Interpreted programs tend to run slower because of this but are more flexible. Famous interpreted languages are Python and Ruby. Javascript is also an interpreted language but that one's a little bit special. Some ABB services use Ruby in which case the ruby interpreter is provided by the container image.

• Javascript (short. 'JS'): This programming language is one of the core technologies of the web and shipped in text using HTTP alongside HTML. The browser executes Javascript code using a 'javascript engine' which is an interpreter and a JIT (just in time) compiler. Because of the JIT technology some sets of Javascript instructions can be run almost as fast as 'native' processor instructions. The most famous JS engines are Google's V8 and Firefox's Spidermonkey. NodeJS is a program that packages a javascript engine (V8) as well as the NodeJS runtime and OS interface. Thanks to NodeJS Javascript can run on a server and interact with the OS (e.g. writing files). ABB's front end web apps ship with lots of Javascript which get executed in the browser. On the back-end javascript is used for many microservices and runs in containers including NodeJS.

• Runtime: (Simplified) The part of a program responsible for execution. Most programming languages have a specific runtime. ABB's front end framework, Ember, can be said to have a higher order runtime which uses Javascript.

• API (full. 'Application programming interface'): A catalog of functions to call and execute. API's exist on many levels. Some API's consist of the functions exposed by a software library (e.g. Nvidia CUDA is an API which allows programs to interact with an Nvidia GPU) and others consist of HTTP endpoints (a 'web API' in this case). In most cases technical writers mean 'web API' when they use the term 'API'. Of some of ABB's microservices it can be said they expose a 'web API' and many utilize web API's from other (public) services such as the 'basisregisters' API provided by the Flemish government.

• REST API: The term 'REST API' (REST means 'representational state transfer') is used for a web API that does not require the server to keep the 'state' of the connection in memory. All the data the server needs to process any request should be in the request itself. Basisregisters is a REST API but most of the ABB API's associated with an ABB app are not because they require the 'server' (microservice system) to maintain a 'session'. This does not mean that REST API's can never require a login. It just means that the client will have to store the login credentials associated with the connection and send it along with each request.

• Cookie:

• Session: Data structure describing the state and relevant properties of of an interaction between server and client. Sessions are stored in the server. In ABB apps a data structure describing the session is stored in the database and is identified by an ID string. After a successful login the ABB backend sends back a session ID. In subsequent requests the session ID is attached to the requests as a HTTP header (the infamous MU-SESSION-ID). Because of this the server knows which user has made the request so it can managed access control (it will not send data to users who are not allowed to see it). In some communication systems the session is used to store the progression (state) of the information exchanged. For ABB this is less relevant as the session is mostly used to store which user is logged in.

• URL (full 'Universal Resource Locator'): A text string encoded in ASCII pointing to a resource on the web (something to connect with using HTTP). Because ASCII is a small symbol set a higher order encoding is used called 'percent encoding' so one can encode control characters and non-ASCII characters (e.g. Japanese). An 'IRI' (international resource identifier) is a set of specifications that allow for URL's with unicode characters. It also applies to URI's which can be confusing. IRI often confused with URL. Common URL's consist of these parts: <protocol>://<hostname><path>?<query>#<fragment>. (More complex URL's are possible). This means that the 'hostname' does not end with a slash and may be a registered name or an IP address. The path begins with a slash. For example: https://lokaalbeslist.vlaanderen.be/agendapunten has lokaalbeslist.vlaanderen.be as the hostname and /agendapunten as the path. Don't forget the slash and don't confuse the URL paths with a file path in your OS. They are different things entirely.

Database and linked data related knowledge:

• Database: Specialized computer program to store and retrieve data using the computer's memory and persistent storage. Relational databases store data in table and can be queried using SQL. Graph databases store data in graphs containing nodes and links. Linked data databases are graph databases which operate using linked data and can be queried using SPARQL. ABB uses Openlink virtuoso as its database of choice. It is deployed with ABB apps (which use the mu-semtech framework) in container form.

• Linked data: According to W3C: The term Linked Data refers to a set of best practices for publishing structured data on the Web. The principles are: Use URI's to name things, provide useful data when someone request information concerning a specific URI and include links to other URI's.

• Semantic web: An extension of the WWW through standards. The goal of this extension is to make (part of) the internet machine readable (linked data) as opposed to only human readable (HTML).

• RDF (full. 'Resource Description Framework'): RDF is a data structure describing graphs consisting of nodes and edges/links. RDF models graphs as a series of triples (cf. triple below) which refer to sets of two or three symbols referenced by an unique string or URI. RDF is expressed in many formats; most of which are text based. Examples of ways to express RDF in a file (serialized) or over the network is the turtle format, n-triples, n-quads JSON-LD or RDF/XML. Using RDF hypothetically all human knowledge can be expressed and many other knowledge systems are built on top of RDF such as RDFS and OWL. Virtuoso is capable of reading files in all common serialization formats and inserting them in the database. The semantic web consists of servers serving knowledge graphs expressed in RDF. RDF graphs may contain links to other graphs and in this case the data can be considered 'linked'.

• URI (full. 'Universal Resource Identifier'): Not to be confused with URL. URL's point to web pages and are ambiguous (two URL's can point to the same page). URI's only point to one and only one unique concept. An URI is always encoded in ASCII. IRI is a set of specifications to allow writing of URI's with unicode characters. Be cause this also applies to URL this can be confusing. But again. An URI is NOT an URL and does not have the same structure. URI's can be any string as long as they are guaranteed to be unique. Because internet domain names are registered and guaranteed to be unique an URL like structure if often used in URI's; hence the confusion. Dog is a valid URI but it's not very likely to be unique. http://my-unique-domain-that-i-paid-for.com/ontology#Dog is. It's also common, but not mandatory to make URI's valid URL's so they can be put in the address bar of the browser to get more human readable information concerning the specific concept.

• Triple: Smallest and atomic part of a graph structure expressed as linked data. A triple is expressed in RDF and can be regarded as a sentence consisting of subject, predicate and object. These are three parts; hence the name. In common serialization formats triples are expressed as <subject-uri> <predicate-uri> <literal|object-uri>. (Note the point). Subjects and predicates are always symbols referenced by a URI. The object can either be a symbol or a literal data value.

• Triple store: A database or other storage system capable of storing or serving triples. Virtuoso, the database ABB apps use, is a graph database and a triple store.

• Graph: This concept can be confusing because it can mean different things depending on the context. A graph is a mathematical construct consisting of nodes and edges. It's also, in this context, a collection of triples which form a graph like structure. Tables with records can be used to store data which models things in the real world and graphs as well. But graphs have a much higher capacity to model more complex things. Triple stores, like virtuoso, can store many graphs as a high level data structure not too dissimilar to a table in a relational database. In ABB apps we've got many different graphs and most queries specify in which graph triples should be searched for and selected. When no graph is mentioned in a query the database engine assumes you are lookin in the 'default graph'. Each graph is also identified by an URI.

• SQL (full. 'structured query language'): Declarative language which can be used to express commands to retrieve, store, modify or delete data from a relational database. It can also be used to manage the database itself; i.e. create tables etc.

• SPARQL (full. 'SPARQL Protocol and RDF Query Language'): Declarative language which can be used to express commands to retrieve, store, modify or delete data from an RDF based graph database. It can also be used to manage the database itself; i.e. create graphs etc.

• N-triples and N-quads: N-triples is a text based file serialization format that can be used to store triples as tekst in a file. Each new line in an n-triples file contains a triple expressed as <subject-uri> <predicate-uri> <literal|object-uri>. A common file extension for these files is .nt. N-quads (file extension .nq) is exactly the same except there are 4 elements in each new line: the first being the URI of the graph the triple belongs to and the last three making up the triple itself. Because of this an n-triple file can only be ingested by virtuoso if you specify which graph the triples should be put in. If you don't virtuoso will insert them into the default graph. N-quads file contain graph information so virtuosos can process these without graph related parameterization. Both n-triples and n-quads can serve as a format to be put on the wire and sent using HTTP as it's a text based standard. It's important to point out that this format is highly verbose and inefficient and can benefit very, very greatly from compression ('zipping').