SecHub Architecture

SCHEDULE

SCHEDULE_SECHUBJOB

Represents a SecHub Job instance

SCAN

SCAN_RESULT

Contains the collected scan result from all scan product. Shall represent entity ScanResult

SCAN

SCAN_DATA

Does contain (intermediate) information from job scans etc.

…​

…​

…​

Server development shall be easy to maintain and developers shall have experience with the language and frameworks

All

Spring Boot, Java

Users shall have a convenient and simple way to start scans

All

We provide not only a REST API but provide a convenient CLI client which just needs a JSON config file

Client shall be small, fast and usable on many platforms without much effort to install

All

Go client

Developers shall be able to test and develop server parts without an existing infrastructure (means no security products installed etc)

Development

Mock profile and also Mocked Adapters for every supported product

Developers shall be able to write adapters without starting server

Development

Adapters are "dumb" and can be started as simple java application.

For corporate continous-integration scans we need a scalable solution which can be clustered

Cluster installation

With SpringBoot and DDD we ensure we can run server components inside Kubernetes

People or smaller companies shall be able to install SecHub without Kubernetes

Single installation

Server software will be always startable as a simple jar as well (done by Spring Boot Jar)

Developers shall be able to start SecHub without Kubernetes

Development

Server software will be always startable as a simple jar as well (done by Spring Boot Jar)

SecHub sserver integration tests shall be possible without available products, PostgreSQL or Kubernetes

Development

Server software will be always startable as a simple jar as well (done by Spring Boot Jar), we provide also different profiles for normal database (postgres) and in memory DB (h2), we provide a profile for mocked products

We need a reliable database which is mature, stable and good documented

Cluster installation, Single installation, Development

We will provide postgres

We must avoid "big ball of mud" in development and also be able to separate server parts to smaller deployments easily in future (when needed)

Development

Domain driven design is choosen to support this. On the other side we do not want to have a large event bus system (e.g. KAFKA), because not really necessary at the beginning (and maybe never).

So as a KISS approach we provide a self contained system but in conjunction with a very simple DDD implementation:
We create and use a wrapper API for events, start with implementation by simple Observer pattern inside same JVM, so can be replaced later by something else (e.g. a KAFKA implementation). Direct communication between different domains is forbidden, only events are allowed. This is tested by automated tests.

see DomainMessageService.java

We want to have many integration tests which are easy to read, maintain, make fun to write, have reduced test code and using server nearly as done in production.

It shall be possible to execute the integration tests to mocked or even to real products. Changes in server structure or even in code, shall does not enforce changes to tests, but only to test framework implementation.

Development

We implement a specialized integration-test framework which provides a DSL like, fluent API which is able to test again a running test server instance. We implement spring profiles for real-products and mocked-products and a integration-test profile as well on server side.

We want to have an agile, living documentation, which is matching to source code and a diff can be easily done by GIT

Development

We use AsciiDoc and checkin documenteation together inside one GIT repository

see https://asciidoctor.org/docs/asciidoc-syntax-quick-reference/

We want to have a way to provide special scan configurations in a dynamic way

Deployment

We provide a JSON based scan configuration where different provider meta information can be defined. Dedicated providers can use this information and handle dynamic configuration - e.g. based on name of SecHub project.

see https://github.com/mercedes-benz/sechub/issues/127 ,ScanMappingConfigurationService class or description inside technical documentation.

1

Rest API call

2

Rest api called to register user. Normally done by user itself

2

Persistence

3

Valid self registration input will be persisted to database.

3

Email to user

4

A notification is send per email to user that a new user signup has been created and waits for acceptance.

4

Email to admin

A notification is send per email to admins that a new user signup has been created and waits for acceptance.

1

Rest call

SUPERADMIN

All self registrations are returned as json

1

Rest call

SUPERADMIN

2

Administrator accepts a persisted self registration entry by calling rest api

2

Create user and send events

SUPERADMIN

3, 4

The service will create the user a one time token for api token generation and triggers asynchronous events. It will also remove the existing user signup because no longer necessary.

3

Email to user

A notification is send per email to user that a new api token was requested. The mail contains a link for getting the secure API token

4

Give user access

Authorization layer is informed about new user and gives access to sechub. But without any project information

1

Rest call

SUPERADMIN

2

All userids of sechub users are returned as json

2

Service call

SUPERADMIN

All userids of sechub users are returned as json

1

Authenticated REST call

SUPERADMIN, USER

2

2

Persistence and result

Persist a new job entry and return Job UUID

1

Authenticated REST call

SUPERADMIN, USER

2

2

Try to find project and upload sourcecode as zipfile

USER

When project is found and user has access and job is initializing the sourcecode file will be uploaded

1

Authenticated REST call

SUPERADMIN, USER

2

2

Try to find job annd update execution state

When job is found and user has access job will be marked as ready for execution

1

Scheduling

2

Fetches next schedule job from queue and trigger execution.

2

Execution

3, 4

Triggers job execution - done parallel, but with synchronous domain communication (to wait for result).

4

Store admin job info

5

Fetches event about started job and store info in admin domain.

5

Update admin job info

Deletes store info in admin domain when job is done.

1

Authenticated REST call

SUPERADMIN, USER

2

2

Try to find project and fail or return job status

1

REST API call to get JSON report

SUPERADMIN, USER

3

2

REST API call to get HTML report

SUPERADMIN, USER

3

3

Resolve scan report result

1

create new job

SUPERADMIN, USER

2, 3

2

upload sourcecode

SUPERADMIN, USER

2

2

upload binaries

SUPERADMIN, USER

3

3

approve job

SUPERADMIN, USER

4

4

download job report and traffic light

SUPERADMIN, USER

4

4

get job status

SUPERADMIN, USER

1

Rest call

2

User opens url by browser

2

Validation and update

3, 4

When its a valid one time token a new api token is generated and persisted hashed to user. The token itself is returned. When not valid an emtpy string is the result …​

3

Update auth data

4

Inform user about api token change done

1

Rest call

SUPERADMIN

2

Administrator creates a new project by calling rest api

2

Create project

SUPERADMIN

The service will create the project when not already existing with such name.

1

Rest call

SUPERADMIN

All project ids of sechub are returned as json

1

Rest call

SUPERADMIN

2

Administrator does call rest API to assign user

2

Assign user

SUPERADMIN

2

The service will add the user to the project. If user does not have ROLE_USER it will obtain it

2

Update schedule authorization parts

3

3

Update scan authorization parts

4

4

Roles changed in auth

Authorization layer adds ROLE_USER

1

Rest call

SUPERADMIN

2

Administrator does call rest API to unassign user

2

Unassign user

SUPERADMIN

2

The service will remove the user to the project. If users has no longer access to projects ROLE_USER will be removed

2

Update authorization parts

2

2

Update authorization parts

4

4

Roles changed in auth

Authorization layer removes ROLE_USER

1

Rest call

SUPERADMIN

2

Json returned containing details about user and her/his projects

2

Service fetches user details.

SUPERADMIN

The service will fetch user details for given user id

1

Rest call

SUPERADMIN

2

User will be deleted

2

Service deletes user.

SUPERADMIN

3, 4

The service will delete the user with dependencies and triggers asynchronous events

3

revoke user from schedule access

3

3

revoke user from schedule access

4

4

Delete user access

Authorization layer is informed about user deltete and removes access to sechub. But without any project information

5

Inform user that the account has been deleted by administrator

1

Rest API call

SUPERADMIN

2

Rest api called to remove user signup

2

Persistence

SUPERADMIN

Existing signup will be deleted

1

Rest call

SUPERADMIN

2

Project will be deleted

2

Service deletes projects.

SUPERADMIN

3, 4, 5, 6, 7

The service will delete the project with dependencies and triggers asynchronous events

3

Inform sechub admins that project has been deleted

4

4

Inform project owner that the project has been deleted

5

5

Inform users that the project has been deleted

6

6

Update authorization parts - remove entries for deleted project

7

7

revoke any scan access from project

8

8

delete all project scan data

1

Rest call

SUPERADMIN

2

Json returned containing details about project

2

Service fetches project details.

SUPERADMIN

The service will fetch project details

1

Rest call

SUPERADMIN

2

White list will be updated

2

Update project

SUPERADMIN

The service will update the Project whitelist.

1

Rest call

SUPERADMIN

2

Administrator lists all running jobs by calling rest api

2

Fetchjob information from database

Fetches stored job information from administration database.

1

Rest API call

Rest api called to request new user api token. Normally done by user itself

1

REST API call to get JSON list

SUPERADMIN

2

1

REST API call to zip file containing full scan data

SUPERADMIN

2

2

Collect all scan data

SUPERADMIN

1

Rest call

SUPERADMIN

2

User will be granted admin rights

2

Service grants user admin rights.

SUPERADMIN

3, 4

The service will grant user admin rights and triggers asynchronous events

3

Inform user that he/she became administrator

4

4

Inform SecHub admins that another user became administrator

1

Rest call

SUPERADMIN

2

Admin rights will be revoked from admin

2

Service revokes user admin rights.

SUPERADMIN

3, 4

The service will revoke user admin righs and triggers asynchronous events

3

Inform user about loosing administrator rights

4

4

Inform SecHub admins that another admin is no longer admin

1

Rest call

SUPERADMIN

2

All userids of sechub administrators are returned as json

2

Service call

SUPERADMIN

All userids of sechub administrators are returned as json

1

Rest call

SUPERADMIN

2

Administrator wants to stop (pause) scheduler job processing

2

Service call

SUPERADMIN

2

Sends request to scheduler to send updates about current status.

2

Service call

SUPERADMIN

3

Sends request to scheduler domain to disable scheduler job processing

3

Disable processing

4

Disables job processing inside scheduler database

4

Inform SecHub admins that scheduler job processing has been disabled

1

Rest call

SUPERADMIN

2

Administrator wants to start (unpause) scheduler job processing

2

Service call

SUPERADMIN

3

Sends request to scheduler domain to enable scheduler job processing

3

Enable processing

4

Enables job processing inside scheduler database

4

Inform SecHub admins that scheduler job processing has been enabled

1

Rest call

SUPERADMIN

Administrator wants to trigger a refresh of scheduler status. Will update information about running, waiting and all jobs in scheduler etc. etc.

1

Rest call

SUPERADMIN

Administrator wants to list status information about sechub

1

Rest call

SUPERADMIN

2

Triggers job cancellation request, owners of project will be informed

2

Cancel job

3

Will trigger event that job cancel requested

3

Try to find job and mark as being canceled

4

When job is found and user has access the state will be updated and marked as canceled

4

Inform user that her/his job has been canceled

1

REST API call to define mock configuration byJSON data

SUPERADMIN, USER

2

2

Service call to store mock configuration

1

REST API call to get JSON data

SUPERADMIN, USER

2

2

Service call to get JSON data

1

Rest call

SUPERADMIN

2

Administrator wants to update a mapping configuration

2

Service call

3

Services updates data in database and sends event

3

Event handler

4

Receives mapping configuration change event

4

Service call

5

Updates scan mapping in DB

5

Trigger service

6

Checks periodically for updates in scan configuration

6

Service call

Checks if current mappings in DB lead to a new scan configuration.

1

Rest call

SUPERADMIN

2

Administrator wants to fetch a mapping configuration

2

Service call

Services fetches data from database, if not set an empty mapping data result will be returned

1

REST API call

An anonymous user checks if the server is alive and running using the REST API

1

REST API Call

SUPERADMIN

Administrator wants to fetch server runtime data. This data contains for example the server version

1

Rest call

SUPERADMIN

2

Triggeres job restart (soft)

2

Restart job

3

Will trigger event that job restart (soft) requested

3

Inform sechub admins when job restart was canceled

4

4

Inform sechub admins when job has been restarted

5

5

Inform sechub admins when job results have been purged

1

Rest call

SUPERADMIN

2

Triggeres job restart (hard)

2

Restart job

3

Will trigger event that job restart (hard) requested

3

Inform sechub admins when job restart was canceled

3

3

Try to restart job

3

When job is found and job is not already finsihed, a restart will be triggered. Existing batch jobs will be terminated

3

Try to rstart job (hard)

4

When job is found, a restart will be triggered. Existing batch jobs will be terminated

4

Inform sechub admins when job has been restarted

5

5

Inform sechub admins when job results have been purged

1

send domain message that new scheduler instance has been started and information about potential zombie jobs

2

2

Inform sechub admins that new scheduler job has been started

1

REST API call to define false positives by JSON data containing identifiers for existing jobs or false positive project data

SUPERADMIN, USER

1

REST API call to remove existing false positive definition

SUPERADMIN, USER

1

REST API call to fetch existing false positive configuration of project

SUPERADMIN, USER

1

Rest call

SUPERADMIN

2

Administrator adds a new product executor configuration by calling REST API

2

Service call

SUPERADMIN

Service creates a new product executor configuration

1

Rest call

SUPERADMIN

2

Administrator deletes an existing product executor configuration by calling REST API

2

Service call

SUPERADMIN

Service deletes an existing product executor configuration by its UUID

1

Rest call

SUPERADMIN

2

Administrator fetches list of existing product executor configurations by calling REST API, will not contain setup information

2

Service call

SUPERADMIN

Service fetches data and creates a list containing all executor configurations

1

Rest call

SUPERADMIN

2

Administrator fetches setup of an existing product executor configuration by calling REST API

2

Service call

SUPERADMIN

Service reads setup information for an existing product executor configuration

1

Rest call

SUPERADMIN

2

Administrator updates setup for an existing product executor configuration by calling REST API

2

Service call

SUPERADMIN

2

Service updates existing executor configuration

2

Service call

SUPERADMIN

Service updates existing executor configuration

1

Rest call

SUPERADMIN

2

Administrator adds a new product execution profile by calling REST API

2

Service call

SUPERADMIN

Service creates a new product executor configuration

1

Rest call

SUPERADMIN

2

Administrator deletes an existing product execution profile by calling REST API

2

Service call

SUPERADMIN

Service deletes an existing product execution profile by its profile id

1

Rest call

SUPERADMIN

Administrator updates existing profile by calling REST API

1

Rest call

SUPERADMIN

2

Administrator fetches setup of an existing product executor configuration by calling REST API

2

Service call

SUPERADMIN

Service reads setup information for an existing product executor configuration

1

Rest call

SUPERADMIN

2

Administrator fetches lsit of all available execution profiles by calling REST API

2

Service call

SUPERADMIN

Service fetches data and creates a list containing all executor profiles

1

Rest call

SUPERADMIN

2

Administrator adds profile relation to project by calling REST API

2

Service call

SUPERADMIN

Services creates a new association between project id and profile

1

Rest call

SUPERADMIN

2

Administrator removes profile relation to project by calling REST API

2

Service call

SUPERADMIN

Services deletes an existing association between project id and profile

1

Rest call

SUPERADMIN

2

MetaData will be updated

2

Update project

SUPERADMIN

The service will update the Project metadata.

1

Rest call

SUPERADMIN

2

Administrator does call rest API to set new project owner

2

Change project owner

SUPERADMIN

4

The service will set the user as the owner of the project. If user does not have ROLE_OWNER it will obtain it. The old owner will loose project ownership.

4

Inform new and previous project owners that the project owner ship has changed

1

Rest call

SUPERADMIN

2

Changes project description. Json returned containing details about changed project

2

Service changes project description.

SUPERADMIN

The service will change project description.

1

Rest call

SUPERADMIN

2

User emaill address will be changed

2

Service updates user email address.

SUPERADMIN

3

The service will update the user email address and also trigger events.

3

Inform user that the email address has been changed

1

Rest call

SUPERADMIN

2

Administrator fetches auto cleanup configuration

2

Fetches auto cleanup config

Fetches auto cleanup configuration from database

1

Rest call

SUPERADMIN

2

Administrator changes auto cleanup configuration

2

Updates auto cleanup config

3, 4, 5

Updates auto cleanup configuration as JSON in database and sends event

3

Administration domain receives auto cleanup event

4

Received event in administration domain about auto cleanup configuration change. Stores data, so available for next auto clean execution

4

Schedule domain receives auto cleanup event

5

Received event in schedule domain about auto cleanup configuration change. Stores data, so available for next auto clean execution

5

Scan domain receives auto cleanup event

Received event in scan domain about auto cleanup configuration change. Stores data, so available for next auto clean execution

1

Scheduling

2

Checks for parts to auto clean.

2

Delete old data

deletes old job information

1

Scheduling

2

Checks for parts to auto clean.

2

Delete old data

deletes old job information

1

Scheduling

2

Checks for parts to auto clean.

2

Delete old data

3

deletes old job information

3

Schedule cipher pool data cleanup

Removes cipher pool data entries from database which are no longer used by any job

1

Authenticated REST call

SUPERADMIN, USER

2

2

Try to find project and upload binaries as tar

USER

When project is found and user has access and job is initializing the binaries file will be uploaded

1

REST API call to get SPDX JSON report

SUPERADMIN, USER

1

get pageable list of jobs in project

SUPERADMIN, USER

2

2

Assert access by service and fetch job information for user

1

Rest call

SUPERADMIN

2

Json returned containing details about user and her/his projects

2

Service fetches user details.

SUPERADMIN

The service will fetch user details for given user email address

1

Rest call

SUPERADMIN

2

Admin triggers rotation of encryption via REST

2

Service call

3

Triggers rotation of encryption via domain message

3

Service call

4

Forces new cipher pool entry creation and triggers encryption service pool refresh

4

Service call

5

Creates new cipher pool entry in database in own transaction

5

Refresh encryption pool

6

Encryption pool is refreshed (necessary because pool changed before this method call)

6

Update encrypted data

Encrypted data is updated (a direct pool refresh was triggered by admin action)

1

Init encryption pool

3

Encryption pool is created on startup

1

Encryption pool data refresh trigger

2

Scheduler instance will check if encryption pool is in sync with the database definitions. If not, the instance will try to create new encryption pool object and provide the new setup.

2

Refresh encryption pool

3

Encryption pool is refreshed (if necessary)

3

Update encrypted data

Encrypted data is updated (all other cluster members)

1

Update encrypted data

Final update of encrypted job data. Will update all SecHub jobs having a pool id which is lower than latest from encryption pool

1

Rest call

SUPERADMIN

1

Admin fetches encryption status from domains via REST

1

Service call

Services collects encryption status from domains via event bus

1

Schedule cipher pool data cleanup

Removes cipher pool data entries from database which are no longer used by any job

1

REST API call to remove existing false positive project data definition by id

SUPERADMIN, USER

1

Scheduler terminates

2

Scheduler instance is terminating. Will stop processing new jobs and inform job executor to suspend

2

Scheduler job executor suspends current jobs

3

Scheduler instance is terminating. Will mark current running jobs of this instance as SUSPENDED

3

Scheduler job executor suspends current jobs

4

Scheduler instance is terminating. Will mark current running jobs of this instance as SUSPENDED

4

Scan job executor stops suspended jobs

5

Scheduler instance has marked jobs as suspended. Will stop execution of scans of these jobs

5

Scan job executable handles SUSPEND state

6

Scan job executable stops execution because suspended

6

Inform listeners

7

Inform listeners about job suspension

7

Administration handles suspended job

Administration domain removes suspended job from its running job list

1

Schedule suspended jobs

2

Scheduler checks not only for new jobs but also for resumed ones.

2

Resolve next job

3

Resolves UUID of job which shall be executed at next time. If there is a suspended job and shall be resumed this job will be returned. Otherwise the selected schedule strategy will be used to determine next job uuid.

Remark: A suspended job shall only be executed when the minium duration time has been reached. The time period can be configured and prevents side effects at deployments.

3

Mark next job to execute

4

When a suspended job is the next one, the job execution state will be changed to RESUMING

4

Resume job

The SecHub job will be resumed. This is done by triggering a soft restart request for the job.

You can read Admin restarts a job for the steps which are done at restart process. The steps are the same, except there is no audit logging and the event is not triggered from administration but schedule domain.

Path

/api/admin/users

Method

GET

Status code

200 OK

[]

Array

List of user Ids

Path

/api/admin/project/{projectId}/membership/{userId}

Method

POST

Status code

200 OK

projectId

The id for project

userId

The user id of the user to assign to project

Path

/api/admin/project/{projectId}/membership/{userId}

Method

DELETE

Status code

200 OK

projectId

The id for project

userId

The user id of the user to unassign from project

Path

/api/admin/user/{userId}

Method

GET

Status code

200 OK

userId

The user id of user to show details for

userId

String

The name of the user

email

String

The email address of the user

superAdmin

Boolean

True, when this user is a super administrator

projects

Array

The projects the user has access to

ownedProjects

Array

The projects the user is owner of

Path

/api/admin/user/{userId}

Method

DELETE

Status code

200 OK

userId

The userId of the user who shall be deleted

Path

/api/admin/project/{projectId}

Method

GET

Status code

200 OK

projectId

The id for project to show details for

projectId

String

The name of the project

users

Array

A list of all users having access to the project

owner

String

Username of the owner of this project. An owner is the person in charge.

whiteList

Array

A list of all whitelisted URIs. Only these ones can be scanned for the project!

metaData

Object

An JSON object containing metadata key-value pairs defined for this project.

metaData.key1

String

An arbitrary metadata key

accessLevel

String

The project access level

description

String

The project description.

Path

/api/admin/scan/download/{jobUUID}

Method

GET

Status code

200 OK

jobUUID

The job UUID

Path

/api/admin/user/{userId}/grant/superadmin

Method

POST

Status code

200 OK

userId

The userId of the user who becomes admin

Path

/api/admin/user/{userId}/revoke/superadmin

Method

POST

Status code

200 OK

userId

The userId of the user who becomes admin

Path

/api/admin/admins

Method

GET

Status code

200 OK

[]