Tehnical Architecture
Tehnical Architecture
Summary
- Top Level Overview
- Solution Technical Goals
- Implementation details
- OpenAPI Documentation
- Options for Modification
1. Top Level Overview
Gradient Image Processing is built as a single dedicated service that is applied for image processing. The solution supports multiple services that perform the upload of original images and retrieve processed result images. The result images are retrieved from the Gradient Image Processing service. The intended application of this version (and current implementation) is web traffic optimization on web pages, where images are compressed and scaled to the appropriate size.
Top Level overview of Gradient Image Processing is presented in the following illustration:
Illustration is explained in more details in the following sections.
1.1. Consumer Services
Under consumer services are considered services, software solutions, and technical implementations that utilize Gradient Image Processing. Every service that needs processing (and retrieval) of images is able to communicate that with Gradient Image Processing.
For example, some "CMS Service" might be using the Public API Layer to perform an upload of an image and get designated result images. Later on, it can download the resulting image when needed. After upload consumer service is informed about result images and necessary meta-information about them.
With all this in mind, every work with images that consumer service needs, should be performed inside the Gradient Image Processing Service.
1.2. Authorization Layer and Public API
For consumer service to manage the images, authorization is needed. The authorization layer recognizes three types of actors in the system: Admin, Client, and ClientApp. Consumer services are client applications (ClientApp).
Admins and Clients can use Dashboard (built-in GUI interface), which provides standard CRUD operations over Admins, Clients, and ClientApps. Every inserted Client Application gets automatically generated AppID and SecretID.
Client Apps, with credentials, are able to upload original images that need processing and get information about result images for later usage. Additionally, Client Apps can request deletion of the original image and all its resulting images.
When the original image is uploaded, processing starts. The image is ready for processing, and the result images are created. Gradient Image Processing opens one WebSocket server through which Admins, Clients, and Client Apps can track (listen to) the progress of processing.
1.3. Internal API and Storage
Internal API currently supports Image Configurations management, scaling, and compression of images, with separation between original and runtime images. Runtime image is the resulting image of compression.
The design supports multiple data storage for Client Application keys and Image information and configurations. Original and runtime images have different characteristics, so Internal API is designed with the possibility of having different storage types.
2. Solution Technical Goals
Based on the top-level picture, expected actors in the system and operations needed to be performed by Gradient Image Processing, a detailed feature set is defined. That feature set is visually presented in the following illustration:
After defining the feature set, for achieving the goal of extendable, scalable, and maintainable image processing service, the following technical goals are defined:
- There is clear separation between public and internal layer (API).
- Data integrity of the informational layer of the system has to be preserved and protected.
- Client Application keys must be "extractable" out of the main system for isolated role-based access control.
- System has a clear Authorization layer (which resources are public and which are not).
- Single API endpoint to upload original images. Decoupled from the rest of the system.
- Single module for scheduling and managing original and result images processing. Decoupled from actual image processor.
- Image processing (compressing, scaling, deleting, and reading images) is decoupled from the informational layer.
- Image processing can be parallelized for higher load handling.
- File Storage operations (saving and loading) are decoupled from the rest of the system (so sources and destinations can be easily changed).
- File Storage initially assumes two different locations for Archive and Runtime images (originals and results).
- Image Processing can be followed up in real-time by applying WebSocket technology.
- Robustness of the system. Retry algorithm is included in the case that image processing does not succeed for any reason.
By considering enlisted technical goals, final solution architecture is built and implementation is performed.
3. Implementation Details
Decision has been made that the solution utilizes Typescript as a programming language. Considering the extremely modular approach that is performed, platforms and frameworks that are explicit about modules are picked. The criterion for the framework choice is defined by the amount of provided module abstractions in it. The selected framework must support a module description, which includes its internal services/components, its exported components, and its dependency modules. Frameworks that satisfy this criterion are Nest.js for the backend and Angular for the frontend implementation.
Related to the persistence layer, a relational database is picked and provided tooling about data integrity is fully applied. For Gradient Image Processing, PostgreSQL is utilized.
When it comes to processing that needs to be executed, one can recognize two layers: "Data Processing" and "Image Processing". These layers can be decoupled.
"Data Processing" includes operations related to managing a database, its state, and the information it stores. This layer is making sure that information about everything that Gradient Image Processing is about can be retrieved and is always valid. It handles information about Users, Client Applications, Configurations, original images information, result images information, and state of individual image processing.
"Image Processing" includes operations related to working with images. Image scaling, compression, crop, storing, and retrieving from adequate storage are parts handled in the "Image Processing" layer. It is imperative to make sure that this layer does not have any operations related to data processing.
Necessary input for the "Image Processing" is provided by the "Data Processing" layer. The "Data Processing" layer updates the database based on outputs received from "Image Processing".
3.1. Entity Relationship Diagram
Detailed information about all entities and their attributes can be checked here.
3.2. Modules Diagram
The modules diagram presents all implemented controllers and services. Isolated layers are:
- Public API (with controllers for http and WebSocket API),
- Data Processing (services that work with Entities),
- Image Processing (services that process images in Worker Queue design).
Some of the components in the diagram are marked with different colors, repeated or even skipped being drawn, for the purpose of easier understanding of the diagram. Hence the following notes:
- Authorization Guard is a single service implementation. Above the controllers is presented usage of it and input parameters provided to it.
- Entities. Usage of applicable Repository (TypeORM) to work with is assumed in it. So, when the service is dependent on Client App Entity, for instance, it means that it applies ClientAppRepository to manage that entity and database.
- CRUD. Means standard Create, Read (with filters), Update, and Delete operations.
- Queue Control Service. This is a service that every other service that works in Worker Queue mode utilizes to communicate with Queue implementation. Its absence from some queue illustrations like Image Compression and Scaling Queue is there just for the purpose of cleaner design. In reality, Image Processor Service and Image Compression And Scaling Service utilize it to communicate.
Detailed information about every single service in this diagram, with its methods, signatures, and purpose, can be checked in technical documentation here.
4. OpenAPI Documentation
As visible from previous chapters, the service communicates using http and WebSocket protocols.
For http API usage, the OpenAPI specification is provided for download here. It can be visually presented in online tools like swagger.
To utilize communication using WebSocket (for real-time part of API) check usage guidelines here.
5. Options for Modification
Gradient Image Processing can be extended based on specific feature requirements. In this chapter, some of the possible extensions are described.
In the following illustration, one can see an approach where Gradient Image Processing implementation is scaled into multiple independent services each with its own purpose and responsibilities.
By this approach, Gradient Image Processing functionality is separated into the following independent modules:
- Http API Interface
- In this module, HTTP controllers are implemented. Authorization, Authentication, Configuration, and Client Settings services are included.
- The module is stateless so running multiple instances on multiple hosts at the same time is supported.
- Option for building and running different APIs at the same time is supported.
- WebSocket Interface
- In this module, ProgressController (WebSocket server) is implemented.
- Module by itself is stateless from the business functionality perspective. Running multiple instances on multiple hosts, hence sharing the load between connected client applications, is supported.
- Image Operations Service
- In this service, the data layer of Gradient Image Processing is implemented.
- Service is stateless.
- Service fully utilizes Database (as persistence) and Event Broadcast Service as a hub to send real-time information about the processing of the image.
- PostgreSQL Database Cluster
- This module represents the persistence layer of the whole system.
- By this design, PostgreSQL cluster is considered stateful, so it is not supported to simply run multiple instances of this module on multiple hosts and point dependent services to these instances.
- Sharding based on Client Application data (Horizontal sharding) is supported since there are no data references between images and configurations that belong to different Client Applications. NOTE: There is connectivity that Client Applications can belong to the same Client so this should be considered in this approach.
- Event Broadcast Service
- Service for managing Event based broadcast communication (realtime).
- Service is considered stateful.
- Redundancy is possible by sharding based on Client Application Data.
- Queue Messaging System
- Service for managing queues and message communication through queues.
- Service is considered stateful in the sense that the same queue instances should be utilized for services that are communicating.
- Redundancy is possible by sharding based on Client Application Data.
- Image File Service
- Loading and saving files. Service that coordinates between adequate storage managers (runtime and archive).
- Service is considered stateless so it can be multiplied.
- Runtime Images Storage Manager
- Manager that reads and saves runtime files.
- Service is considered stateful since it relies on one specific storage place (hard drive in local case).
- Archive Images Storage Manager
- Manager that reads and saves archive files.
- Service is considered stateful since it relies on one specific storage place (hard drive in local case).
- Image Compression And Scaling Service
- Service that takes an original image as input and necessary configurations and performs proper compression and scaling operations.
- Service is considered stateless.
Modules/Services that are considered stateless can safely be instantiated as more than one running instance, and it does not matter when and under which circumstances are these services executed. Since the logic of these services is pure (analogy with pure functions), it can be executed in concurrency without affecting the final result and product stability.
On the other side, modules that are considered stateful can not be simply instantiated in concurrency mode since the state these are holding is interconnected. Additionally, the availability of requested information from the state would depend on which instance that information is requested.