Change Log
All notable changes to this project will be documented in this file.
Version 2.19
Features
Version 2.18.1
Features
- We have added support to Service Bus Topics and Subscriptions.
A new
Subscriptionparameter was added to the queue configuration. WhenSubscriptionis provided,QueueNamebecomes the name of the topic.
Fixes
- We have fixed an issue where intermittent delays were observed when receiving messages from Service Bus queues. Fixing the issue requires configuration changes.
- The
ImportWorker:ReceiveTimeoutparameter controls the timeout of a message receive call, with a default value of 100 ms. Increasing the timeout might help to fix the issue. -
The
ServiceBus:PrefetchCountparameter specifies the number of messages prefetched and cached before the actual message is requested. Default number is 0, meaning no prefetch. Increasing the value ofPrefetchCountmight help to fix the issue. -
We have fixed an issue where the import requests could not be identified in the debug logs.
Version 2.18
Breaking changes
The configuration file of the Trigger Relay service should be used as the base for the configuration of the new Mesh Data Transfer service.
The database configuration from the Database Gateway service needs to be applied to the Mesh Data Transfer settings file. The HttpEndpoints:DatabaseGateway node should be removed from Mesh Data Transfer configuration. The Kestrel configuration from the Database Gateway service should be dropped.
The configuration from Export Data Store settings file needs to be applied to Mesh Data Transfer settings file. Specifically:
- the
Storage, node should be moved to Mesh Data Transfer settings, - if present, the
MercatoMapping,UnitScheduleMappingandTsVolumesWebServiceExportnodes should be moved to Mesh Data Transfer settings, - the export queue should be added to the
QueuesConfigurationsection, - the contents of
ParticipantSettingsnode should be merged (previously the node existed in the configuration files of both services), - the
HttpEndpoints:ExportDataStorenode should be removed from Mesh Data Transfer configuration.
The Kestrel configuration from the Export Data Store service should be dropped.
The configuration from Mesh AMQP Relay settings file needs to be applied to Mesh Data Transfer settings file. Specifically:
ImportWorkerandFailedMessagesnodes should be applied to Mesh Data Transfer settings,QueuesConfigurationin Mesh Data Transfer should be updated with the queues used for the imports (the import, import reply and failure queues),ImportExportCommonSettings:Delayconfiguration parameter in Mesh AMQP Relay configuration is now moved toImportWorker:Delayin Mesh Data Transfer configuration,ExportWorkersettings were removed and should not be applied in Mesh Data Transfer configuration file.
Additionally, the export related queues should be removed from AmqpSender:Queues, as they were used to forward export requests to Mesh AMQP Relay and are not relevant anymore.
Fixes
Version 2.17.3
Breaking changes
Fixes
Version 2.17.2
Fixes
Version 2.16.3
Fixes
Version 2.15.6
Fixes
Version 2.16.2
Fixes
Version 2.15.5
Fixes
Version 2.17.1
Fixes
Version 2.17
Breaking changes
-
When exporting revision availability events, the category attribute will now contain a string "Revision". Previously the category was an empty string for the revision events.
Features
-
We have updated reading empty time series to align with changes in Mesh 2.17 changes.
-
We have added a new
ReadOnlyparameter to the DatabaseGateway service that prevents the service from modifying the database. Note: it does not prevent Mesh from database modifications. The read only mode in Mesh needs to be set separately in the Mesh configuration file.
Fixes
Version 2.16.1
Features
- We have added support for Time Series Volumes Web Service time series exports.
The settings for the new protocol are placed in
TsVolumesWebServiceExportnode in Export Data Store configuration file.
Fixes
Version 2.16
Features
-
We have added full support for the Service Bus queues. If the queues defined in the configuration file do not exist, data-transfer will attempt to create them automatically. The message lock time will be set to 5 minutes, other settings will remain default.
Fixes
Version 2.15.4
Features
- We have added an option to choose between
oraandperiodonodes in Bidding export protocol. A newUsePeriodoflag in the Bidding export protocol section in EDS configuration is used to switch between the nodes.
Fixes
Version 2.15.3
Features
- We have added an option to report
ImportSuccess = truefor partially successful time series import requests. The feature is toggled with an optionalPartialImportSuccessparameter in theImportWorkersection of Mesh AMQP Relay configuration file.
Fixes
Version 2.15.2
Breaking changes
-
We have added support for the Decimals export attribute in GS2 time series export. Since this attribute was previously ignored, GS2 time series export might start to produce different results if the Decimals attribute was already defined in the time series export definition.
-
We have added support for the UTC offset in GS2 time series export. The standard time is used when custom UTC offset is not provided.
Features
- We have added a new optional parameter named DefaultDecimals that can be applied per export protocol in the Export Data Store configuration. The DefaultDecimals parameter is supported for the EDIEL DELFOR and GS2 export types only.
Fixes
Version 2.15.1
Breaking changes
- We have changed the behaviour of the time series export when different export senders are used. The export results are now grouped per sender. If no sender is defined as the export request parameter, then the time series sender is either the sender host from the export definition or the default sender when the sender host is not defined. If the sender is defined as the export request parameter, the time series export definitions with a defined sender host that doesn't match the request sender are ignored during the export; the sender of all the time series is the request sender.
Fixes
Version 2.15
Breaking changes
- We have added a new mandatory configuration parameter for TriggerRelay:
ParticipantSettings:DefaultSender. It was done as a part of #283 and #316. TheDefaultSenderparameter is a number that denotes a participant key to be used by default when exporting data from Mesh. Example configuration:
"ParticipantSettings": {
"DefaultSender": 13
}
Features
-
We have added support for Bidding Strategy export with a 15-minute time series resolution.
-
We have added support for changing the resolution of the exported time series. NB! The behavior of an export with a target resolution differs from the old export in release 2.15. When exporting time series points from an attribute that holds a physical time series, the
Idfield in the Standard ExportPointsnode will contain a time series source ID instead of a physical time series ID. If the target export resolution is not defined, theIdwill still contain the physical time series ID.
<Timeseries QueryId="2ad565bc-412c-4674-b981-b1ad24537799">
<Points Id="d3163dae-81de-413a-8c55-bc11ad5080c8" Path="XYZ" DeltaT="PT15M" Reference="ABC" UnitOfMeasurement="MW" UnitOfMeasurementName="MW" CurveType="StaircaseStartOfStep">
<Segments>
...
</Segment>
We plan to place the time series source ID in the Standard Export's Id field unconditionally starting from the release 2.16 (Smart Power 2024.5).
Fixes
-
We have changed the type of the attribute
UnitOfMeasurementinTimeseriesPointstype fromUUIDtoxsd:string. In consequence, theAmqpMessageTypes.xsdschema version has been bumped to 1.1. To start using version 1.1, please setStorage:StoragePerProtocol:StdExport:SchemaVersionitem to "1.1" in Export Data Storeappsettings.json. If this value is not specified, it defaults to version 1.0. Note! We will stop supporting schema version 1.0 in Mesh Data Transfer release 2.17. Note! When using schema version 1.0, theUnitOfMeasurementattribute value does not comply with its type defined in the schema. -
We have fixed an issue where the Mesh export response message size would exceed the default 4 MB
-
We have fixed an issue where Message Log timestamps would not be aligned to the database time zone.
Version 2.14.2
Version 2.14.1
Version 2.14
As of time of writing this note (release 2024.3 in progress) data-transfer uses a separate version for every service. Since data-transfer is part of Smart Power, we see value in unifying the data-transfer versioning scheme with the Mesh versioning system.
All releases (originating from new development and from backporting to the past releases) will start to receive a version according to the Mesh versioning system.
All new data-transfer executables will be stamped with release indicator and the build number: Major.Minor.Patch.(BuildNumber), where Major and Minor will match with corresponding Mesh version.
Changelog will be divided into release sections. We will leave current changelog entries unchanged as all data-transfer deployed at the customers use the old versioning scheme.
-
We have fixed an issue where the import request could not be opened using
MessageLogapplication. -
We have fixed an issue where an export of complex availability events would fail.
-
We have fixed a memory leak that occurred during timeseries export.
Version 2.13
[TR3.6.3] [MAR4.5.3] [EDS4.1.1] [DBGW2.1.1]
- We have renamed the
appsettings.jsonfile toappsettings.json.sampleto improve the upgrade process.
[TR3.6.2] [MAR4.5.2]
[TR3.6.1] [MAR4.5.1]
[TR3.6.0] [MAR4.5.0]
[DBGW2.1.0] [EDS4.1.0] [MAR4.4.0] [TR3.5.0]
[MAR4.3.13]
Version 2.12.1
Version 2.12.0
We are backporting .NET8 upgrade to data-transfer in Smart Power 2024.1. As of time of writing this note (release 2024.3 in progress) data-transfer uses a separate version for every service. Since data-transfer is part of Smart Power, we see value in unifying the data-transfer versioning scheme with the Mesh versioning system.
All releases (originating from new development and from backporting to the past releases) will start to receive a version according to the Mesh versioning system.
All new data-transfer executables will be stamped with release indicator and the build number: Major.Minor.Patch.(BuildNumber), where Major and Minor will match with corresponding Mesh version.
Changelog will be divided into release sections. We will leave current changelog entries unchanged as all data-transfer deployed at the customers use the old versioning scheme.
[MAR4.3.12]
[DBGW2.0.2]
[TR3.4.4] [MAR4.3.11]
- We have added Kerberos authentication support to the gRPC interface.
Services talking to Mesh using gRPC will now be able to authenticate to Mesh using Kerberos.
We have added a
ServerPrincipalconfiguration item inHttpEndpoints:Meshnode. The user should set its value to the server principal of the Mesh server. We have also removed theTlsCertificateconfiguration item fromHttpEndpoints:Meshnode. Users should enable secure Mesh communication by usinghttpsprotocol inHttpEndpoints:Mesh:Uri.
[TR3.4.3] [MAR4.3.10]
[TR3.4.2] [DBGW2.0.1] [MAR4.3.9] [EDS4.0.1]
- We have changed the way
CreatedandTransferredcolumns work inMessage Log. Until now,Createdwould display the time when the export has completed andTransferredwould be empty.Createdwill now display the time when the export order was received inTrigger RelayandTransferredwill display the time when the export has completed.
[EDS4.0.0] [DBGW2.0.0]
- We have changed the way EDS retrieves target queue names. In the previous versions there were 2 ways to specify target queue names to send the exports to.
- Using the
Queuesarray filled with queue aliases in the config file - Using
RECADRdatabase table to store the queue names. Broker should be specified in theBrokerconfiguration item. Queue names were associated with an export protocol and a receiver identifier.
Approach #1 turned out to be not flexible enough and approach #2 is too messy (keeping the broker name and the queue name separately). In this version we introduce (protocol, receiver)-to-queue mapping into EDS configuration file:
"Storage": {
"StoragePerProtocol": {
"PVPLAN": {
"StorageKinds": [ "Queue" ],
"DestinationQueues": {
"1": [ "exportReplyQueue" ],
"13": [ "exportReplyQueue", "someOtherQueue" ]
},
"DefaultQueues": [ "exportReplyQueue" ]
},
"PVPLAN2023": {
"StorageKinds": [ "File" ],
"StorePath": "C:\\files\\"
},
"StdExport": {
"StorageKinds": [ "Queue" ],
"DestinationQueues": {
"1": [ "exportReplyQueue" ],
"13": [ "exportReplyQueue", "someOtherQueue" ]
},
"DefaultQueues": [ "exportReplyQueue" ]
},
"APORAPOTExport": {
"StorageKinds": [ "Queue" ],
"DestinationQueues": {
"1": [ "exportReplyQueue" ],
"2": [ "exportReplyQueue", "someOtherQueue" ]
},
"DefaultQueues": [ "exportReplyQueue" ]
},
"BiddingTool": {
"StorageKinds": [ "File" ],
"StorePath": "C:\\files\\"
}
}
},
"QueuesConfiguration": {
"exportReplyQueue": {
"Broker": "B1",
"QueueName": "exportReplyQueue",
"Role": "Sender"
},
"someOtherQueue": {
"Broker": "B1",
"QueueName": "someOtherQueue",
"Role": "Sender"
}
},
-
We introduce
DestinationQueuesmapping intoStoragePerProtocolchildren nodes. It is a dictionary consisting of keys being export receiver identfiers and values being queue aliases arrays. When theDestinationQueueshas a key equal to an export receiver value for given export protocol, the export data will be sent to the queues in the respective array. -
In the
StoragePerProtocolnodes theQueuesarray is renamed toDefaultQueues. If theDestinationQueuesmapping does not provide target queue information,DefaultQueuesvalues are used as target queues.
[MAR4.3.8]
[TR3.4.1]
- We have added a possibility to specify a
SessionIdin timeseries and availability export requests. The session ID can be provided in an optionalSessionIdheader inOrderandAvailabilityExportendpoints
[TR3.4.0]
- We have introduced status reporting to Trigger Relay time series and availability export requests. (issue #196)
[EDS3.3.3]
[MAR4.3.6]
[MAR4.3.5]
[EDS3.3.2]
[MAR4.3.4]
[MAR4.3.3]
[MAR4.3.2]
[MAR4.3.1]
[MAR4.3.0]
[DBGW1.1.1]
[MAR4.2.8]
[MAR4.2.7]
[MAR4.2.6]
[MAR4.2.5]
[EDS3.3.1] [MAR4.2.4]
[MAR4.2.3]
[MAR4.2.2]
[MAR4.2.1]
[MAR4.2.0]
[MAR4.1.1]
[EDS3.3.0] [TR3.3.0] [MAR4.1.0] [DBGW1.1.0]
[TR3.2.1] [EDS3.2.3] [MAR4.0.1] [DBGW1.0.1]
[EDS3.2.2]
[EDS3.2.1]
gRPC Interface Support [MAR4.0]
- Mesh AMQP Relay is now able to communicate with Mesh using gRPC interface.
New parameters are introduced to
HttpEndpoints::Meshnode:MeshInterface- allows the user to choose how MAR communicates with Mesh. Possible values: {gRPC,XML}.TslCertificate- allows the user to specify the certificate used to encrypt the communication when usinggRPCinterface andHTTPS. It can be set tonullwhen usingXMLinterface.Urishould be adjusted to the URI ofXMLMesh server orgRPCMesh server.
```json "HttpEndpoints": { "Mesh": { "Uri": "https://localhost:50051", "MeshInterface": "gRPC", "TslCertificate": "C:\certs\certificate.crt", "RequestTimeout": 60 }, "ExportDataStore": { "Uri": "http://localhost:17000" }, "DatabaseGateway": { "Uri": "http://localhost:7137" } }
```
[EDS 3.2.0]
Single File Builds
- Since build #390 we will publish services with
Single File Buildoption enabled. This will reduce number of files in the directories greatly. This change requires adding following settings toSerilogoptions:"Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Enrichers.Thread" ],
[TR 3.1.1]
HttpEndpoints/DatabaseGateway configuration is required for the database connection.
Example:
"HttpEndpoints": {
"DatabaseGateway": {
"Uri": "http://localhost:7137"
}
}
[MAR3.2.0]
There is a new config container introduced: HttpEndpoints.
MeshHttpEndpoint/MeshHttpInterface is now moved to HttpEndpoints/Mesh/Uri.
MeshHttpEndpoint/RequestTimeout becomes HttpEndpoints/Mesh/RequestTimeout.
ExportWorker/ExportServiceAddress becomes HttpEndpoints/ExportDataStore/Uri.
Support of MeshHttpEndpoint and ExportWorker/ExportServiceAddress nodes is dropped.
HttpEndpoints/DatabaseGateway configuration is required for the database connection.
All the HTTP endpoints accept an optional RequestTimeout parameter (unit: seconds, default value: 60).
Example HttpEndpoints section:
"HttpEndpoints": {
"Mesh": {
"Uri": "http://localhost:20000/mesh",
"RequestTimeout": 60
},
"ExportDataStore": {
"Uri": "http://localhost:17000/",
"RequestTimeout": 30
},
"DatabaseGateway": {
"Uri": "http://localhost:7137"
}
}
[DBGW1.0.0]
It is supposed to be a database proxy for other data-transfer services. Currently DatabaseGateway is utilized by MAR and Trigger Relay. EDS still invokes database operations on its own.
There is a README for DatabaseGateway service.
[MAR3.0.9]
[EDS3.0.7]
[EDS3.0.6]
[TR3.0.5] [EDS3.0.5]
[MAR3.0.8]
[MAR3.0.7] [TR3.0.4] [EDS3.0.4]
[MAR3.0.6] [TR3.0.3] [EDS3.0.3]
[MAR3.0.5] [TR3.0.2] [EDS3.0.2]
[MAR3.0.4]
[TR3.0.1] [MAR3.0.3] [EDS3.0.1]
[MAR3.0.2]
New OpenTelemetry can be used in the MAR configuration file.
It offers following options:
"OpenTelemetry": {
"UseTraces": true, // Enable traces
"UseMetrics": true, // Enable metrics
"MetricsEndpoint": "http://localhost:3303" //Endpoint for metrics reading.
}
[MAR3.0.1]
[TR3.0.0] [MAR3.0.0] [EDS3.0.0]
Import Worker settings must contain ICC_TRANSLOG_DIR