Change Log
All notable changes to this project will be documented in this file.
Version 2.19
Features
Version 2.18.1
Features
- We have added support to Service Bus Topics and Subscriptions.
A new
Subscription
parameter was added to the queue configuration. WhenSubscription
is provided,QueueName
becomes the name of the topic.
Fixes
- We have fixed an issue where intermittent delays were observed when receiving messages from Service Bus queues. Fixing the issue requires configuration changes.
- The
ImportWorker:ReceiveTimeout
parameter controls the timeout of a message receive call, with a default value of 100 ms. Increasing the timeout might help to fix the issue. -
The
ServiceBus:PrefetchCount
parameter specifies the number of messages prefetched and cached before the actual message is requested. Default number is 0, meaning no prefetch. Increasing the value ofPrefetchCount
might help to fix the issue. -
We have fixed an issue where the import requests could not be identified in the debug logs.
Version 2.18
Breaking changes
The configuration file of the Trigger Relay service should be used as the base for the configuration of the new Mesh Data Transfer service.
The database configuration from the Database Gateway service needs to be applied to the Mesh Data Transfer settings file. The HttpEndpoints:DatabaseGateway
node should be removed from Mesh Data Transfer configuration. The Kestrel
configuration from the Database Gateway service should be dropped.
The configuration from Export Data Store settings file needs to be applied to Mesh Data Transfer settings file. Specifically:
- the
Storage
, node should be moved to Mesh Data Transfer settings, - if present, the
MercatoMapping
,UnitScheduleMapping
andTsVolumesWebServiceExport
nodes should be moved to Mesh Data Transfer settings, - the export queue should be added to the
QueuesConfiguration
section, - the contents of
ParticipantSettings
node should be merged (previously the node existed in the configuration files of both services), - the
HttpEndpoints:ExportDataStore
node should be removed from Mesh Data Transfer configuration.
The Kestrel
configuration from the Export Data Store service should be dropped.
The configuration from Mesh AMQP Relay settings file needs to be applied to Mesh Data Transfer settings file. Specifically:
ImportWorker
andFailedMessages
nodes should be applied to Mesh Data Transfer settings,QueuesConfiguration
in Mesh Data Transfer should be updated with the queues used for the imports (the import, import reply and failure queues),ImportExportCommonSettings:Delay
configuration parameter in Mesh AMQP Relay configuration is now moved toImportWorker:Delay
in Mesh Data Transfer configuration,ExportWorker
settings were removed and should not be applied in Mesh Data Transfer configuration file.
Additionally, the export related queues should be removed from AmqpSender:Queues
, as they were used to forward export requests to Mesh AMQP Relay and are not relevant anymore.
Fixes
Version 2.17.3
Breaking changes
Fixes
Version 2.17.2
Fixes
Version 2.16.3
Fixes
Version 2.15.6
Fixes
Version 2.16.2
Fixes
Version 2.15.5
Fixes
Version 2.17.1
Fixes
Version 2.17
Breaking changes
-
When exporting revision availability events, the category attribute will now contain a string "Revision". Previously the category was an empty string for the revision events.
Features
-
We have updated reading empty time series to align with changes in Mesh 2.17 changes.
-
We have added a new
ReadOnly
parameter to the DatabaseGateway service that prevents the service from modifying the database. Note: it does not prevent Mesh from database modifications. The read only mode in Mesh needs to be set separately in the Mesh configuration file.
Fixes
Version 2.16.1
Features
- We have added support for Time Series Volumes Web Service time series exports.
The settings for the new protocol are placed in
TsVolumesWebServiceExport
node in Export Data Store configuration file.
Fixes
Version 2.16
Features
-
We have added full support for the Service Bus queues. If the queues defined in the configuration file do not exist, data-transfer will attempt to create them automatically. The message lock time will be set to 5 minutes, other settings will remain default.
Fixes
Version 2.15.4
Features
- We have added an option to choose between
ora
andperiodo
nodes in Bidding export protocol. A newUsePeriodo
flag in the Bidding export protocol section in EDS configuration is used to switch between the nodes.
Fixes
Version 2.15.3
Features
- We have added an option to report
ImportSuccess = true
for partially successful time series import requests. The feature is toggled with an optionalPartialImportSuccess
parameter in theImportWorker
section of Mesh AMQP Relay configuration file.
Fixes
Version 2.15.2
Breaking changes
-
We have added support for the Decimals export attribute in GS2 time series export. Since this attribute was previously ignored, GS2 time series export might start to produce different results if the Decimals attribute was already defined in the time series export definition.
-
We have added support for the UTC offset in GS2 time series export. The standard time is used when custom UTC offset is not provided.
Features
- We have added a new optional parameter named DefaultDecimals that can be applied per export protocol in the Export Data Store configuration. The DefaultDecimals parameter is supported for the EDIEL DELFOR and GS2 export types only.
Fixes
Version 2.15.1
Breaking changes
- We have changed the behaviour of the time series export when different export senders are used. The export results are now grouped per sender. If no sender is defined as the export request parameter, then the time series sender is either the sender host from the export definition or the default sender when the sender host is not defined. If the sender is defined as the export request parameter, the time series export definitions with a defined sender host that doesn't match the request sender are ignored during the export; the sender of all the time series is the request sender.
Fixes
Version 2.15
Breaking changes
- We have added a new mandatory configuration parameter for TriggerRelay:
ParticipantSettings:DefaultSender
. It was done as a part of #283 and #316. TheDefaultSender
parameter is a number that denotes a participant key to be used by default when exporting data from Mesh. Example configuration:
"ParticipantSettings": {
"DefaultSender": 13
}
Features
-
We have added support for Bidding Strategy export with a 15-minute time series resolution.
-
We have added support for changing the resolution of the exported time series. NB! The behavior of an export with a target resolution differs from the old export in release 2.15. When exporting time series points from an attribute that holds a physical time series, the
Id
field in the Standard ExportPoints
node will contain a time series source ID instead of a physical time series ID. If the target export resolution is not defined, theId
will still contain the physical time series ID.
<Timeseries QueryId="2ad565bc-412c-4674-b981-b1ad24537799">
<Points Id="d3163dae-81de-413a-8c55-bc11ad5080c8" Path="XYZ" DeltaT="PT15M" Reference="ABC" UnitOfMeasurement="MW" UnitOfMeasurementName="MW" CurveType="StaircaseStartOfStep">
<Segments>
...
</Segment>
We plan to place the time series source ID in the Standard Export's Id
field unconditionally starting from the release 2.16 (Smart Power 2024.5).
Fixes
-
We have changed the type of the attribute
UnitOfMeasurement
inTimeseriesPoints
type fromUUID
toxsd:string
. In consequence, theAmqpMessageTypes.xsd
schema version has been bumped to 1.1. To start using version 1.1, please setStorage:StoragePerProtocol:StdExport:SchemaVersion
item to "1.1" in Export Data Storeappsettings.json
. If this value is not specified, it defaults to version 1.0. Note! We will stop supporting schema version 1.0 in Mesh Data Transfer release 2.17. Note! When using schema version 1.0, theUnitOfMeasurement
attribute value does not comply with its type defined in the schema. -
We have fixed an issue where the Mesh export response message size would exceed the default 4 MB
-
We have fixed an issue where Message Log timestamps would not be aligned to the database time zone.
Version 2.14.2
Version 2.14.1
Version 2.14
As of time of writing this note (release 2024.3 in progress) data-transfer uses a separate version for every service. Since data-transfer is part of Smart Power, we see value in unifying the data-transfer versioning scheme with the Mesh versioning system.
All releases (originating from new development and from backporting to the past releases) will start to receive a version according to the Mesh versioning system.
All new data-transfer executables will be stamped with release indicator and the build number: Major.Minor.Patch.(BuildNumber)
, where Major
and Minor
will match with corresponding Mesh version.
Changelog will be divided into release sections. We will leave current changelog entries unchanged as all data-transfer deployed at the customers use the old versioning scheme.
-
We have fixed an issue where the import request could not be opened using
MessageLog
application. -
We have fixed an issue where an export of complex availability events would fail.
-
We have fixed a memory leak that occurred during timeseries export.
Version 2.13
[TR3.6.3] [MAR4.5.3] [EDS4.1.1] [DBGW2.1.1]
- We have renamed the
appsettings.json
file toappsettings.json.sample
to improve the upgrade process.
[TR3.6.2] [MAR4.5.2]
[TR3.6.1] [MAR4.5.1]
[TR3.6.0] [MAR4.5.0]
[DBGW2.1.0] [EDS4.1.0] [MAR4.4.0] [TR3.5.0]
[MAR4.3.13]
Version 2.12.1
Version 2.12.0
We are backporting .NET8 upgrade to data-transfer in Smart Power 2024.1. As of time of writing this note (release 2024.3 in progress) data-transfer uses a separate version for every service. Since data-transfer is part of Smart Power, we see value in unifying the data-transfer versioning scheme with the Mesh versioning system.
All releases (originating from new development and from backporting to the past releases) will start to receive a version according to the Mesh versioning system.
All new data-transfer executables will be stamped with release indicator and the build number: Major.Minor.Patch.(BuildNumber)
, where Major
and Minor
will match with corresponding Mesh version.
Changelog will be divided into release sections. We will leave current changelog entries unchanged as all data-transfer deployed at the customers use the old versioning scheme.
[MAR4.3.12]
[DBGW2.0.2]
[TR3.4.4] [MAR4.3.11]
- We have added Kerberos authentication support to the gRPC interface.
Services talking to Mesh using gRPC will now be able to authenticate to Mesh using Kerberos.
We have added a
ServerPrincipal
configuration item inHttpEndpoints:Mesh
node. The user should set its value to the server principal of the Mesh server. We have also removed theTlsCertificate
configuration item fromHttpEndpoints:Mesh
node. Users should enable secure Mesh communication by usinghttps
protocol inHttpEndpoints:Mesh:Uri
.
[TR3.4.3] [MAR4.3.10]
[TR3.4.2] [DBGW2.0.1] [MAR4.3.9] [EDS4.0.1]
- We have changed the way
Created
andTransferred
columns work inMessage Log
. Until now,Created
would display the time when the export has completed andTransferred
would be empty.Created
will now display the time when the export order was received inTrigger Relay
andTransferred
will display the time when the export has completed.
[EDS4.0.0] [DBGW2.0.0]
- We have changed the way EDS retrieves target queue names. In the previous versions there were 2 ways to specify target queue names to send the exports to.
- Using the
Queues
array filled with queue aliases in the config file - Using
RECADR
database table to store the queue names. Broker should be specified in theBroker
configuration item. Queue names were associated with an export protocol and a receiver identifier.
Approach #1 turned out to be not flexible enough and approach #2 is too messy (keeping the broker name and the queue name separately). In this version we introduce (protocol, receiver)-to-queue mapping into EDS configuration file:
"Storage": {
"StoragePerProtocol": {
"PVPLAN": {
"StorageKinds": [ "Queue" ],
"DestinationQueues": {
"1": [ "exportReplyQueue" ],
"13": [ "exportReplyQueue", "someOtherQueue" ]
},
"DefaultQueues": [ "exportReplyQueue" ]
},
"PVPLAN2023": {
"StorageKinds": [ "File" ],
"StorePath": "C:\\files\\"
},
"StdExport": {
"StorageKinds": [ "Queue" ],
"DestinationQueues": {
"1": [ "exportReplyQueue" ],
"13": [ "exportReplyQueue", "someOtherQueue" ]
},
"DefaultQueues": [ "exportReplyQueue" ]
},
"APORAPOTExport": {
"StorageKinds": [ "Queue" ],
"DestinationQueues": {
"1": [ "exportReplyQueue" ],
"2": [ "exportReplyQueue", "someOtherQueue" ]
},
"DefaultQueues": [ "exportReplyQueue" ]
},
"BiddingTool": {
"StorageKinds": [ "File" ],
"StorePath": "C:\\files\\"
}
}
},
"QueuesConfiguration": {
"exportReplyQueue": {
"Broker": "B1",
"QueueName": "exportReplyQueue",
"Role": "Sender"
},
"someOtherQueue": {
"Broker": "B1",
"QueueName": "someOtherQueue",
"Role": "Sender"
}
},
-
We introduce
DestinationQueues
mapping intoStoragePerProtocol
children nodes. It is a dictionary consisting of keys being export receiver identfiers and values being queue aliases arrays. When theDestinationQueues
has a key equal to an export receiver value for given export protocol, the export data will be sent to the queues in the respective array. -
In the
StoragePerProtocol
nodes theQueues
array is renamed toDefaultQueues
. If theDestinationQueues
mapping does not provide target queue information,DefaultQueues
values are used as target queues.
[MAR4.3.8]
[TR3.4.1]
- We have added a possibility to specify a
SessionId
in timeseries and availability export requests. The session ID can be provided in an optionalSessionId
header inOrder
andAvailabilityExport
endpoints
[TR3.4.0]
- We have introduced status reporting to Trigger Relay time series and availability export requests. (issue #196)
[EDS3.3.3]
[MAR4.3.6]
[MAR4.3.5]
[EDS3.3.2]
[MAR4.3.4]
[MAR4.3.3]
[MAR4.3.2]
[MAR4.3.1]
[MAR4.3.0]
[DBGW1.1.1]
[MAR4.2.8]
[MAR4.2.7]
[MAR4.2.6]
[MAR4.2.5]
[EDS3.3.1] [MAR4.2.4]
[MAR4.2.3]
[MAR4.2.2]
[MAR4.2.1]
[MAR4.2.0]
[MAR4.1.1]
[EDS3.3.0] [TR3.3.0] [MAR4.1.0] [DBGW1.1.0]
[TR3.2.1] [EDS3.2.3] [MAR4.0.1] [DBGW1.0.1]
[EDS3.2.2]
[EDS3.2.1]
gRPC Interface Support [MAR4.0]
- Mesh AMQP Relay is now able to communicate with Mesh using gRPC interface.
New parameters are introduced to
HttpEndpoints::Mesh
node:MeshInterface
- allows the user to choose how MAR communicates with Mesh. Possible values: {gRPC
,XML
}.TslCertificate
- allows the user to specify the certificate used to encrypt the communication when usinggRPC
interface andHTTPS
. It can be set tonull
when usingXML
interface.Uri
should be adjusted to the URI ofXML
Mesh server orgRPC
Mesh server.
```json "HttpEndpoints": { "Mesh": { "Uri": "https://localhost:50051", "MeshInterface": "gRPC", "TslCertificate": "C:\certs\certificate.crt", "RequestTimeout": 60 }, "ExportDataStore": { "Uri": "http://localhost:17000" }, "DatabaseGateway": { "Uri": "http://localhost:7137" } }
```
[EDS 3.2.0]
Single File Builds
- Since build #390 we will publish services with
Single File Build
option enabled. This will reduce number of files in the directories greatly. This change requires adding following settings toSerilog
options:"Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Enrichers.Thread" ],
[TR 3.1.1]
HttpEndpoints/DatabaseGateway
configuration is required for the database connection.
Example:
"HttpEndpoints": {
"DatabaseGateway": {
"Uri": "http://localhost:7137"
}
}
[MAR3.2.0]
There is a new config container introduced: HttpEndpoints
.
MeshHttpEndpoint/MeshHttpInterface
is now moved to HttpEndpoints/Mesh/Uri
.
MeshHttpEndpoint/RequestTimeout
becomes HttpEndpoints/Mesh/RequestTimeout
.
ExportWorker/ExportServiceAddress
becomes HttpEndpoints/ExportDataStore/Uri
.
Support of MeshHttpEndpoint
and ExportWorker/ExportServiceAddress
nodes is dropped.
HttpEndpoints/DatabaseGateway
configuration is required for the database connection.
All the HTTP endpoints accept an optional RequestTimeout
parameter (unit: seconds, default value: 60).
Example HttpEndpoints
section:
"HttpEndpoints": {
"Mesh": {
"Uri": "http://localhost:20000/mesh",
"RequestTimeout": 60
},
"ExportDataStore": {
"Uri": "http://localhost:17000/",
"RequestTimeout": 30
},
"DatabaseGateway": {
"Uri": "http://localhost:7137"
}
}
[DBGW1.0.0]
It is supposed to be a database proxy for other data-transfer services. Currently DatabaseGateway is utilized by MAR and Trigger Relay. EDS still invokes database operations on its own.
There is a README for DatabaseGateway service.
[MAR3.0.9]
[EDS3.0.7]
[EDS3.0.6]
[TR3.0.5] [EDS3.0.5]
[MAR3.0.8]
[MAR3.0.7] [TR3.0.4] [EDS3.0.4]
[MAR3.0.6] [TR3.0.3] [EDS3.0.3]
[MAR3.0.5] [TR3.0.2] [EDS3.0.2]
[MAR3.0.4]
[TR3.0.1] [MAR3.0.3] [EDS3.0.1]
[MAR3.0.2]
New OpenTelemetry
can be used in the MAR configuration file.
It offers following options:
"OpenTelemetry": {
"UseTraces": true, // Enable traces
"UseMetrics": true, // Enable metrics
"MetricsEndpoint": "http://localhost:3303" //Endpoint for metrics reading.
}
[MAR3.0.1]
[TR3.0.0] [MAR3.0.0] [EDS3.0.0]
Import Worker settings must contain ICC_TRANSLOG_DIR