What are topic views?

A topic tree is typically created and maintained by client applications that publish data. But the consumer applications of the data may want it to be presented in many different ways. Topic views provides a simple way to map a fairly static topic tree design onto many different mapping requirements for consumers.

A topic view maps a selected part of a server’s topic tree to another. It dynamically creates a set of reference topics from a set of source topics, based on a declarative topic view specification.

The capabilities of topic views range from simple mirroring of topics within the topic tree to advanced capabilities including publication of partial values, expanding a single topic value into many topics, calculating new topic values, inserting values from other topics, throttling the rate of publication, and applying a fixed delay to the publication.

For example, you can transform a source topic with a large, complex JSON value into a set of simpler reference topics. Clients can then subscribe only to the individual topics they need.

A topic view can also map topics from another server (in a different cluster). This capability is referred to as remote topic views. The view can specify the server that the source topics are hosted on in terms of a remote server (see Remote topic views).

Each reference topic has a single source topic and by default has the same topic type as its source topic (but this can be changed using the type option).

Reference topics are read-only (they cannot be updated), nor can they be created or removed directly. Otherwise, they behave just like standard topics. A client session can subscribe to a reference topic, and can fetch the reference topic’s current value if it has one.

The source topics of a topic view are defined by a topic selector. One or more reference topics are created for each source topic, according to the topic view specification. If a source topic is removed, reference topics that are derived from it will automatically be removed. If a topic is added that matches the source topic selector of a topic view, corresponding reference topics will be created. Removing a topic view will remove all of its reference topics.

Many of the more powerful features of topic views only apply to JSON topics (marked as JSON only). Some also support TIME_SERIES topics that have an event type of JSON (marked JSON or TIME_SERIES<json> only).

Items within JSON topic values are addressed using JSON pointers as defined by RFC6901.

Topic view specifications

The following is a simple topic view specification that mirrors all topics below the path a to reference topics below the path b.

map ?a/ to b/<path(1)>

A topic view with this specification will map a source topic at the path a/x/y/z to a reference topic at the path b/x/y/z. The specification is simple, so the reference topic will exactly mirror the source topic.

Whitespace is significant when separating clauses, but any amount of whitespace can be used.

For readability, clauses can be separated by line endings. Lines prefaced with # are treated as comments. For example :

map ?A/
from Server1
to <path(1)>
# Join 2 topics
insert Topic2 at /T2
insert Topic3 at /T3
throttle to 1 update every minute
as <value(/foo)>
      	

A topic view specification comprises three main parts:

1. The mapping part which specifies the source topics to map from and the mappings to target reference topics.
2. Optional transformations which transform the topic value in some way.
3. Optional options which specify other changes that the view may apply.

Mapping comprises:

• The map clause identifying the source topics that the view can apply to. This is specified in terms of the word map followed by a topic selector. All topics that match the selector will be considered as potential source topics for the topic view.
• The optional from clause which may identify a remote server that hosts the source topics.
• The to clause which determines how reference topic paths are derived from the source topic(s).

Transformations can be:

• process transformations that allow conditional processing and/or calculations to be applied to the reference topic value.
• patch transformations specifying that a JSON patch is applied to the reference topic value.
• insert transformations specifying that values from other topics are inserted into the reference topic value.

Options can be:

• The with properties option which determines how reference topic properties are derived from source topic properties.
• The throttle option which constrains the rate at which each reference topic is updated when its source topic is updated.
• The delay option which causes a change to a topic view’s source topic to be delayed by a fixed time before it is reflected in reference topics.
• The type option which can specify that the reference topic that is created is of a different topic type from the selected source topic.
• The separator option which can define a replacement path separator for path segments extracted from scalar values within the input value.
• The as option which provides a simple way to extract a part of the value derived from the source topic.
• The preserve topics option which overrides the default removal behavior of generated reference topics.

Mapping

The map clause

The map clause begins with the map keyword and is followed by a topic selector. These topic selectors follow the same parsing rules as standard Diffusion Cloud topic selectors.

When evaluating a topic view, all topics in the topic tree that match the source topic selector are considered, assuming the creator of the topic view has the necessary permission to read the topic. In addition, when a topic view contains a feature that is marked as JSON only or JSON and TIME_SERIES<json> only, then only topics that are of those types will be selected.

The from clause

The from clause optionally follows the map clause. It begins with the from keyword and is followed by a remote server name. The name refers to a remote server created using the remote servers feature (see Remote topic views).

The presence of the clause indicates that the source topics will be selected from the specified server and not from the local server.

For example, to map topic A and all of its descendants to the same structure on the local server :

map ?A// from server1 to <path(0)>

The to clause

The paths of generated reference topics are derived from the source topic according to the to clause which comprises the to keyword followed by a path mapping template. The path mapping template allows the source topic path and the value of the source topic (JSON or TIME_SERIES<json> only) to determine the path of the reference topic. In addition the path mapping template can include expand directives (JSON only) which allow objects and arrays within the source topic values to be expanded to produce many reference topics.

A path mapping template is a topic path with embedded directives. Directives are evaluated when creating the topic reference and substituted into the topic path. Directives are delimited by angle brackets (<, >) and consist of the name of the directive and a list of parameters. The parameter list is comma-separated and surrounded by parentheses ((, )).

The path mapping template can comprise literal segments, path separators (/) and path directives. Some examples using valid templates are :

map ?A/ to Accounts/<path(1)>
map ?A/ to Accounts/<scalar(/Name)>
map ?A/ to Accounts/<scalar(/Name, 0)>
map ?A/ to Accounts/<expand()>/<scalar(/Name)>
map ?A/ to <expand(/Prices)><scalar(/Currency)>
		

The use of the various directives is described in detail below.

path directives

Source path directives extract a portion of the source path and are parameterized by the index of the start part of the source path and the number of parts to include. The number of parts parameter is optional – if it is missing, the selection extends to the end of the source path. The syntax is <path(start,number)>, or <path(start)> when the number of parts parameter is omitted.

Path indexes start from 0, so the first part of a path is addressed as part <path(0,1)> and the full path after the topmost level is addressed as <path(1)>.

For example, given the source path a/b/c/d, the path directive <path(1, 2)> is mapped to the reference topic path b/c, and the path directive <path(2)> is mapped to the reference topic path c/d.

scalar directives

Source value directives use the keyword scalar and are parameterized by a single JSON pointer that extracts a scalar value from the source (or current) value. A scalar value is a string, a number, a boolean, or null, that is, anything other than an array or an object. If the JSON pointer does not refer to a scalar value within the source (or current) value, no reference topic will be created (this includes cases where the JSON pointer refers to an array or an object).

For example, given a source value of :

{
      "account" : "1234",
      "balance" : { "amount" : 12.57, "currency" : "USD" }
}
		

and the source value directive :

currency/<scalar(/balance/currency)>/account/<scalar(/account)>

the reference topic path will be currency/USD/account/1234.

If the extracted value is a string, it is copied literally to the reference topic path. If another type of scalar is extracted its string representation is copied to the reference topic path.

A value that contains path separators (/) will create a reference topic path with more levels than the path mapping template. You can use the separator option to replace path separators with an alternative string in order to avoid such extra levels in the topic tree.

Foreign scalar directives

A useful extension to the scalar directive described above is the ability to specify that the scalar value is to be read from a topic higher up the topic tree hierarchy than the topic being evaluated. The most common reason you would want to do this is so that a whole branch can be copied without the need for the scalar value being in all of the descendant topics being evaluated.

Foreign scalar directives specify a second integer parameter which is an index to the highest segment of the path being processed and is used to locate a topic higher up the current topic's path hierarchy. The index starts from 0 so a value of 0 would indicate the root topic. For example, if the topic being evaluated had the path A/B/C and the directive was <scalar(/Account, 0)> the scalar value of /Account would be read from the topic at path A.

Consider the following view specification:

map ?A// to <scalar(/Account, 0)>/<path(0)>

The above would process A and all of its descendants, generating reference topics for all of them with a path the same as the source topic but prefixed by the value read from /Account in topic A. If /Account contained the value "Smith" then A would map to Smith/A, A/B would map to Smith/A/B and so on.

However, what would occur if this specification was changed to:

map ?A// to <scalar(/Account, 1)>/<path(0)>

In this case, when evaluating A, no mapping would occur as the path has insufficient segments to determine the topic to read from. But when evaluating A/B the value will be read from the topic itself, and when evaluating A/B/C the value will be read from A/B also.

When a value that topics are dependent upon changes, the affected topic views are automatically re-evaluated for those dependent topics and reference topics will be removed and added according to recalculated path mappings.

If the topic to be read from does not exist, or the scalar value could not be found, a dependency is set up which is periodically checked and re-evaluation will occur if the topic is later created and/or the value can be read.

If the topic to be read from exists but is not a JSON or TIME_SERIES<JSON> topic then an error will be logged and the view will not evaluate.

When using this feature, ideally the scalar value that is read from should not change too often as that will cause background re-evaluations involving the addition or removal of reference topics.

expand directives

An expand directive uses the keyword expand and is parameterized by one or two JSON pointers.

The first pointer indicates the element within the value to be expanded, and if omitted, the value is expanded from the root. Expansion of a source topic indicates that every direct child of the element pointed to by the pointer will be used to create a new reference topic (or provide input to later expand or scalar directives and transformations).

For example <expand()> would expand every child item in the input value and <expand(/account)> would expand every child of the account value in the input value. The specified value could be an object, an array or even a scalar value, but a scalar value would expand to only a single new value.

The optional second parameter of the expand directive specifies a pointer to a scalar value within the expanded value which will be used to derive the path fragment of the reference topic path. If the second pointer is not specified or no scalar value is found for the pointer, the path fragment is taken from the key (if the child value is an object) or the index (if the child value is an array). Scalar child values will expand to a reference topic but will not add anything to the generated path. For example <expand(,/name)> would expand from the root of the source value and each child value path fragment would be obtained from the scalar value with the key name.

So if a source topic had a value of

{"values": [1, 5, 7]}

a path mapping of

value<expand(/values)>

would expand the value to the followingJSON reference topics:-

path value0 with a value of 1

path value1 with a value of 5

path value2 with a value of 7

Expansion directives can be nested (i.e. there can be more than one expand directive in a path mapping template). In this case a second expand directive will use the value from the previous expand as its input (root) value and not the value of the source topic. This also applies to scalar directives that follow an expand directive.

If expansion causes more than one mapping to the same topic path, only the first encountered will be created and updated.

For a more complex example, suppose this is the content of a JSON topic called allCars :

{
      "cars": [
          { "reg":"HY58XPA", "type":"Ford", "model":"Sierra" },
          { "reg":"PY59GCA", "type":"Fiat", "model":"Panda"},
          { "reg":"VA63ABC", "type":"Ford", "model":"Ka"}
      ]
}
		

The following topic view specification:

map allCars to cars/<expand(/cars, /reg)>

results in these reference topic paths:

cars/HY58XPA
cars/PY59GCA
cars/VA63ABC
		

The value of cars/HY58XPA is:

{
    "reg":"HY58XPA",
    "type":"Ford",
    "model":"Sierra"
}
		

In the example above, if you used the topic view

map allCars to cars/<expand(/cars)>

the topic path is taken from the index of the current array element resulting in topics:

cars/0
cars/1
cars/2

Suppose the previous array example is extended so that each car can have multiple drivers:

{
    "cars": [
        { "reg":     "HY58XPA",
          "drivers": [{"name" : "Bill"}, {"name" : "Fred"}]
        },
        { "reg":     "PY59GCA",
          "drivers": [{"name" : "Jane"}, {"name" : "Fred"}]
        },
        { "reg":     "VA63ABC",
          "drivers": [{"name" : "Tom"}, {"name" : "John"}]
        }
    ]
}
		

This topic view uses nested expand directives to expand both levels of the array hierarchy:

map allCars to cars/<expand(/cars, /reg)>/drivers/<expand(/drivers, /name)>

resulting in these reference topics:

cars/HY58XPA/drivers/Bill
cars/HY58XPA/drivers/Fred
cars/PY59GCA/drivers/Jane
cars/PY59GCA/drivers/Fred
cars/VA63ABC/drivers/Tom
cars/VA63ABC/drivers/John
		

The second expand directive takes the /drivers values from the previous expand, so the value of each topic is {"name":"Bill"}, {"name":"Fred"}, {"name":"Jane"} and so on.

Transformations

Transformations are specified after the mapping and before any options.

Transformations are applied to the current value (that being the source topic value or the value output from an expand or a previous transformation), in the order specified. There can be any number of transformations interspersed with one another and the output value from one will be that which is input to the next. The only restriction is that all insert transformations must occur at the end, after any others.

process transformations

A process transformation may be used to apply conditional processing to a value (optionally determining whether a reference topic is created) and/or change the value in some way (for example, by applying some calculation to a field within the value).

The format of a process transformation is:-

process {statement}

Where the statement can be:-

• One or more operations separated by ;.

• A conditional statement comprising one or more conditions with operations to perform if they are satisfied.

For example, the following topic view specification could be used to write a field into the value of the reference topic :

map ?a/ to b/<path(1)> process {set(/Name, 'John')};

And the following shows a more complex statement which would set a field according to the value of the input field Price :

map ?a/ to b/<path(1)> process {if '/Price lt 50' set(/Tier, 1) elseif '/Price gt 50' set(/Tier, 2)}

The process statement takes its input value from the source topic, or any previous expand or transformation and applies conditions or operations to that value to produce the output value. Fields that are not changed by the process are copied 'as is' to the output value.

Operations can be 'chained' by separating them with a ;. For example :

set(/Amount, calc "/Value * /Number"); remove(/Value); remove(/Number)

In this case, all of the operations are performed on the original value, creating a chain of deltas which are only applied to the original value at the end. If the set operation fails, no reference topic would be generated, however processing will continue if the fields specified in any remove operation are not present.

Operations

The following operations are supported :

• Set to value
• Set to scalar
• Set to calculated value
• Remove
• Continue

Set to value

This operation sets a scalar field in the output JSON value to a literal value.

The format of the operation is :

set(pointer , value)

Where pointer is a JSON pointer indicating the location in the output value at which the value is to be written.

The value can be any scalar value (e.g. a string, number, boolean or null). For example :

set(/A, "Hello")
set(/B, 99)
set(/C, 123.45)
set(/D, true)
SET(/E, null)
		

If a value already exists at the given pointer of the input value, it is replaced. If the value does not exist it is added to the end of the parent structure.

If a hierarchic pointer is used, the parent node must already exist within the value. For example :

set(/Account/Name, "Jones")

In the above case the Account node must already exist as an object.

set(/Account/2, "Jones")

The above example could be used to replace item 2 in the Account array. To add an item to the end of an array the special pointer value /- can be used :

set(/Account/-, "Jones")

Set to scalar

This operation sets a scalar item to the value of another scalar item within the input value.

The format of the operation is :

set(target , source)

Where target is a JSON pointer indicating the location in the output value at which the value is to be written and source is the location of a scalar within the input value. If there is no scalar value at the specified source pointer, no reference topic will be generated.

For example :

set(/Target, /Source)

The operation follows the same rules as set to value in terms of the target pointer requirements.

By default, the source value will be copied 'as is' to the target location. However, if it is required to change the format of the target value in some way then the following special format can be used :

set(target ,source format)

Where format is a specification of the format of the output value. For example :

set(/A, /B format 'type=string')

For full details of the syntax of the formatting specification see Operation formatting.

Set to calculated value

This operation sets a scalar item to the result of a calculation involving values from within the input value.

The format of the operation is :

set(pointer , calc "calculation")

Where pointer is a JSON pointer indicating the location in the output value at which the calculated result is to be written, and calculation is a calculation string as described below. For example :

set(/A, calc "/Value * 2")

The above would set the scalar field at A to the value at Value multiplied by 2.

A calculation is a simple arithmetic calculation performed upon scalar numbers or scalar strings containing numbers.

Arithmetic operators supported are +, - , * and /.

Operands may be scalar values (which are numbers or strings containing parseable numbers) within the input value or literal number values (string literals are not permitted).

Examples of calculations are:-

set(/Value, calc "/Value * 2")
set(/Result, calc "/Value / 2")
set(/Result, calc "/Value * 2.5)
set(/Bonus, calc "/Salary + 1000")
set(/Bonus, CALC "/Salary + 1000 + /Age * 10")
		

Standard operator precedence is applied, so in the last example above we have (/Salary + 1000 + (/Age * 10)) not ((/Salary + 1000 + /Age) * 10). Brackets may be used to override this.

If any pointer value does not exist, or contains a string value that cannot be parsed as a number, no reference topic will be generated.

An extension to this operation is the ability to provide formatting rules for the output value as follows :

set(pointer ,calc calculation format)

Where format is a specification of the format of the output value. For example :

set(/Value, calc "/Value * 2" format "type=string")

For full details of the syntax of the formatting specification see Operation formatting.

The formatting also defines the rules for rounding and the number of decimal places required in the result.

If, at runtime, all specified values are found to be integers then integer calculations are performed and default formatting will produce an integer value, rounded down. However, if the calculation involves divisions, interim results are maintained as decimals to ensure maximum precision, but the result will still be an integer. This means that the result of 3 / 2 would be 1, but the result of(3 / 2) * 4 would be 6. Therefore, where division is in use it would usually be better to format the result as a decimal value.

Calculations that involve at least one decimal number are performed using decimal calculation with maximum precision, and by default the result is formatted with a scale of 2 and rounding of half_up. To achieve a different number of decimal places, or use a different rounding mode, formatting would need to be applied.

Some examples of calculations and the effects of formatting are shown below.

Arithmetic failures, such as zero divide will result in no reference topic being created.

Remove

This operation removes an element from the input value.

The format of the operation is :

remove(pointer)

Wherepointer is a JSON pointer indicating the item to be removed, which can be a scalar value or a structure (object or array).

For example :

remove(/A)

The operation acts on the output only so a value can still be used in another operation within the process even after it is removed, for example :

remove(/A);set(/B, /A)

If an item specified in a remove is not found, processing will continue, therefore it is not necessary to know that the item exists in the input. This is in contrast to using the patch transformation for removals where removal of a non existent item would cause the transformation to fail.

Continue

This is a special operation that indicates that the topic view evaluation should continue with the value as it is. This is only for use with conditional statements as the default behavior of a conditional statement is not to proceed if no condition is satisifed. For example :

if "/Age > 65" set(/Retired, true) else continue

Operation Formatting

Formatting can be used in set to scalar or set to calculated value operations, for example :

set(/A, /B format 'scale=3')
set(/A, calc "/A * 5" format "type=string")
		

set(/A, /B f"type=string")

The formatting is specified as a string with key=value pairs separated by,.

The allowed formatting options are as follows:-

Any option that is omitted assumes the default as specified above, but if no formatting is specified at all, the default depends upon the operation being used.

The following formatting rules apply:

• If type specifies number and the result cannot be rendered as a number, no reference topic is generated.
• If type specifies string and the input is a number or a string containing a number, the scale and round rules are applied to the number before rendering it as a string.
• If type specifies string and the input is a string that cannot be parsed as a number the value is simply copied.
• If type specifies number and the input is boolean or null, no reference topic is generated.
• If type specifies string and the input is boolean or null then the output is "true", "false" or "null" as appropriate.

Some examples of formatting are

type=string
scale=4,round=up
t=s,s=2,r=down
		

Rounding Modes

The following table shows the result of rounding input to one digit with the given rounding mode.

The most commonly used rounding modes would be half_up or down. The unnecessary mode would very rarely be used unless you want to ensure an exact result as an error would mean that no reference topic would be generated.

Conditionals

A conditional statement is made up of an if clause, optionally followed by one or more elseif clauses and an optional final else clause.

if clause

This takes the form :

if "condition" operation(s)

Where the condition is a quoted string as described in Conditions and the operation(s) are as described previously. If the condition is satisfied, the operations are applied to the value and the process is complete. If the condition is not satisfied, processing moves on to any elseif or else clauses that follow, but if there are none, the topic view evaluation does not proceed and no reference topic is created.

if "/Age >= 65" set(/Retired, true)

The above example filters out any source topics where Age is less than 65 and therefore setting the value may be superfluous, so a simpler filter could be simply :

if "/Age >= 65" continue

elseif clause

This takes the form :

elseif "condition" operation(s)

If the condition is satisfied, the operations following it are applied to the value and the process is complete. If the condition is not satisfied, processing moves on to any elseif or else clauses that follow, but if there are none, the topic view evaluation does not proceed and no reference topic is created.

if "/Age < 13" set(/Type, "Child") elseif "/Age < 20" set(/Type, "Teen")

In the above case, no reference topic would be created if Age is not less than 20. So this is also effectively filtering topics.

else clause

This takes the form :

else operation(s)

And will only be reached if no previous if or elseif conditions were satisfied. If reached then the operations are applied to the value and the topic view evaluation proceeds.

if "/Age < 13" set(/Type, "Child") elseif "/Age < 20" set(/Type, "Teen") else set(/Type, "Adult")

The use of else therefore ensures that there is always a result.

The continue operation may be used to proceed with an unchanged value.

if ("/Age < 13" set(/Restricted, true) else continue

Conditions

A condition is of the form:

pointer operator [constant/pointer]

Where pointer is a JSON pointer, operator is a relational operator and constant is any valid scalar value. For example :

/Age > 40
/Price > 14.50
/Name = "Bill"
/Age > /RetirementAge
/Manager eq true
/Used = null
		

Comparison operators are:

Compound conditions are supported by means of boolean operators:

| or or

& or and

For example:

/Age = 50 or /Age > 80
/Age gt 50 & /Department eq "Accounts"
		

Normal boolean precedence applies but brackets can be used to control precedence. For example:

(/Age > 50 or /Department eq "Accounts") and /Band > 3

Boolean 'not' is also allowed :

not (/Age < 65 or /Retired eq false)

patch transformations

patch transformations specify that a JSON patch is to be applied to the input value.

The format of a patch transformation is

patch "patch string"

The patch string should be formatted according to the JSON patch standard RFC6902.

Patches are a sequence of JSON Patch operations contained in an array. They are applied as an atomic update to the input value if the resulting update is successfully calculated.

The following patch will check the value at a specific key and update if the expected value is correct:

[{"op":"test", "path":"/price", "value" : 22}, {"op":"add", "path":"/price", "value": 23}]

The available operations are:

The test operation checks that the CBOR representation of the value of a topic is identical to the value provided in the patch after converting it to CBOR. If the value is represented differently as CBOR, commonly due to different key ordering, the patch will return the index of the failed operation. e.g the values {"foo": "bar", "count": 43} and {"count": 43, "foo": "bar"} are unequal despite semantic equality due to the differences in a byte for byte comparison.

The following patch clause would add the price field and remove the name field from an input JSON object.

patch '[
    {"op":"add", "path":"/price", "value" : 22.10},
    {"op":"remove", "path":"/name"}
]'
		

Patches can only be applied to JSON arrays or objects and if they fail to apply, no resulting reference topic will be created by the view. If an update patch fails, any previously created reference topic would be removed.

insert transformations

insert transformations are used to insert a value from another topic into the current value.

The name of the topic to insert from can be specified in a similar way to the path mapping in that constants, path directives, and scalar directives (but not expand directives) may be used.

The value from the insertion topic (if found) is inserted into the current value at a specified key position. The current value may be the source topic value, the value output from expand directives (in which case the insertion applies to each value), or the value from a previous transformation. Insertion topics may be JSON, STRING, INT64, or DOUBLE.

If, when evaluating a topic view, the insertion topic is not found (or the specified key within it is not found), or it is of an unsupported topic type, an optional default value may be inserted, but if no default is specified then no insertion takes place and the value is passed to the derived reference topic unchanged.

The values of the insertion topics are only taken at the point when the source topic is evaluated against the topic view (i.e. when the source topic is updated). Changes to the value of the insertion topic that occur in the interim are not applied to the generated reference topics.

The format of an insert transformation is

insert path [key fromKey] at insertionKey [default defaultValue]

The path is specified in exactly the same way as for the path mapping to clause, except it may not contain expand directives. path directives operate on the path of the source topic, whereas scalar directives operate on the current value as defined previously.

The key clause is optional and can specify a fromKey which is a JSON pointer indicating the data within the insertion topic that is to be inserted. If no key is specified then the whole of the insertion topic value is inserted.

The at clause specifies the insertionKey which is a JSON pointer indicating where to insert the insertion topic value. If a value already exists at the specified key then it is replaced otherwise it is inserted. Insertion can only occur if the parent of the key exists in the value and is of a compatible type. Array pointers may only be used to replace existing entries or append one greater than the last entry. The special pointer value /- may be used to append to the end of an existing array.

The default clause is optional and may be used to specify a string defaultValue to be inserted if the insertion topic does not exist, it is of an unsupported topic type, or the specified key within it could not be found.

The following insert transformation would cause the whole value of the topic named AnyTopic to be inserted into the current value at key /T, assuming that the current value is an object.

insert AnyTopic at /T

The following insert transformation would cause the whole value of the topic named AnyTopic to be inserted into the current value at key /T/MyKey, assuming that an object with the key T exists in the current value.

insert AnyTopic at /T/MyKey

The following insert transformation would cause the whole value of the topic named AnyTopic to be appended to the array at the key T in the current value.

insert AnyTopic at /T/-

The following insert transformation would cause the value at the key name within the topic named AnyTopic to be appended to the array at the key T in the current value.

insert AnyTopic key /name at /T/-

In the above examples no insertion would take place if the insertion value was not found, but a default value can be specified to insert into the current value in this case.

insert AnyTopic at /T/- default "unknown"

The path of the topic to insert from can be built from parts of the source topic path and/or scalar values within the current value. For example:

insert AC/<path(1,1)>/<scalar(/myval)> at /T

Any number of insert transformations may be chained together, but all must occur after any other transformations in the topic view specification.

Options

with properties option

The topic specification of a reference topic is derived from the topic specification of the selected source topic. A reference topic has the same topic type as its source topic (unless the type option is used).

The topic properties of a reference topic are derived from the selected source topic. Some topic properties can be tuned using the with properties option.

The following table describes the behavior for each topic property.

The with properties option begins with the keywords with properties followed by a comma-separated list of topic property keys and values, each separated by a colon.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, and disables both conflation and compression for the reference topics.

map ?a/ to b/<path(1)> with properties CONFLATION:off, COMPRESSION:false

throttle option

The throttle option can be used to constrain the rate at which a reference topic is updated when its source topic is updated. The primary application of a throttle option is to restrict the number of updates sent to reference topic subscribers, reducing network utilization or the processing each subscriber must do. Throttling also restricts the rate at which client sessions can observe changes to reference topic values using the fetch API.

The throttle option has the form :

throttle to x updates every period

Where x is a positive integer, and period is a positive integer followed by a time unit which is one of seconds, minutes, or hours.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most twice every five seconds:

map ?a/ to b/<path(1)> throttle to 2 updates every 5 seconds

To improve readability, the throttle option allows 1 update as an alternative to 1 updates, and every second as an alternative to every 1 seconds (and so on, for other time units).

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most once every hour:

map ?a/ to b/<path(1)> throttle to 1 update every hour

The throttle option is only applied when a source topic is updated more frequently than the configured rate. If a source topic is updated less frequently, updates are passed on unconstrained. If the rate is exceeded, a reference topic will not be updated again until the configured period has expired. At this time, the reference topic will be updated based on the source topic updates that happened in the interim, and a single value will be published. Thus, the throttle option provides topic-scoped conflation.

delay option

The delay option causes a change to a topic view’s source topic to be delayed by a fixed time before it is reflected in reference topics. Topic additions, updates, and removals are all delayed. Delays can range from one second to many days​.

Such a publication delay is a useful way to devalue topic data so it can be given away to non-paying users​.

The delay option has the form :

delay by duration

Where duration is a positive integer followed by a time unit which is one of seconds, minutes, or hours.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but changes to a source topic are delayed by five minutes before they are reflected in the corresponding reference topic.

map ?a/ to b/<path(1)> delay by 5 minutes

Topic views with the delay option specified initially create reference topics in an unpublished state. The topics are published once the delay time has expired.

A topic in the unpublished state prevents a lower priority topic view from creating a reference topic with the same path.

type option

The type option can be used to specify the topic type of the generated reference topic.

If the current source topic’s type can be converted to the target type, a reference topic of the specified type will be created.

For example:

map ?a/ to b/<path(1)> type STRING

The specified type must be one of the supported target types (STRING, INT64, DOUBLE, JSON, TIME_SERIES, or BINARY), case insensitive.

The following table describes the supported conversions from the source topic type to the supported target types. A number indicates a note at the foot of the table describing exactly how the conversion is processed. Where there is no number, no conversion is necessary and the derived source value is simply mapped to the target reference topic as if the type option was not specified. An x indicates that the conversion is not supported.

In all cases the value being processed will be the current value as derived from other mappings within the topic view (e.g. expand) which is not necessarily the value of the source topic.

Conversion notes :

1. STRING to number conversions will only occur if the value of the string can be converted to the target number type. If the string cannot be converted then no reference topic will be created.
2. Primitive types to JSON will result in a JSON topic containing just the scalar value.
3. Conversions to TIME_SERIES will result in a time series topic with an event type matching the source topic. Every update to the source topic will result in a new value being appended to the reference time series topic. It is not possible to convert to a time series topic with a different event type from the source topic.
4. Conversions from number types to STRING will result in a simple string representation of the number in the reference topic value.
5. INT64 to DOUBLE conversions perform a simple conversion. For example, 123 becomes 123.0.
6. DOUBLE to INT64 conversions perform rounding to the nearest integer value. For example 12.51 becomes 13.
7. JSON to primitive type conversions only occur if the JSON value is a scalar which can be converted to the target type. If the JSON value is a structure or cannot be converted then no conversion takes place and no reference topic will be created.
8. The conversion of TIME_SERIES to other types follows the same rules as for conversion from the source topic type that matches the source TIME_SERIES topic’s event value type. So if the TIME_SERIES event type is double, the conversion rules from source topic type DOUBLE to the target type will apply. Each value appended to the source TIME_SERIES topic will result in an update to the reference topic. If a failure to convert occurs at any point then the reference topic would be removed and only recreated if a value is appended that can be converted.
9. BINARY to TIME_SERIES, and vice-versa is supported in the same way as for other TIME_SERIES conversions.

separator option

Topic views can use the scalar and expand directives in path mappings to extract text from the source value. By default, any / characters in the text are interpreted as path separators and will introduce extra levels in reference topic paths. If this is undesirable, the separator option can be used to replace any / characters.

For example, consider the topic view:

map ?a/path/ to b/<scalar(/markets/name)>

Suppose the value at /markets/name in the source topic is "USD/Sterling". The / character is treated as a path separator, so the reference topic is created at b/USD/Sterling.

The separator option enables you to prevent this behavior by specifying a replacement string to be used instead of a path separator in the names of generated reference topics.

In the above example, by adding a separator option to the topic view:

map ?a/path/ to b/<scalar(/markets/name)> separator '-'

The reference topic is now created at b/USD-Sterling.

The separator value is a string, so it can be longer than a single character. It can contain / characters which will be interpreted as path separators. It must not contain empty path segments, that is //.

as option

By default, a reference topic’s value is a copy of the source topic value, or part of the source value produced by an expand path mapping directive and/or modified by transformations. The as option can be applied to extract part of the resulting value (the latest value in the case of TIME_SERIES topics).

The as option begins with the keyword as and is followed by a value directive. A value directive is delimited by angle brackets (<, >), and consists of the value keyword and a single JSON pointer parameter. The JSON pointer selects the part of the current value to copy.

For example, given a source topic with a value of

{
    "account" : "1234",
    "balance" : { "amount" : 12.57, "currency" : "USD" }
}
		

Using a topic view with the specification :

map ?accounts/ to balances/<scalar(/account)> as <value(/balance)>

A reference topic at the path balances/1234 with the following value will be created :

{
    "amount"   : 12.57,
    "currency" : "USD"
}
		

preserve topics option

The default behavior for a topic view is that only the reference topics that can be derived from the current value of the source topic are maintained. This applies to topic views using directives that derive the path of the reference topic(s) from a value within the source topic (e.g. scalar or expand).

However, in some applications it may be desirable to keep the reference topics that are created over time.

For example, suppose you have a data feed coming from some external source that provides you with foreign exchange rates, updating a topic (called Rate) with the following structure:

{"currency":"GBP/USD", "rate":1.45}

And we have a topic view that has the specification:

map Rate to <scalar(/currency)> as <value(/rate)>

In this case, you would get a reference topic at the path GBP/USD and a value of 1.45.

However, if the Rate topic is then updated to the following:

{"currency":"GBP/EUR, "rate":1.16}

The GBP/USD topic will be removed, and a new GBP/EUR topic created with a value of 1.16. In this application, the preferred behavior might be to preserve the original reference topic, so that a record of all rates that have been passed from the feed remain. This is where preserve topics becomes useful, so we change the topic view specification to:

map Rate to <scalar(/currency)> as <value(/rate)> preserve topics

And now after the above update, both the GBP/USD and GBP/EUR topics would remain.

Updates for further new currencies would result in new reference topics. Updates for a currency that already has a reference topic would simply update the reference topic value.

Reference topics that are created when the preserve topics option is used remain until the source topic is removed or the topic view is removed.

The preserve topics option applies only to topic views that contain path mappings that change the path of the target reference topic(s), so those that contain scalar or expand directives. For other topic views, the clause would be ignored.

Escaping and quoting special characters

Each part of a topic view expression has characters with special significance. Many clauses are delimited by white space. Directives are delimited by the < and > characters, and parameters may be terminated by , or ). Other delimiters, such as : may be used in other places.

Sometimes a topic view must refer to or generate topics with paths that contain special characters, or use a JSON pointer containing special characters. The escape sequence \x can be used to literally insert any character x, with a one exception: \/ cannot be used in path fragments since the path delimiter / is always significant.

Here is an example topic view expression containing escape sequences. It maps the topic at the path A topic to a reference topic at the path Another topic.

map A\ topic to Another\ topic

Here is an example with a scalar directive that uses the JSON pointer /x()/y to extract the target path from the source value. The ) character in the JSON pointer must be escaped so it is not treated as the end of the parameter list.

map ?a/ to <scalar(/x(\)/y)>

To insert \, the escape sequence \\ must be used.

There is no need to escape white space in JSON pointers directive parameters. However, white space is significant. For example, the following expressions have different topic value mapping clauses since the JSON pointer in the second expression is /x ; that is, it has a trailing space:

map a to b as <value(/x)>
map a to b as <value(/x )>
		

Instead of using escape sequences, white space characters can be included in various clauses using quotes. A clause is quoted by wrapping it in single quote (') or double quote (") characters. For example:

map "a topic" to "another topic"

Within a quoted clause, quotes of the same type must be escaped:

map 'alice\'s topic' to 'bob\'s topic'

Dealing with topic path conflicts

If you create a topic view which tries to generate a reference topic at the same path as an existing topic, the result is a topic path conflict.

Reference topics have a lower priority than normal topics created through the API, including replicas of normal topics created by topic replication or fan-out. A reference topic will only be created if no non-reference topic is already bound to its derived topic path.

Topic views have a precedence based on order of creation. If two topic views define mappings to the same topic path, the earliest-created topic view will generate the reference topic. If a topic view is updated, it retains its original precedence.

Remote topic views

A remote topic view is one that maps its source topics from a different Diffusion Cloud server (or server cluster).

This is indicated using the from clause:

map ?a/ from server1 to b/<path(1)>

The server name server1 in this example refers to the name of a remote server created using the remote servers feature. The remote server component maintains a connection between the secondary server (the one where the topic views reside) and a primary server (the one where the source topics reside).

Upon establishing a successful connection between secondary and primary servers a remote topic view will generate reference topics locally based upon the topics selected by the topic view’s selector at the primary server.

It is important to note that the selector only refers to topics that match it at the primary server and not on the local (secondary) server, and there is no reason why there could not be a source topic at the primary server that has the same path as an entirely different topic on the local server.

More than one topic view can specify the same remote server.

If a remote server completely loses its connection to the primary server, all reference topics created by topic views that use the remote server will be removed and will only be recreated when the connection is re-established. It is not necessary for the named remote server component to exist before creating the topic view, and if it does not, the topic view will simply remain dormant until the remote server is created and a successful connection to the primary server is established. Similarly, if a remote server that is in use by remote topic views is removed then all of the reference topics generated by the topic views will be removed and the topic views will become dormant until the named remote server is created again or the topic views are changed to name a different remote server.

The rules of precedence for remote topic views are the same as for other topic views. If the remote server for a remote topic view does not exist or does not have an established connection then the remote topic view is not evaluated (i.e. it is as if the source topics for the view did not exist), but if the remote server later connects then the topic view will be evaluated and rules of precedence will determine whether reference topics will replace those created by other topic views.

In a Diffusion Cloud cluster, both topic views and remote server definitions are automatically distributed across the cluster. Each member of a secondary cluster will automatically be connected to a primary server and produce the same reference topics from the remote topic views.

All topic view capabilities can be applied to a remote topic view.

Secondary to primary remote server connections

A remote server that initiates the connection from the secondary server only makes a physical connection when it is in use, therefore the first topic view that specifies a remote server will cause it to establish a connection. Similarly, if the last topic view that uses a remote server is removed then the connection will be closed.

Secondary initiated remote servers have a reconnection policy and reference topic will remain whilst attempting to reconnect. But if reconnection fails, reference topics will be removed and the remote server will go into retry mode.

Primary to secondary remote server connections

A remote server that initiates the connection from a primary server will always maintain a connection whilst the servers are running. For this type of connection there is no reconnection policy so as soon as the connection is lost any reference topics generated from topic views that use it will be removed. When such a connection is lost the primary server will retry it and reference topics will only be generated again when the connection has been re-established.

Topic view persistence and replication

Reference topics are neither replicated nor persisted. They are created and removed based on their source topics. However, topic views themselves are replicated and persisted. A server that restarts will restore topic views during recovery. Each topic view will then generate reference topics based on the source topics that have been recovered.

The server records all changes to topic views in a persistent store. Topic views are restored if the server is restarted.

If a server belongs to a cluster, topic views will be replicated to each server in the cluster. Topic views are evaluated locally within a server. Replicated topic views that select non-replicated source topics can create different reference topics on each server in the cluster.

Access control

The following access control restrictions are applied:

• To list the topic views, a session needs the READ_TOPIC_VIEWS global permission.
• To create, replace, or remove a topic view, a session needs the MODIFY_TOPIC_VIEWS global permission and SELECT_TOPIC permission for the path prefix of the source topic selector.
• Each topic view records the principal and security roles of the session that created it as the topic view security context. When a topic view is evaluated, this security context is used to constrain the creation of reference topics. A reference topic will only be created if the security context has READ_TOPIC permission for the source topic path, and MODIFY_TOPIC permission for the reference topic path. The topic view security context is copied from the creating session at the time the topic view is created or replaced, and is persisted with the topic view. The topic view security context is not updated if the roles associated with the session are changed.