This feature allows a client session to manage topic views. More...

Data Structures
struct	diffusion_create_topic_view_params_s
	Structure describing a request to create a topic view. More...

struct	diffusion_topic_views_list_params_s
	Structure describing a request to list topic views. More...

struct	diffusion_remove_topic_view_params_s
	Structure describing a request to remove a topic view. More...

Typedefs
typedef struct DIFFUSION_TOPIC_VIEW_T	DIFFUSION_TOPIC_VIEW_T
	Opaque topic view struct.

typedef int(*	on_topic_view_created_cb )(const DIFFUSION_TOPIC_VIEW_T topic_view, void context)
	Callback when a topic view has successfully been created. More...

typedef int(*	on_topic_view_response_cb )(void *context)
	Callback when a response is received from the server regarding a topic view operation. More...

typedef int(*	on_topic_views_list_cb )(const LIST_T topic_views, void context)
	Callback when a response is received from the server regarding a `diffusion_topic_views_list_topic_views` operation. More...

typedef struct diffusion_create_topic_view_params_s	DIFFUSION_CREATE_TOPIC_VIEW_PARAMS_T
	Structure describing a request to create a topic view.

typedef struct diffusion_topic_views_list_params_s	DIFFUSION_TOPIC_VIEWS_LIST_PARAMS_T
	Structure describing a request to list topic views.

typedef struct diffusion_remove_topic_view_params_s	DIFFUSION_REMOVE_TOPIC_VIEW_PARAMS_T
	Structure describing a request to remove a topic view.

Functions
char *	diffusion_topic_view_get_name (const DIFFUSION_TOPIC_VIEW_T *topic_view)
	Return a memory allocated copy of the topic view's name. More...

char *	diffusion_topic_view_get_specification (const DIFFUSION_TOPIC_VIEW_T *topic_view)
	Return a memory allocated copy of the topic view's specification. More...

SET_T *	diffusion_topic_view_get_roles (const DIFFUSION_TOPIC_VIEW_T *topic_view)
	Return a memory allocated `SET_T` of the topic view's roles. More...

DIFFUSION_TOPIC_VIEW_T *	diffusion_topic_view_dup (const DIFFUSION_TOPIC_VIEW_T *topic_view)
	Returns a memory allocated copy of a `DIFFUSION_TOPIC_VIEW_T` More...

void	diffusion_topic_view_free (DIFFUSION_TOPIC_VIEW_T *topic_view)
	Free a memory allocated `DIFFUSION_TOPIC_VIEW_T` More...

bool	diffusion_topic_views_create_topic_view (SESSION_T session, const DIFFUSION_CREATE_TOPIC_VIEW_PARAMS_T params, DIFFUSION_API_ERROR api_error)
	Create a new named topic view. More...

bool	diffusion_topic_views_list_topic_views (SESSION_T session, const DIFFUSION_TOPIC_VIEWS_LIST_PARAMS_T params, DIFFUSION_API_ERROR api_error)
	List all the topic views that have been created. More...

bool	diffusion_topic_views_remove_topic_view (SESSION_T session, const DIFFUSION_REMOVE_TOPIC_VIEW_PARAMS_T params, DIFFUSION_API_ERROR api_error)
	Remove a named topic view if it exists. More...

Detailed Description

This feature allows a client session to manage topic views.

A topic view maps one part of the server's topic tree to another. It dynamically creates a set of reference topics from a set of source topics, based on a declarative topic view specification. The capabilities of topic views range from simple mirroring of topics within the topic tree to advanced capabilities that include publication of partial values and throttling the rate of publication.

Each reference topic has a single source topic and has the same topic type as its source topic. Reference topics are read-only (they cannot be updated), nor can they be created or removed directly. Otherwise, they behave just like standard topics. A client session can subscribe to a reference topic, and can fetch the reference topic's current value if it has one.

The source topics of a topic view are defined by a topic selector. A reference topic is created for each source topic, according to the topic view. If a source topic is removed, reference topics that are derived from it will automatically be removed. If a topic is added that matches the source topic selector of a topic view, a corresponding reference topic will be created. Removing a topic view will remove all of its reference topics.

Topic view specifications

The following is a simple topic view specification that mirrors all topics below the path a to reference topics below the path b.

map ?a// to b/<path(1)>

A topic view with this specification will map a source topic at the path a/x/y/z to a reference topic at the path b/x/y/z. The specification is simple, so the reference topic will exactly mirror the source topic.

A general topic view specification comprises several parts:

The source topic clause identifies the source topics.
The path mapping clause determines how reference topic paths are derived from the source topic paths.
The optional topic property mapping clause determines how reference topic properties are derived from source topic properties.
The optional value mapping clause determines how reference topic values are derived from source topic values.
The optional throttle clause constrains the rate at which each reference topic is updated when its source topic is updated.

Topic view specification

Topic view specifications are described using a domain specific language (DSL). The DSL has two required clauses: the source clause and the path mapping clause. The source clause identifies which topics should be the source topics for the view. The path mapping clause describes how the paths of reference topics should be derived. A specification begins with the source clause, followed by the path mapping clause.

The source clause begins with the map keyword and is followed by a topic selector. These topic selectors follow the same as other topic selectors.

The path mapping clause begins with the to keyword and is followed by a path mapping template. A path mapping template is the topic path with embedded directives. Directives are evaluated when creating the topic reference and substituted into the topic path. Directives are surrounded by angle brackets, they contain the name of the directive and the parameters. The parameter list begins with an opening round bracket, ends with a closing round bracket and parameters in the list should be separated by colons.

Topic selectors and path mapping templates can be quoted or unquoted. They are quoted using the single quote mark. To include whitespace, single quotes or literal opening angle brackets they must be quoted. In quoted selectors and templates single quotes, literal opening angle brackets and backslashes must be escaped with a single backslash. In templates the opening angle bracket should be unescaped when beginning a directive. Characters in unquoted selectors and templates can't be escaped.

Any whitespace can be used to separate keywords, statements and clauses.

The following example creates reference topics with 'prefix/' prepended to the paths of the source topics.

map ?a/path/ to prefix/<path(0)>

Source topic clause

The source topic clause begins with the map keyword and is followed by a topic selector. These topic selectors follow the same parsing rules as other topic selectors.

When evaluating a topic view, topics in the topic tree that match the source topic selector are considered, with the following exceptions:

Topics created through the Publisher API;
TOPIC_TYPE_ROUTING topics.

Both TOPIC_TYPE_SLAVE and reference topics are valid source topics. In particular, chaining of topic views is supported; that is, a reference topic created by one topic view can be the source topic of another topic view. Additionally, a reference topic can be the master topic of a slave topic, or the source topic of a routing topic subscription.

Prefer topic views to slave topics. Individual topics can be mirrored by creating a slave topic, but maintaining slave topics for a branch of the topic tree quickly becomes tedious. A topic view will maintain such a branch automatically, and provides more sophisticated mapping options.

Path mapping clause

The path of a reference topic is derived from the source topic according to the topic view path mapping. The path mapping allows the source topic path and the value of the source topic to determine the path of the reference topic.

A path mapping clause begins with the to keyword and is followed by a path mapping template. A path mapping template is a topic path with embedded directives. Directives are evaluated when creating the topic reference and substituted into the topic path. Directives are delimited by angle brackets (<, >) and consist of the name of the directive and a list of parameters. The parameter list is comma-separated and surrounded by parentheses ((, )).

Two path mapping directives are supported:

Source path directives

Source path directives extract a portion of the source path and are parameterized by the index of the start part of the source path and the number of parts to include. The number of parts parameter is optional – if it is missing, the selection extends to the end of the source path. The syntax is <path(start, number)>, or <path(start)> when the number of parts parameter is omitted.

For example, given the source path a/b/c/d, the source path directive <path(1, 2)> is mapped to the reference topic path b/c, and the source path directive <path(2)> is mapped to the reference topic path c/d.

Source value directives

Source value directives are only applied to TOPIC_TYPE_JSON source topics; if the path mapping contains a source value directive, non-JSON topics matching the source topic selector are ignored. Source value directives use the keyword scalar and are parameterized by a single JSON pointer that extracts a scalar value from the source value. A scalar value is a string, a number, true, false, or null, that is, anything other than an array or a object. If the JSON pointer does not refer to a scalar value in the source value, no reference topic will be created. This includes the cases where the JSON pointer refers to an array or an object), or when no part of the source value is selected.

Deriving the reference topic paths from part of the source topic value effectively creates a secondary index on the value. For source value directives to work efficiently, the selected scalar values should be relatively stable. If an update to the source topic changes the selected scalar value, the corresponding reference topic will be removed and a new reference topic will be created.

For example, given a source value of

{
  "account" : "1234",
  "balance" : { "amount" : 12.57, "currency" : "USD" }
}

and the source value directive currency/<scalar(/balance/currency)>/account/<scalar(/account)>, the reference topic path will be currency/USD/account/1234.

If the extracted value is a string, it is copied literally to the reference topic path. A value that contains path separators (/) will create a reference topic path with more levels than the path mapping template.

An extracted value of null will be copied to the reference topic path as the string "null".

Topic property mapping clause

The topic specification of a reference topic is derived from the topic specification of the source topics. A reference topic has the same topic type as its source topic. The topic properties are mapped according to the following table:

Source topic property	Reference topic specification default	Can be set by topic property mapping?	Notes
`DIFFUSION_COMPRESSION`	Copied from source topic specification	Yes
`DIFFUSION_CONFLATION`	Copied from source topic specification	Yes
`DIFFUSION_DONT_RETAIN_VALUE`	Copied from source topic specification	Yes
`DIFFUSION_OWNER`	Not set	No
`DIFFUSION_PERSISTENT`	Not set	No	Reference topics are not persisted. Topic views are persisted, so a reference topic will be recreated on server restart if its source is persistent.
`DIFFUSION_PUBLISH_VALUES_ONLY`	Copied from source topic specification	Yes
`DIFFUSION_REMOVAL`	Not set	No	Reference topics cannot be removed directly.
`DIFFUSION_SCHEMA`	Copied from source topic specification	No	A `TOPIC_TYPE_RECORDV2` reference topic has the same schema as its source topic.
`DIFFUSION_SLAVE_MASTER_TOPIC`	Not set	No	If a reference topic has a slave topic as its source topic, it indirectly references the slave's master topic.
`DIFFUSION_TIDY_ON_UNSUBSCRIBE`	Copied from source topic specification	Yes
`DIFFUSION_TIME_SERIES_EVENT_VALUE_TYPE`	Copied from source topic specification	No	A `TOPIC_TYPE_TIME_SERIES` reference topic has the same value type as its source topic.
`DIFFUSION_TIME_SERIES_RETAINED_RANGE`	Copied from source topic specification	Yes, with restrictions	A topic property mapping cannot increase the time series retained range by overriding the `DIFFUSION_TIME_SERIES_RETAINED_RANGE` property. The retained range of a reference time series topic will be constrained to be no greater than that of its source topic.
`DIFFUSION_TIME_SERIES_SUBSCRIPTION_RANGE`	Copied from source topic specification	Yes
`DIFFUSION_VALIDATE_VALUES`	Not set	No	A reference topic reflects updates to its source topic. It cannot reject updates.

A topic property mapping clause is optional. It begins with properties and consists of a comma-separated list of topic property keys and values, each separated by a comma. For example, the following topic view specification maps all topics below the path a to reference topics below the path b, and disables both conflation and compression for the reference topics.

`map ?a// to b/<path(1)> with properties 'CONFLATION':'off', 'COMPRESSION':'false'`

Topic value mapping

The value of a reference topic is derived from the source topic according to the topic view value mapping. By default, a reference topic's value is the same as its source topic.

For JSON source topics, the optional topic value mapping clause can be used to extract part of the source value. A topic value mapping begins the keyword as and is followed by a value directive. A value directive is delimited by angle brackets (<, >), and consists of the value keywords and a single JSON pointer parameter. The JSON pointer selects the part of the source value to copy.

For example, given a source value of

{
    "account" : "1234",
    "balance" : { "amount" : 12.57, "currency" : "USD" }
}

and the value mapping clause as <value(/balance)>, the reference topic value will be

{
    "amount" : 12.57,
    "currency" : "USD"
}

Topic value mappings are often used with path value mappings. For example:

map ?accounts// to balances/<scalar(/account)> as <value(/balance)>;

Throttle clause

The optional throttle clause can be used to constrain the rate at which a reference topic is updated when its source topic is updated. The primary application of a throttle clause is to restrict the number of updates sent to reference topic subscribers, reducing network utilization or the processing each subscriber must do. Throttling also restricts the rate at which client sessions can observe changes to reference topic values using the fetch API.

The throttle clause has the form throttle to <em>X</em> updates every <em>period</em>, where X is a positive integer, and period is a positive integer followed by a time unit which is one of seconds, minutes, or hours.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most twice every five seconds:

map ?a// to b/<path(1)> throttle to 2 updates every 5 seconds

To improve readability, the throttling clause allows 1 update as an alternative to 1 updates, and every second as an alternative to every 1 seconds (and so on, for other time units). For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most once every hour:

map ?a// to b/<path(1)> throttle to 1 update every minute

The throttle clause is only applied when a source topic is updated more frequently than the configured rate. If a source topic is updated less frequently, updates are passed on unconstrained. If the rate is exceeded, a reference topic will not be updated again until the configured period has expired. At this time, the reference topic will be updated based on the source topic updates that happened in the interim, and a single value will be published. Thus, a throttle clause provides topic-scoped conflation.

The throttle clause is ignored for time series topics because time series updates do not support efficient conflation. Updates to source time series topics are passed on immediately to the corresponding reference topics, regardless of any throttle clause.

Quoting and white space

Topic selectors and path mapping templates can be quoted or unquoted. They are quoted using the single quote mark. To include whitespace, single quotes or literal opening angle brackets they must be quoted. In quoted selectors and templates single quotes, literal opening angle brackets and backslashes must be escaped with a single backslash. In templates the opening angle bracket should be unescaped when beginning a directive. Characters in unquoted selectors and templates can't be escaped.

Any whitespace can be used to separate keywords, statements and clauses.

Dealing with topic path conflicts

Reference topics have a lower priority than normal topics created through the API, including replicas of normal topics created by topic replication or fan-out. A reference topic will only be created if no topic or reference topic is already bound to its derived topic path.

Topic views have a precedence based on order of creation. If two topic views define mappings the same topic path, the earliest-created topic view will create a reference topic. If a topic view is updated, it retains its original precedence.

Topic view persistence and replication

Reference topics are neither replicated nor persisted. They are created and removed based on their source topics. However, topic views are replicated and persisted. A server that restarts will restore topic views during recovery. Each topic view will then create reference topics based on the source topics that have been recovered.

The server records all changes to topic views in a persistent store. Topic views are restored if the server is started.

If a server belongs to a cluster, topic views will be replicated to each server in the cluster. Topic views are evaluated locally within a server. Replicated topic views that select non-replicated source topics can create different reference topics on each server in the cluster.

Access control

The following access control restrictions are applied:

To list the topic views, a session needs the READ_TOPIC_VIEWS global permission.
To create, replace, or remove a topic view, a session needs the MODIFY_TOPIC_VIEWS global permission and SELECT_TOPIC permission for the path prefix of the source topic selector.
Each topic view records the principal and security roles of the session that created it as the topic view security context. When a topic view is evaluated, this security context is used to constrain the creation of reference topics. A reference topic will only be created if the security context has READ_TOPIC permission for the source topic path, and MODIFY_TOPIC permission for the reference topic path. The topic view security context is copied from the creating session at the time the topic view is created or replaced, and is persisted with the topic view. The topic view security context is not updated if the roles associated with the session are changed.

Typedef Documentation

typedef int(* on_topic_view_created_cb)(const DIFFUSION_TOPIC_VIEW_T *topic_view, void *context)

Callback when a topic view has successfully been created.

Parameters

topic_view	The created topic view.
context	User supplied context.

Returns: HANDLERS_SUCCESS or HANDLER_FAILURE.

typedef int(* on_topic_view_response_cb)(void *context)

Callback when a response is received from the server regarding a topic view operation.

Parameters

context User supplied context.

Returns: HANDLERS_SUCCESS or HANDLER_FAILURE.

typedef int(* on_topic_views_list_cb)(const LIST_T *topic_views, void *context)

Callback when a response is received from the server regarding a diffusion_topic_views_list_topic_views operation.

Parameters

topic_views	A `LIST_T` containing `DIFFUSION_TOPIC_VIEW_T`s
context	User supplied context.

Returns: HANDLERS_SUCCESS or HANDLER_FAILURE.

Data Structures

Typedefs

Functions

Detailed Description

Topic view specifications

Topic view specification

Source topic clause

Path mapping clause

Topic property mapping clause

Topic value mapping

Throttle clause

Quoting and white space

Dealing with topic path conflicts

Topic view persistence and replication

Access control

Typedef Documentation