Read BeanMetadata changes from another service

Metadata
1

We have a requirement to modify an existing metadatabean based on metadatabeans created at another service. The goal is to add a field to an existing metadatabean every time a new metadatabean that meet certain criteria (i.e extends a known metadatabean) is created in another service. It’s my understanding that there are 2 ways to do it, either making rest calls to the first service at specified intervals, or listening to a kafka topic (or the eventservice in use) for metadata changes.

What is the recommended way (best practice) to do something like this? I assume is going to be the kafka topic since is more real time. Where can I find more information/examples of how to implements something like this since there is no much information in the documentation.

2

To give some more context, we will have the ability to edit live BeanMetadata using the metadata-tool-editor. We expect that the another service will have its BeanMetadata edited in real time, and then the service that needs to know about it needs to receive updates to update its own BeanMetadata.

If there is another approach for this kind of use case, let us know. If you need more information, we can go into a bit more detail as well.

3

Hello @jmartin :wave:
Going off what you have written, my first instinct would be to subscribe to BeanMetadata DataChangeEvents from the service whose BeanMetadata you need to know about. You can check if they meet the criteria by leveraging the filters on a DataChangeEventQuery. Then, you can upsert a new edition of BeanMetadata every time you receive a DataChangeEvent.

Couple things you need to know:

  1. The default Kafka topic name format for a generated Data Service is
<servicename>.datachangeevents.<type>

So if your service was called entity-service, you the topic you would subscribe to is:

entityservice.datachangeevents.beanmetadata
  1. To do this, I would generate a Functional Service, and configure it to subscribe to that aforementioned Kafka topic:
$ yo @leaf/functional-service

... (series of prompts)

? Enter the REST URI used for REST clients: http://entity-service/rest

Then, in the values/values.yaml, add the following under kafka:

kafka:
  ...
  beanMetadataTopic: entityservice.datachangeevents.beanmetadata

Out of the box, Functional Services wire up a DataChangeEventManager and have a method that gets executed on each DataChangeEvent. In this method, you can leverage a MetadataServiceRestClient to add the fields to the existing BeanMetadata.

IMetadataService metadataServiceRestClient = MetadataServiceRestClient.newBuilder(webTarget).build()

You should be able to use the same webTarget that the generated DataObjectServiceRestClient uses.

void onDataChangeEvent(final DataChangeEvent<DataObject> dataChangeEvent) {

    ....

    metadataServiceRestClient.upsertBeanMetadata(modifiedBeans, Context.makeSystemContext());

    ...

}
  1. In the setupEventHandler() method, a DataChangeEventQuery should already be generated for you:
DataChangeEventQuery eventQuery = .... ;

Here you can tailor the Query to include whatever criteria you wish by adding filters.

2 Likes
4

Are there any downsides to just implementing the DataChangeEvent subscription in a data-service? Or is a functional-service the recommended way to go? Suppose we wanted the service itself to listen for changes on entity-service and update its beanmetadata itself (as opposed to doing rest calls from a functional service).

5

I think the biggest consideration for me would be how many BeanMetadata events are you expecting? If it’s many such that you’ll need to horizontally scale the Data Service just to support listening to and processing BeanMetadata events, I would consider using a Functional Service so you can scale the data service and the listener separately.

If, like I would expect, you’re not changing BeanMetadata that often, I think a DataService listener is definitely fine. The only downside there is that the generated Data Services don’t come with a nice listener callback like the Functional Services do, but it’d be easy enough to copy that over from a generated Functional Service into your Data Service. That way you can basically still do what @jacob.rexroad suggested, which I definitely agree with, and if you want to copy that logic into your Data Service, that should be easy enough.

3 Likes