Rules editor app not seeing data service

Data Services
1

Hello,

I’m working to make several data services discoverable by the rules editor tool. I have been successful with one service so far, but am failing with another. I have followed the necessary steps in the leaf documentation to set this up. My data service deploys fine with these changes but is not listed in the ‘services’ drop down in the rules editor.

I do have this in the rules-service logs, where it seems to find my data service (openSkyService)

12:52:26.509 [Thread-1] e[32mINFO e[m o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=RulesEditorToolServiceaa951582-4b0f-4be2-ac82-a98be478d8f7, groupId=RulesEditorToolService-bc176144-03c7-4826-8143-2739bb0cecb1] Successfully joined group with generation Generation{generationId=1, memberId='RulesEditorToolServiceaa951582-4b0f-4be2-ac82-a98be478d8f7-bf0109a6-79e5-4e74-9d19-c493b281bdb8', protocol='range'}
12:52:26.511 [Thread-1] e[32mINFO e[m o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=RulesEditorToolServiceaa951582-4b0f-4be2-ac82-a98be478d8f7, groupId=RulesEditorToolService-bc176144-03c7-4826-8143-2739bb0cecb1] Finished assignment for group at generation 1: {RulesEditorToolServiceaa951582-4b0f-4be2-ac82-a98be478d8f7-bf0109a6-79e5-4e74-9d19-c493b281bdb8=Assignment(partitions=[targetdefinitionservice.datachangeevents.datachangeeventquery-0, litetrackledgerservice.datachangeevents.datachangeeventquery-0, dashboardservice.datachangeevents.editorlayout-0, targetdefinitionservice.datachangeevents.dataobject-0, serviceregistry.datachangeevents.datachangeeventquery-0, litetrackledgerservice.datachangeevents.lookuplist-0, dashboardservice.datachangeevents.beanmetadata-0, testledgerservice.datachangeevents.lookuplist-0, dashboardservice.datachangeevents.metadatatemplate-0, dashboardservice.datachangeevents.datachangeeventquery-0, litetrackservice.datachangeevents.other-0, targetdefinitionservice.datachangeevents.lookuplist-0, openskyservice.datachangeevents.listingview-0, litetrackledgerservice.datachangeevents.dataobjectquery-0, litetrackledgerservice.datachangeevents.editorlayout-0, openskyservice.datachangeevents.lookuplist-0, litetrackledgerservice.datachangeevents.listingview-0, ruleseditortoolservice.datachangeevents.metadatatemplate-0, falconoverlayservice.datachangeevents.listingview-0, serviceregistry.datachangeevents.lookuplist-0, litetrackservice.datachangeevents.dataobject-0, openskyservice.datachangeevents.dataobjectquery-0, litetrackservice.datachangeevents.listingview-0, dataregistryservice.datachangeevents.metadatatemplate-0, openskyservice.datachangeevents.dataobject-0, ruleseditortoolservice.datachangeevents.lookuplist-0, testledgerservice.datachangeevents.datachangeeventquery-0, falconoverlayservice.datachangeevents.other-0, testledgerservice.datachangeevents.editorlayout-0, testledgerservice.datachangeevents.dataobjectquery-0, falconoverlayservice.datachangeevents.dataobjectquery-0, falconoverlayservice.datachangeevents.beanmetadata-0, dataregistryservice.datachangeevents.lookuplist-0, dataservice.datachangeevents.dataobject-0, litetrackservice.datachangeevents.editorlayout-0, ruleseditortoolservice.datachangeevents.beanmetadata-0, targetdefinitionservice.datachangeevents.other-0, openskyservice.datachangeevents.other-0, falconoverlayservice.datachangeevents.dataobject-0, dataregistryservice.datachangeevents.beanmetadata-0, dashboardservice.datachangeevents.dataobject-0, testledgerservice.datachangeevents.metadatatemplate-0, testledgerservice.datachangeevents.listingview-0, litetrackledgerservice.datachangeevents.metadatatemplate-0, dataregistryservice.datachangeevents.dataobjectquery-0, testledgerservice.datachangeevents.other-0, ruleseditortoolservice.datachangeevents.editorlayout-0, openskyservice.datachangeevents.editorlayout-0, serviceregistry.datachangeevents.beanmetadata-0, dataregistryservice.datachangeevents.editorlayout-0, falconoverlayservice.datachangeevents.datachangeeventquery-0, dataregistryservice.datachangeevents.datachangeeventquery-0, targetdefinitionservice.datachangeevents.editorlayout-0, openskyservice.datachangeevents.datachangeeventquery-0, dataregistryservice.datachangeevents.other-0, serviceregistry.datachangeevents.listingview-0, openskyservice.datachangeevents.metadatatemplate-0, litetrackservice.datachangeevents.beanmetadata-0, serviceregistry.datachangeevents.dataobjectquery-0, litetrackservice.datachangeevents.metadatatemplate-0, ruleseditortoolservice.datachangeevents.dataobject-0, dataregistryservice.datachangeevents.listingview-0, ruleseditortoolservice.datachangeevents.dataobjectquery-0, dataregistryservice.datachangeevents.dataobject-0, serviceregistry.datachangeevents.metadatatemplate-0, serviceregistry.datachangeevents.dataobject-0, falconoverlayservice.datachangeevents.editorlayout-0, dashboardservice.datachangeevents.dataobjectquery-0, openskyservice.datachangeevents.beanmetadata-0, litetrackservice.datachangeevents.dataobjectquery-0, serviceregistry.datachangeevents.editorlayout-0, serviceregistry.datachangeevents.other-0, litetrackservice.datachangeevents.lookuplist-0, litetrackledgerservice.datachangeevents.other-0, falconoverlayservice.datachangeevents.lookuplist-0, dashboardservice.datachangeevents.other-0, targetdefinitionservice.datachangeevents.dataobjectquery-0, targetdefinitionservice.datachangeevents.metadatatemplate-0, dashboardservice.datachangeevents.listingview-0, litetrackservice.datachangeevents.datachangeeventquery-0, falconoverlayservice.datachangeevents.metadatatemplate-0, targetdefinitionservice.datachangeevents.listingview-0, targetdefinitionservice.datachangeevents.beanmetadata-0, dashboardservice.datachangeevents.lookuplist-0])}

Then I have this error repeatedly

Sep 27, 2022 1:29:37 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
SEVERE: Failed to export spans. Server is UNAVAILABLE. Make sure your collector is running and reachable from this network. Full error message:UNAVAILABLE: Unable to resolve host jaeger-collector.observability
13:29:37.225 [pool-4-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceRegistryScheduler - message: Exception encountered in service registration
javax.ws.rs.ProcessingException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'RulesEditorToolService': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (org.jboss.resteasy.specimpl.AbstractBuiltResponse$InputStreamWrapper); line: 1, column: 24]
	at org.jboss.resteasy.client.jaxrs.internal.ClientResponse.readFrom(ClientResponse.java:254) ~[resteasy-client-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.specimpl.BuiltResponse.readEntity(BuiltResponse.java:90) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.specimpl.AbstractBuiltResponse.readEntity(AbstractBuiltResponse.java:270) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at com.leidos.leaf.platform.servicediscovery.rest.ServiceRegistryRestClient.getDataObjectOrThrow(ServiceRegistryRestClient.java:103) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.platform.servicediscovery.rest.ServiceRegistryRestClient.registerService(ServiceRegistryRestClient.java:68) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.platform.servicediscovery.ServiceRegistryExtensible.lambda$new$0(ServiceRegistryExtensible.java:68) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.hooked.IMethod.invoke(IMethod.java:125) ~[framework-services-core-3.7.1.jar:?]
	at com.leidos.leaf.hooked.default_.hooks.TracerHook.lambda$hookInvoker$2(TracerHook.java:163) ~[framework-services-core-3.7.1.jar:?]
	at com.leidos.leaf.hooked.BaseExtensibleService.invoke(BaseExtensibleService.java:129) ~[framework-services-core-3.7.1.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceRegistryExtensible.registerService(ServiceRegistryExtensible.java:76) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.platform.servicediscovery.ServiceRegistryScheduler.lambda$getRegisterAction$0(ServiceRegistryScheduler.java:124) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'RulesEditorToolService': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (org.jboss.resteasy.specimpl.AbstractBuiltResponse$InputStreamWrapper); line: 1, column: 24]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:745) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3635) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2734) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:902) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:794) ~[jackson-core-2.13.2.jar:2.13.2]
	at org.jboss.resteasy.plugins.providers.jackson.ResteasyJackson2Provider.readFrom(ResteasyJackson2Provider.java:181) ~[resteasy-jackson2-provider-4.6.1.Final.jar:4.6.1.Final]
	at org.jboss.resteasy.core.interception.jaxrs.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:101) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.core.interception.jaxrs.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:80) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.client.jaxrs.internal.ClientResponse.readFrom(ClientResponse.java:217) ~[resteasy-client-5.0.2.Final.jar:5.0.2.Final]
	... 16 more

Any ideas on what could cause this error?

2

Hi @jacob.i.bostwick,

Is the stack trace you posted occurring in the provided rules-editor-tool-service or is that occurring in the service you created that is not showing up in the dropdown?

Also, what version of the service registry do you have deployed and what version of the rules editor?

I would also suggest you take a look at the service-registry logs, as it looks like the service is having trouble registering with the service registry.

3

The stack trace was from the rules-editor-tool-service.

See the version info below

NAME                                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
rules-editor-tool                       default         2               2022-09-27 14:21:48.7928697 -0400 EDT   deployed        rules-editor-tool-7.0.0 3.7.x      
service-registry                        default         1               2022-09-01 11:10:36.7685545 -0400 EDT   deployed        service-registry-4.0.4  3.5.3

The service-registry logs show the data service as registered, not seeing any errors

18:41:18.082 [qtp42968385-59] e[32mINFO e[m c.l.l.p.ServiceRegistryExtensible - OpenskyService registered.

4

What service type do you have your OpenskyService set as in the service registry? Looks like we are only looking for type DataService, and not looking for type RulesService. This is an oversight on the part of the rules editor app, but if you switch that, then it should work for you. We’ll worked to get this updated.

5

I’m not sure, I’m not familiar with how to configure the SR. Is there leaf documentation somewhere showing how this is done?

6

Hello, @jacob.i.bostwick. I might be able to shed some light on the error you provided in your initial post, and perhaps that will help clear up some things.

tl:dr

You’re using the 3.5.3 image of the Service Registry with a 3.7.x image of the Rules Editor Tool Service. Unfortunately, these will not be compatible. To resolve the error, choose a later version of the Service Registry. For example, instead of installing the 4.0.4 version of the service-registry Helm chart, install the 6.0.2 version or later version.

Details

LEAF Platform made a transition from 3.5.x to 3.6.x to migrate all Service Registry logic into our library to enable domains to build their own Service Registries. Before 3.6, all registration logic was built into the Service Registry, and the LEAF Platform libraries only contained the logic for communicating with the Service Registry. That was done via the class ServiceDiscoveryEmitter. As part of the transition, we made several improvements including breaking changes. One such breaking change was deprecating the ServiceDiscoveryEmitter and introducing new classes to replace it.

The core of the problem is that the 3.5.x Service Registry isn’t compatible with the new classes introduced in 3.6.x, and the 3.7.x version of the Rules Editor Tool Service is using the newer classes to try and communicate with it. (see in the error where it says ServiceRegistryScheduler? This is one such new class.)

We did want to maintain compatibility as much as we could however, so newer (3.6, 3.7, and 3.8 at minimum) LEAF Service Registries can still register older services using the now-deprecated ServiceDiscoveryEmitter. In other words, you can use a 3.6 or newer LEAF Service Registry with newer and older services (within a range), but if you are using an older LEAF Service Registry (3.5 or older), then newer services will not successfully register. Another consideration for using 3.6 is that it is an LTS release.

In summary, it should be a painless transition to go from the 3.5 to the 3.6 LEAF Service Registry, and if you encounter issues, always feel free to reach out on here for help. I hope this helps clear up the exception you posted with initially. I’m uncertain whether this will help with the other services you’re trying to register, and if you continue to encounter errors then post them and we can try to take a look. I’ll include some links below that may help you learn more about Service Discovery in general.

Links

1 Like
7

Thanks Hayden,

I went ahead an upgraded the SR. However, several of the services are unable to register with the SR. See this error for another data service in my cluster that is now unable to register.

14:30:56.853 [main] [32mINFO [m c.l.f.r.s.a.DataSubscriber - Query to find existing services: {"filters":{"type":[{"negated":false,"propertyIdentifier":"type","values":["DataRegistryEntry"],"caseSensitive":true,"objectType":"StringEqualsFilterBean"}]},"name":"<Unnamed Query>","version":0,"systemPropertiesRule":"ALL","systemProps":[],"nonSystemPropertiesRule":"ALL","nonSystemProps":[],"sortingSpec":{"directives":[]},"pagingSpec":{"pageSize":2147483647,"pageNumber":0},"leafVersion":"3.5"}
14:30:56.855 [main] [32mINFO [m c.l.f.r.s.a.DataSubscriber - Attempting to retrieve registry data information from http://data-registry-service.default.svc.cluster.local/rest/
14:30:56.990 [main] [33mWARN [m c.l.f.r.s.a.DataSubscriber - Unable to setup using existing registry information
java.lang.RuntimeException: java.lang.NoSuchMethodError: 'com.leidos.leaf.beans.BaseContext$BaseContextBuilder com.leidos.leaf.beans.Context$ContextBuilder.withTraceId(java.lang.String)'
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:65) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.dataobject.service.rest.DataObjectServiceRestClient.load(DataObjectServiceRestClient.java:522) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.falc2on.registry.subscriber.api.DataSubscriber.getDataSubscriptions(DataSubscriber.java:338) [data-subscriber-api-7.0.0-9-g162325f.jar:?]
	at com.leidos.falc2on.registry.subscriber.api.DataSubscriber.subscribe(DataSubscriber.java:145) [data-subscriber-api-7.0.0-9-g162325f.jar:?]
	at com.leidos.falc2on.registry.subscriber.api.DataSubscriber.subscribe(DataSubscriber.java:463) [data-subscriber-api-7.0.0-9-g162325f.jar:?]
	at com.leidos.falc2on.registry.subscriber.api.data.DataSubscriptionManager.doInitialize(DataSubscriptionManager.java:139) [data-subscriber-api-7.0.0-9-g162325f.jar:?]
	at com.leidos.falc2on.registry.subscriber.api.data.DataSubscriptionManager.initialize(DataSubscriptionManager.java:114) [data-subscriber-api-7.0.0-9-g162325f.jar:?]
	at com.leidos.falc2on.services.data.FalconDataServiceApplication.preServerInit(FalconDataServiceApplication.java:218) [classes/:?]
	at com.leidos.falc2on.leafbase.LeafBaseApplication.initialize(LeafBaseApplication.java:139) [leaf-app-base-3.5.5-21-g2cb8115.jar:?]
	at com.leidos.falc2on.leafbase.LeafBaseDataServiceApplication.initialize(LeafBaseDataServiceApplication.java:95) [leaf-app-base-3.5.5-21-g2cb8115.jar:?]
	at com.leidos.leaf.lifecycle.BaseApplication.run(BaseApplication.java:48) [framework-services-core-3.5.5.jar:?]
	at com.leidos.falc2on.services.data.FalconDataServiceApplication.main(FalconDataServiceApplication.java:403) [classes/:?]

Regardless of that, I did want to see if this might fix the issue I am having with my dataService not showing in the rules app. Unfortunately I am recieving the same error

Sep 28, 2022 6:11:09 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
SEVERE: Failed to export spans. Server is UNAVAILABLE. Make sure your collector is running and reachable from this network. Full error message:UNAVAILABLE: Unable to resolve host jaeger-collector.observability
18:11:09.905 [pool-4-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceRegistryScheduler - message: Exception encountered in service registration
javax.ws.rs.ProcessingException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'RulesEditorToolService': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (org.jboss.resteasy.specimpl.AbstractBuiltResponse$InputStreamWrapper); line: 1, column: 24]
	at org.jboss.resteasy.client.jaxrs.internal.ClientResponse.readFrom(ClientResponse.java:254) ~[resteasy-client-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.specimpl.BuiltResponse.readEntity(BuiltResponse.java:90) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.specimpl.AbstractBuiltResponse.readEntity(AbstractBuiltResponse.java:270) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at com.leidos.leaf.platform.servicediscovery.rest.ServiceRegistryRestClient.getDataObjectOrThrow(ServiceRegistryRestClient.java:103) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.platform.servicediscovery.rest.ServiceRegistryRestClient.registerService(ServiceRegistryRestClient.java:68) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.platform.servicediscovery.ServiceRegistryExtensible.lambda$new$0(ServiceRegistryExtensible.java:68) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.hooked.IMethod.invoke(IMethod.java:125) ~[framework-services-core-3.7.1.jar:?]
	at com.leidos.leaf.hooked.default_.hooks.TracerHook.lambda$hookInvoker$2(TracerHook.java:163) ~[framework-services-core-3.7.1.jar:?]
	at com.leidos.leaf.hooked.BaseExtensibleService.invoke(BaseExtensibleService.java:129) ~[framework-services-core-3.7.1.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceRegistryExtensible.registerService(ServiceRegistryExtensible.java:76) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at com.leidos.leaf.platform.servicediscovery.ServiceRegistryScheduler.lambda$getRegisterAction$0(ServiceRegistryScheduler.java:124) ~[platform-services-service-discovery-3.7.0.jar:3.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'RulesEditorToolService': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (org.jboss.resteasy.specimpl.AbstractBuiltResponse$InputStreamWrapper); line: 1, column: 24]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:745) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3635) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2734) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:902) ~[jackson-core-2.13.2.jar:2.13.2]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:794) ~[jackson-core-2.13.2.jar:2.13.2]
	at org.jboss.resteasy.plugins.providers.jackson.ResteasyJackson2Provider.readFrom(ResteasyJackson2Provider.java:181) ~[resteasy-jackson2-provider-4.6.1.Final.jar:4.6.1.Final]
	at org.jboss.resteasy.core.interception.jaxrs.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:101) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.core.interception.jaxrs.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:80) ~[resteasy-core-5.0.2.Final.jar:5.0.2.Final]
	at org.jboss.resteasy.client.jaxrs.internal.ClientResponse.readFrom(ClientResponse.java:217) ~[resteasy-client-5.0.2.Final.jar:5.0.2.Final]
	... 16 more
8

I also tried to step down my version of the rules app to 3.5.x, as that seemed like a good idea to me and another team member. However the rules tool service is unable to deploy, although the app deployed.

rules-editor-tool                       default         1               2022-09-28 15:10:42.2267529 -0400 EDT   deployed        rules-editor-tool-5.0.3 3.5.x

default        rules-editor-tool-app-658f5975f5-kfqth                  2/2     Running            0                   18m
default        rules-editor-tool-service-7c5956b49b-p5z4d              1/2     CrashLoopBackOff   7 (58s ago)         18m

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  19m                   default-scheduler  Successfully assigned default/rules-editor-tool-service-7c5956b49b-p5z4d to docker-desktop
  Normal   Pulled     19m                   kubelet            Container image "docker.io/istio/proxyv2:1.13.3" already present on machine
  Normal   Created    19m                   kubelet            Created container istio-init
  Normal   Started    19m                   kubelet            Started container istio-init
  Normal   Pulled     19m                   kubelet            Container image "docker.io/istio/proxyv2:1.13.3" already present on machine
  Normal   Created    19m                   kubelet            Created container istio-proxy
  Normal   Started    19m                   kubelet            Started container istio-proxy
  Normal   Pulled     19m (x2 over 19m)     kubelet            Container image "docker-group.leidos-leaf.com/rules-editor-tool-service:3.5.3" already present on machine
  Normal   Created    19m (x2 over 19m)     kubelet            Created container leaf-service
  Normal   Started    19m (x2 over 19m)     kubelet            Started container leaf-service
  Warning  Unhealthy  14m (x32 over 19m)    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff    4m45s (x48 over 18m)  kubelet            Back-off restarting failed container

This error repeats a few times until the pod restarts

19:22:05.110 [main] e[32mINFO e[m i.j.i.r.LoggingReporter - Span reported: 118a40ff8845a040:e2fc1fb4394f39f8:118a40ff8845a040:1 - IFieldMetadataOverrideService.processAnyOverriddenFieldMetadataWithSingleTypeAndEdition
19:22:05.111 [main] e[32mINFO e[m i.j.i.r.LoggingReporter - Span reported: 118a40ff8845a040:f4d25902692c0f5e:118a40ff8845a040:1 - IBeanMetadataCalculationService.processCalculatedFieldsOnBean
19:22:05.208 [jaeger.RemoteReporter-QueueProcessor] e[33mWARN e[m i.j.i.r.RemoteReporter - FlushCommand execution failed! Repeated errors of this command will not be logged.
io.jaegertracing.internal.exceptions.SenderException: Failed to flush spans.
	at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:116) ~[jaeger-thrift-1.6.0.jar:1.6.0]
	at io.jaegertracing.internal.reporters.RemoteReporter$FlushCommand.execute(RemoteReporter.java:160) ~[jaeger-core-1.6.0.jar:1.6.0]
	at io.jaegertracing.internal.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:182) [jaeger-core-1.6.0.jar:1.6.0]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.jaegertracing.internal.exceptions.SenderException: Could not send 13 spans
	at io.jaegertracing.thrift.internal.senders.HttpSender.send(HttpSender.java:70) ~[jaeger-thrift-1.6.0.jar:1.6.0]
	at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-1.6.0.jar:1.6.0]
	... 3 more
Caused by: java.net.UnknownHostException: jaeger-collector.observability
	at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
	at java.net.InetAddress.getAllByName0(InetAddress.java:1509) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1368) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1302) ~[?:?]
	at okhttp3.Dns$Companion$DnsSystem.lookup(Dns.kt:49) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.kt:164) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.kt:129) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.RouteSelector.next(RouteSelector.kt:71) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:205) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.9.0.jar:?]
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.9.0.jar:?]
	at io.jaegertracing.thrift.internal.senders.HttpSender.send(HttpSender.java:68) ~[jaeger-thrift-1.6.0.jar:1.6.0]
	at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-1.6.0.jar:1.6.0]
	... 3 more
19:22:05.208 [main] e[32mINFO e[m i.j.i.r.LoggingReporter - Span reported: 118a40ff8845a040:be25ac57085d179d:118a40ff8845a040:1 - IFieldMetadataOverrideService.processAnyOverriddenFieldMetadataWithSingleTypeAndEdition
19:22:05.209 [main] e[32mINFO e[m i.j.i.r.LoggingReporter - Span reported: 118a40ff8845a040:8f1ce22b42a07422:118a40ff8845a040:1 - IBeanMetadataCalculationService.processCalculatedFieldsOnBean
19:22:05.212 [main] e[32mINFO e[m i.j.i.r.LoggingReporter - Span reported: 118a40ff8845a040:20fdaee0d60a1923:118a40ff8845a040:1 - IFieldMetadataOverrideService.processAnyOverriddenFieldMetadataWithSingleTypeAndEdition
9

The First Error Block

As far as I can discern, this error is happening when a domain team’s class called DataSubscriber is issuing a REST call using a LEAF Framework Services class called DataObjectServiceRestClient to the address http://data-registry-service.default.svc.cluster.local/rest/. It seems like the objective is to issue a DataObjectQuery to this data-registry-service to load DataObjects of type DataRegistryEntry. Admittedly, I’m not totally sure what’s going on beyond that. To the best of my knowledge, this DataSubscriber class is not something LEAF generates or otherwise offers, but I could be mistaken. Furthermore, loading DataObjects by DataObjectQuery would often sent to the following endpoint: /rest/data-objects/load/by-query, while registration requests would be sent to: /rest/service/register.

When deploying the LEAF Service Registry, are you configuring it to be hosted at /data-registry-service/? Otherwise, by default, it will be reachable at http://service-registry/... (at least for internal cluster communication like a REST call from one service to another). To see the default configuration used in the service-registry Helm chart, run helm show values leidos-leaf/service-registry --version 6.0.2 in your terminal. The version may differ depending upon what you chose, and remember that if you passed in your own values.yaml file when you install the service-registry Helm chart that some defaults will have been overridden.

Can you provide a little more context around the DataSubscriber? Do you know if that’s code produced by your team or otherwise? Is that the proper URI to be trying to make that request to? Since that’s a RuntimeException, I’m curious if there were any recent changes made to the service such as dependency upgrade. What I can say is that this error doesn’t appear to be related to an attempt to register a Data Service, but rather an attempt to load DataObjects.

As I think more about it, is it possible that your team has extended the LeafService BeanMetadata normally used for Service Registration?

The Second Error Block

That’s unfortunate. I was able to reproduce the error before I replied yesterday by deploying the same versions of the Helm charts that you listed. Then, I deployed the 6.0.2 Service Registry and I successfully saw the Rules Editor Tool Service register, so I was confident this would be solved by the upgrade. My apologies that it doesn’t appear to be the case.

In order to debug further, can you provide the versions for the Service Registry you upgraded to? Also, do you see anything in the logs of the updated Service Registry related to the failed service registration attempts?

10

In Response to the 3.5 Rules Editor Tool Error

The Jaeger Tracing error is caused by Tracing being enabled for the service, but it cannot find a deployment of Jaeger to push the traces to. This can be resolved by either: disabling Tracing with configuration, or deploying Jaeger to the cluster.

I’m uncertain on whether the Rules Editor Tool Service supports disabling Tracing in 3.5. I have to defer to @dustin.teel on that.

More information on deploying Jaeger to a local cluster can be found in the LEAF Knowledge Base.

EDIT:
That said, I don’t believe that Tracing error is fatal to the Rules Editor Tool Service. If I was to take a guess, I’d think you could try increasing the CPU resources allotted to the service. My suspicion is that the service is timing out when upserting data on startup and that is the real cause of the service restarting and never fully standing up.

11

There is not an option to disable tracing, so if you would like to remove that error, you would need to deploy Jaeger to your cluster. Hayden is correct in that the error is not fatal, though, so you can safely ignore it.

He is correct in that the issue is most likely the CPU/Memory resources not being sufficient, so I would also recommend increasing them.

Another thing to note is that if you deployed the 3.5 service after having the 3.7 service deployed, its also possible you would need to wipe the database, as if the metadata changed from 3.5 to 3.7 going backwards like that without wiping the database could also cause issues.

12

Apologies for some confusion here, I was mistaken about the first error block. Still learning how our backend works. So we have a data-registry-service and a service-registry. The difference in the two is:

the service registry contains data object representations of services
the data registry service contains bean metadata of services. A service can register what bean metadata they have and then other services can discover those services indirectly.

I.e.

The data-service1 registers that it’s producing car dataObjects to the data registry service. It provides the kafka topic it’s producing to.

The data-service2 doesn’t know about the data-service1 directly. It scans the registrations of the data registry service for anything that produces car dataObjects and finds data-service1. It then subscribes to the kafka topic it’s producing to.

So that first error was on our end.

13

Ok, increasing CPU/Mem did the trick. So here is where I am at now.

I have the service-registry rules-editor-tool-app rules-editor-tool-service and openSkyService all deployed. Additionally, I have our teams data-registry-service deployed. See this post

In the SR, I see each service being registered.

19:20:33.612 [qtp1075390175-51] e[32mINFO e[m c.l.l.p.ServiceRegistryExtensible - OpenskyService registered.
19:18:18.023 [qtp1075390175-43] e[32mINFO e[m c.l.l.p.ServiceRegistryExtensible - RulesEditorToolService registered.
19:10:58.964 [qtp1075390175-42] e[32mINFO e[m c.l.l.p.ServiceRegistryExtensible - DataRegistryService registered.

However, periodically, I will get this error message in both the openSkyService and the Data-registry-service.

19:43:07.243 [pool-3-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceDiscoveryEmitter - Error encountered registering the service object
java.lang.RuntimeException: Response indicates failure, but unable to parse exception message. Status: 503 Entity: null
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:58) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.registerService(ServiceDiscoveryEmitter.java:145) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.lambda$scheduleRegistration$0(ServiceDiscoveryEmitter.java:124) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

I was able to deploy the older version of the rules-editor-tool, and step back down to the original version I had of the SR (was causing issues with other services).

rules-editor-tool       default         1               2022-09-29 15:17:55.5927173 -0400 EDT   deployed        rules-editor-tool-5.0.3 3.5.x
service-registry        default         1               2022-09-29 15:09:02.6771666 -0400 EDT   deployed        service-registry-4.0.4  3.5.3

The opensky service does not show in the rules app however.

Oddly enough, I am able to get another service showing up in the rules app even though it throws the same error.

19:53:28.829 [Thread-2] e[32mINFO e[m c.l.l.s.k.StreamReaderKafka - Polled 100 times, received 0 records
19:53:33.563 [pool-3-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceDiscoveryEmitter - Error encountered registering the service object
java.lang.RuntimeException: Response indicates failure, but unable to parse exception message. Status: 503 Entity: null
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:58) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.registerService(ServiceDiscoveryEmitter.java:145) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.lambda$scheduleRegistration$0(ServiceDiscoveryEmitter.java:124) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
19:53:38.855 [Thread-2] e[32mINFO e[m c.l.l.s.k.StreamReaderKafka - Polled 100 times, received 0 records
19:53:48.879 [Thread-2] e[32mINFO e[m c.l.l.s.k.StreamReaderKafka - Polled 100 times, received 0 records
19:53:58.000 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - ========================== Memory Info ==========================
19:53:58.000 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Free memory: 90 MB
19:53:58.000 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Allocated memory: 146 MB
19:53:58.000 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Max memory: 1,024 MB
19:53:58.000 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Total free memory: 968 MB
19:53:58.000 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - =================================================================

19:53:58.899 [Thread-2] e[32mINFO e[m c.l.l.s.k.StreamReaderKafka - Polled 100 times, received 0 records
19:54:08.923 [Thread-2] e[32mINFO e[m c.l.l.s.k.StreamReaderKafka - Polled 100 times, received 0 records
19:54:18.943 [Thread-2] e[32mINFO e[m c.l.l.s.k.StreamReaderKafka - Polled 100 times, received 0 records
19:54:28.966 [Thread-2] e[32mINFO e[m c.l.l.s.k.StreamReaderKafka - Polled 100 times, received 0 records
14

With respect to this periodic ServiceDiscoveryEmitter failure in the OpenSkyService and DataRegistryService:

  • Does it seem to happen once or twice on startup and then cease, or does it continue to happen throughout the life of the deployment?
    • If it is only once or twice at the beginning, this could be a side effect of the service reaching out to the Service Registry asking to be registered, but then itself not being ready to accept traffic. Any Discoverable Services, OpenSkyService and DataRegistryService in this case, must be able to pass a health check before the Service Registry will allow them to be “registered”. If the Discoverable Service isn’t ready to reply with 200 to a health check request, but it has made a request to be registered, then registration will fail until the service can pass health checks.
    • If it is happening later after successful registration at seemingly random intervals, does the service stay registered after this failed registration request, or does it disappear from the Service Registry’s data store?
      • It would be new for me to hear about a periodic failure in service registration after a Discoverable Service has already registered previously. I’m not sure what could cause that behavior off the top of my head. If it does remain registered despite the periodic request failures, then at least it shouldn’t be disruptive to the availability of the service, but of course a periodic error is not ideal either way.
  • Do you see anything in the Service Registry logs at the same time that these error occur?
    • Unfortunately, one of the downsides of the now-deprecated ServiceDiscoveryEmitter is that it did not maintain and log error information well. Thus, all we have to work with in the Discoverable Service side of things is this 503 Entity: null and we’re better off looking in the Service Registry for more info.

You can upgrade older services to use the newer Service Discovery classes if you wish. They should do a better job informing you about what went wrong on the Discoverable Service side of things. Discoverable Service Registration Guide I definitely understand that you may not have time to tack such an upgrade, but I thought to mention it anyway.

15

This happens to both services AFTER they have registered with the SR. For instance, the openSky service is registered here in the SR logs(See time stamp).

19:20:27.519 [qtp1075390175-51] e[1;31mERRORe[m c.l.l.h.d.h.LogHook - LEAF traceId=00031138e3534fb8a93c677ff05f4bf6, OpenTracing traceId=, OpenTracing spanId=, username=AnonymousSystemUser called IDataObjectService.loadByIds with arguments (ids = [OpenskyService]) before throwing com.leidos.leaf.beans.exception.not.found.DataObjectNotFoundException: Could not find DataObject with identifier: OpenskyService
[Note that this could be due to the entities not existing, or the User having insufficient permission to access them.]
com.leidos.leaf.beans.exception.not.found.DataObjectNotFoundException: Could not find DataObject with identifier: OpenskyService
[Note that this could be due to the entities not existing, or the User having insufficient permission to access them.]
	at com.leidos.leaf.beans.exception.not.found.DataObjectNotFoundException.exceptionForIdentifiers(DataObjectNotFoundException.java:40) ~[framework-services-core-3.5.5.jar:?]
	at com.leidos.leaf.internal.dataobject.service.IDataObjectServiceRx.lambda$loadRx$1(IDataObjectServiceRx.java:243) ~[framework-services-core-3.5.5.jar:?]
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onComplete(FlowableDoOnEach.java:128) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onComplete(FlowableDoOnEach.java:135) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.operators.mixed.CompletableAndThenPublisher$AndThenPublisherSubscriber.onComplete(CompletableAndThenPublisher.java:83) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onComplete(SingleFlatMapPublisher.java:112) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$BaseObserveOnSubscriber.checkTerminated(FlowableObserveOn.java:199) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$ObserveOnSubscriber.runAsync(FlowableObserveOn.java:394) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$BaseObserveOnSubscriber.run(FlowableObserveOn.java:176) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) ~[rxjava-3.0.4.jar:?]
	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) ~[rxjava-3.0.4.jar:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
19:20:27.521 [qtp1075390175-51] e[32mINFO e[m c.l.l.p.ServiceRegistryApplication - LEAF traceId=00031138e3534fb8a93c677ff05f4bf6, OpenTracing traceId=, OpenTracing spanId=, username=AnonymousSystemUser called IDataObjectService.constructLatestEditionsWithDefaultValues with arguments (types = [LeafService]) which returned [LeafService () {  }]
19:20:30.534 [qtp1075390175-51] e[33mWARN e[m c.l.l.p.ServiceRegistryExtensible - Service OpenskyService failed health check. Health URI: http://opensky-service.default/health
19:20:32.723 [pool-3-thread-1] e[32mINFO e[m c.l.l.p.ServiceRegistryApplication - LEAF traceId=9f0c9938420a4084a3cf1c341715c416, username=AnonymousSystemUser called IDataObjectService.loadByQuery with arguments (dataObjectQuery = DataObjectQuery [name=<Unnamed Query>, id=null]) which returned [LeafService (WebsocketService) {  }, LeafService (DashboardService) {  }, LeafService (falcon-overlay-service) {  }, LeafService (DataRegistryService) {  }, LeafService (RulesEditorToolService) {  }]
19:20:32.723 [pool-3-thread-1] e[32mINFO e[m c.l.l.p.ServiceRegistryExtensible - Periodic health check - checking 5 service(s).
19:20:33.611 [qtp1075390175-51] e[32mINFO e[m c.l.l.p.ServiceRegistryApplication - LEAF traceId=00031138e3534fb8a93c677ff05f4bf6, OpenTracing traceId=, OpenTracing spanId=, username=AnonymousSystemUser called IDataObjectService.addRoots with arguments (dataObjects = [LeafService (OpenskyService) {  }]) which returned [LeafService (OpenskyService) {  }]
19:20:33.612 [qtp1075390175-51] e[32mINFO e[m c.l.l.p.ServiceRegistryExtensible - OpenskyService registered.

However, here are time stamps after that where I recieve this error trying to register the service from the openSky logs.

19:52:47.232 [pool-3-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceDiscoveryEmitter - Error encountered registering the service object
java.lang.RuntimeException: Response indicates failure, but unable to parse exception message. Status: 503 Entity: null
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:58) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.registerService(ServiceDiscoveryEmitter.java:145) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.lambda$scheduleRegistration$0(ServiceDiscoveryEmitter.java:124) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
19:52:50.234 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - ========================== Memory Info ==========================
19:52:50.234 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Free memory: 7 MB
19:52:50.234 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Allocated memory: 75 MB
19:52:50.234 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Max memory: 989 MB
19:52:50.234 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Total free memory: 921 MB
19:52:50.234 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - =================================================================

19:53:07.233 [pool-3-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceDiscoveryEmitter - Error encountered registering the service object
java.lang.RuntimeException: Response indicates failure, but unable to parse exception message. Status: 503 Entity: null
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:58) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.registerService(ServiceDiscoveryEmitter.java:145) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.lambda$scheduleRegistration$0(ServiceDiscoveryEmitter.java:124) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
19:53:27.232 [pool-3-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceDiscoveryEmitter - Error encountered registering the service object
java.lang.RuntimeException: Response indicates failure, but unable to parse exception message. Status: 503 Entity: null
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:58) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.registerService(ServiceDiscoveryEmitter.java:145) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.lambda$scheduleRegistration$0(ServiceDiscoveryEmitter.java:124) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
19:53:50.235 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - ========================== Memory Info ==========================
19:53:50.235 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Free memory: 23 MB
19:53:50.235 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Allocated memory: 75 MB
19:53:50.235 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Max memory: 989 MB
19:53:50.235 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Total free memory: 937 MB
19:53:50.235 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - =================================================================

If it is happening later after successful registration at seemingly random intervals, does the service stay registered after this failed registration request, or does it disappear from the Service Registry’s data store?

I’m not sure how to see this, can you send instructions on how to see the data store?

16

Also, I have this repeating in the rules-editor-tool-service logs.

12:01:57.338 [qtp801922889-277] e[32mINFO e[m c.l.l.r.e.t.s.RulesEditorToolServiceApplication - LEAF traceId=c9bc0e9d8bd243e293265831bc226faf, OpenTracing traceId=033e0e49d87ea34a, OpenTracing spanId=92dd7d2d43604081, username=AnonymousSystemUser called IMetadataService.loadBeanMetadata with arguments (beanMetadataQuery = BeanMetadataQuery [name=<Unnamed Query>]) which returned [BeanMetadata [type = HierarchyPermission, edition = 1, extendsCsv = Permission-1, displayName = HierarchyPermission, listable = true, concrete = true, loggableAttribute = , preferredDisplayPropName = displayLabel, serviceName = RulesEditorToolServiceMetadataService, version = 1]
    superTypes: [Permission-1]
    tagsCsv: [leaf, leaf-authorization]
    Validation Strings: <NONE>Calculated Validation Strings: <NONE>Overriding System FieldMetadata: <NONE>
    Inherited Non-System FieldMetadata: <NONE>
    Non-Inherited Overriding Non-System FieldMetadata: <NONE>
    Non-Inherited Non-Overriding Non-System FieldMetadata: 
        BooleanFieldMetadata propName=canAddDescendants
            shortName=canAddDescendants
            longName=canAddDescendants
            defaultLayoutIndex=1
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can add descendants of this DataObject
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can add descendants of this DataObject
        BooleanFieldMetadata propName=canDelete
            shortName=canDelete
            longName=canDelete
            defaultLayoutIndex=4
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can delete this DataObject or its descendants
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can delete this DataObject or its descendants
        BooleanFieldMetadata propName=canRead
            shortName=canRead
            longName=canRead
            defaultLayoutIndex=2
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can load this DataObject or its descendants
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can load this DataObject or its descendants
        BooleanFieldMetadata propName=canUpdate
            shortName=canUpdate
            longName=canUpdate
            defaultLayoutIndex=3
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can update this DataObject or its descendants
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can update this DataObject or its descendants
        StringFieldMetadata propName=dataObjectID
            shortName=dataObjectID
            longName=dataObjectID
            defaultLayoutIndex=0
            defaultListed=true
            helpText=The ID of the DataObject instance this permission authorizes
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=The ID of the DataObject instance this permission authorizes]
12:01:57.338 [qtp801922889-277] e[32mINFO e[m c.l.l.r.e.t.s.RulesEditorToolServiceApplication - LEAF traceId=c9bc0e9d8bd243e293265831bc226faf, OpenTracing traceId=033e0e49d87ea34a, OpenTracing spanId=92dd7d2d43604081, username=AnonymousSystemUser called IMetadataService.loadBeanMetadata with arguments (beanMetadataQuery = BeanMetadataQuery [name=<Unnamed Query>]) which returned [BeanMetadata [type = HierarchyPermission, edition = 1, extendsCsv = Permission-1, displayName = HierarchyPermission, listable = true, concrete = true, loggableAttribute = , preferredDisplayPropName = displayLabel, serviceName = RulesEditorToolServiceMetadataService, version = 1]
    superTypes: [Permission-1]
    tagsCsv: [leaf, leaf-authorization]
    Validation Strings: <NONE>Calculated Validation Strings: <NONE>Overriding System FieldMetadata: <NONE>
    Inherited Non-System FieldMetadata: <NONE>
    Non-Inherited Overriding Non-System FieldMetadata: <NONE>
    Non-Inherited Non-Overriding Non-System FieldMetadata: 
        BooleanFieldMetadata propName=canAddDescendants
            shortName=canAddDescendants
            longName=canAddDescendants
            defaultLayoutIndex=1
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can add descendants of this DataObject
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can add descendants of this DataObject
        BooleanFieldMetadata propName=canDelete
            shortName=canDelete
            longName=canDelete
            defaultLayoutIndex=4
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can delete this DataObject or its descendants
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can delete this DataObject or its descendants
        BooleanFieldMetadata propName=canRead
            shortName=canRead
            longName=canRead
            defaultLayoutIndex=2
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can load this DataObject or its descendants
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can load this DataObject or its descendants
        BooleanFieldMetadata propName=canUpdate
            shortName=canUpdate
            longName=canUpdate
            defaultLayoutIndex=3
            defaultListed=true
            defaultValue=false
            helpText=Denotes whether a User can update this DataObject or its descendants
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=Denotes whether a User can update this DataObject or its descendants
        StringFieldMetadata propName=dataObjectID
            shortName=dataObjectID
            longName=dataObjectID
            defaultLayoutIndex=0
            defaultListed=true
            helpText=The ID of the DataObject instance this permission authorizes
            originatingType=HierarchyPermission
            owningType=HierarchyPermission
            required=true
            tooltipText=The ID of the DataObject instance this permission authorizes]

This was happening here, and the solution was to increase the pod resources. However that time the pod was throwing an error and never deploying. The pod is deployed now, but I’m not sure if the repeating metadata is normal or not

17

Service Registry 3.5.x Exception

The following exception is unfortunately expected behavior in older (3.5 or lower) Service Registries.

com.leidos.leaf.beans.exception.not.found.DataObjectNotFoundException: Could not find DataObject with identifier: OpenskyService
[Note that this could be due to the entities not existing, or the User having insufficient permission to access them.]

To detail out why, here’s a rough overview of the registration process:

  • A Discoverable Service makes a request to the Service Registry to be registered. This request includes a DataObject that describes the Discoverable Service.
  • The Service Registry receives the request and attempts to load the DataObject contained in the request to find out if the Discoverable Service is already registered. That has two common outcomes.
    • If the DataObject is successfully loaded, then the Service Registry already has that service registered, so the registration request no longer needs to proceed.
    • If the DataObject is not loaded, this exception is thrown due to how the load is performed. We’ve sense fixed this in later versions of the Service Registry (3.6+), but not finding a DataObject is acceptable in this instance and the Service Registry will simply continue on with the registration process. This is a misleading exception, but it is behaving correctly.

Registration Failures

I’m just going to post my understanding of what is happening based upon the logs above, and feel free to correct anything I’m not getting right. At 19:20 in the Service Registry logs, we see the first attempt by the OpenSkyService to register itself. The misleading exception is printed which indicated that the Service Registry has found this to be an unregistered service. So, the Service Registry moves on to check the health of the service it’s about to register. The OpenSkyService fails the first health check. This is no problem, because even the old ServiceRegistryExtensible class is clever enough to try a configurable number of times before deciding a service is truly unhealthy.

In the middle of all of this, the Service Registry kicks off its periodic health check of all registered services. This is unrelated to the OpenSkyService’s registration request. In this case, there are 5 registered services that it has to confirm are still alive.

A short delay later, and the Service Registry has moved on with registering the OpenSkyService because the service was probably ready to respond to health checks on the second health check attempt and was still just starting up on the first health check attempt.

Approximately 30 minutes later, we see in the OpenSkyService logs that it has failed service registration. Do I understand correctly in saying that? If so, here are some questions to help narrow down what is happening.

  • When we see these registration failures what do we see at the same time in the Service Registry’s logs?
  • At this point in time, so 30 minutes after initial registration, has the OpenSkyService been alive and well for all 30 minutes, or is this a restart of the service?

Checking the Data Store

In 3.5, the Service Registry only supported two options for its data store: MongoDB and Transient. With that being the case, I’d assume MongoDB is being used and can thus recommend a tool like MongoDB Compass to connect and directly inspect the data store’s contents.

Alternatively, we can send a REST request to the Service Registry to query for the services it has registered. I’ll provide a cURL that you can directly issue via a terminal, or you could import it into a tool like Postman.

cURL

Run this in your terminal to issue a request. Note that you may have to update the URL to actually reach the Service Registry where you have it deployed.

curl --location --request POST 'http://localhost/service-registry/rest/data-objects/load/by-query' \
--header 'user: {"username":"AnonymousSystemUser","authorizationProp":"LEAF_SYSTEM_USER"}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "filters": {},
    "name": "<Unnamed Query>",
    "systemPropertiesRule": "ALL",
    "systemProps": [],
    "nonSystemPropertiesRule": "ALL",
    "nonSystemProps": [],
    "sortingSpec": {},
    "pagingSpec": {
        "pageSize": 2147483647,
        "pageNumber": 0
    }
}'
18

Repeating Print of Metadata in the Rules Editor Tool Service

I have to defer to @dustin.teel again for this one.

19

We have verbose logging on, so this is normal. Each time you make a call, it has to check permissions, so that is likely why you are seeing this log a lot.

20

Registration Failures

Ok, so the SR seems to be restarting for unknown reasons, and I’m assuming this is causing the other services to throw the registration error. See the 26 restarts here,

default        service-registry-859f999fb5-8npd4           2/2     Running     26 (4m43s ago)      20h

And see the logs where it is spinning up and at the same time the openSky says it can’t register.

SR:

15:01:57.941 [main] e[32mINFO e[m c.l.l.p.ServiceRegistryApplication - ServiceRegistryApplication starting

Opensky:

15:01:58.736 [pool-3-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceDiscoveryEmitter - Error encountered registering the service object
java.lang.RuntimeException: Response indicates failure, but unable to parse exception message. Status: 503 Entity: null
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:58) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.registerService(ServiceDiscoveryEmitter.java:145) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.lambda$scheduleRegistration$0(ServiceDiscoveryEmitter.java:124) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
15:02:01.630 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - ========================== Memory Info ==========================
15:02:01.630 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Free memory: 24 MB
15:02:01.630 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Allocated memory: 90 MB
15:02:01.630 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Max memory: 989 MB
15:02:01.630 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Total free memory: 923 MB
15:02:01.630 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - =================================================================

15:02:18.736 [pool-3-thread-1] e[1;31mERRORe[m c.l.l.p.s.ServiceDiscoveryEmitter - Error encountered registering the service object
java.lang.RuntimeException: Response indicates failure, but unable to parse exception message. Status: 503 Entity: null
	at com.leidos.leaf.rest.util.exception.RestExceptionUtils.checkResponseForException(RestExceptionUtils.java:58) ~[framework-services-rest-3.5.5.jar:?]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.registerService(ServiceDiscoveryEmitter.java:145) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at com.leidos.leaf.platform.servicediscovery.ServiceDiscoveryEmitter.lambda$scheduleRegistration$0(ServiceDiscoveryEmitter.java:124) ~[platform-services-service-discovery-3.5.3.jar:3.5.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
15:03:01.631 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - ========================== Memory Info ==========================
15:03:01.631 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Free memory: 44 MB
15:03:01.631 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Allocated memory: 90 MB
15:03:01.631 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Max memory: 989 MB
15:03:01.631 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - Total free memory: 943 MB
15:03:01.631 [pool-2-thread-1] e[32mINFO e[m c.l.f.l.LeafBaseDataServiceApplication - =================================================================

In mongoDB compass, the only collections I have for the SR are:

  1. beanMetadata - with one obj, the bean for the SR
  2. dataObjectQueries - empty
  3. dataChangeEventQueries - empty

cURL

Here is the output of the curl request, which you can see the openSkyService is the last element in the array

bostwickj@6TH9CF3 MINGW64 ~/development/smac-core-components/data-registry-service (master)
$ curl --location --request POST 'http://localhost/service-registry/rest/data-objects/load/by-query' --header 'user: {"username":"AnonymousSystemUser","authorizationProp":"LEAF_SYSTEM_USER"}' --header 'Content-Type: application/json' --data-raw '{
    "filters": {},
    "name": "<Unnamed Query>",       
    "systemPropertiesRule": "ALL",   
    "systemProps": [],
    "nonSystemPropertiesRule": "ALL",
    "nonSystemProps": [],
    "sortingSpec": {},
    "pagingSpec": {
        "pageSize": 2147483647,
        "pageNumber": 0
    }
}' | json_pp -json_opt pretty,canonical
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6178    0  5898  100   280   171k   8352 --:--:-- --:--:-- --:--:--  188k
{
   "data" : [
      {
         "createdByUsername" : "AnonymousSystemUser",
         "createdDate" : 1664478590013,
         "dataObjectAuthorizationToken" : {
            "canUserDelete" : true,
            "canUserEdit" : true,
            "nonEditableFields" : [],
            "nonViewableFields" : []
         },
         "displayLabel" : "WebsocketService",
         "edition" : 1,
         "healthCheckURI" : "http://websocket-service.default/health",
         "id" : "WebsocketService",
         "lastModifiedByUsername" : "AnonymousSystemUser",
         "lastModifiedDate" : 1664478590013,
         "leafVersion" : "3.5",
         "qualifiedId" : "/WebsocketService",
         "qualifiedType" : "/LeafService",
         "rootId" : "WebsocketService",
         "rootType" : "LeafService",
         "serviceAlias" : "WebsocketService",
         "serviceContext" : "websocket-service",
         "serviceName" : "ServiceRegistryDataObjectService",
         "serviceType" : "WebsocketService",
         "superTypes" : [],
         "type" : "LeafService",
         "version" : 1
      },
      {
         "createdByUsername" : "AnonymousSystemUser",
         "createdDate" : 1664478591159,
         "dataObjectAuthorizationToken" : {
            "canUserDelete" : true,
            "canUserEdit" : true,
            "nonEditableFields" : [],
            "nonViewableFields" : []
         },
         "displayLabel" : "DashboardService",
         "edition" : 1,
         "healthCheckURI" : "http://dashboard-service.default/health",
         "id" : "DashboardService",
         "lastModifiedByUsername" : "AnonymousSystemUser",
         "lastModifiedDate" : 1664478591159,
         "leafVersion" : "3.5",
         "qualifiedId" : "/DashboardService",
         "qualifiedType" : "/LeafService",
         "rootId" : "DashboardService",
         "rootType" : "LeafService",
         "serviceAlias" : "DashboardService",
         "serviceContext" : "dashboard-service",
         "serviceName" : "ServiceRegistryDataObjectService",
         "serviceType" : "DataService",
         "superTypes" : [],
         "type" : "LeafService",
         "version" : 1
      },
      {
         "createdByUsername" : "AnonymousSystemUser",
         "createdDate" : 1664478592936,
         "dataObjectAuthorizationToken" : {
            "canUserDelete" : true,
            "canUserEdit" : true,
            "nonEditableFields" : [],
            "nonViewableFields" : []
         },
         "displayLabel" : "falcon-overlay-service",
         "edition" : 1,
         "healthCheckURI" : "http://falcon-overlay-service.default/health",
         "id" : "falcon-overlay-service",
         "lastModifiedByUsername" : "AnonymousSystemUser",
         "lastModifiedDate" : 1664478592936,
         "leafVersion" : "3.5",
         "qualifiedId" : "/falcon-overlay-service",
         "qualifiedType" : "/LeafService",
         "rootId" : "falcon-overlay-service",
         "rootType" : "LeafService",
         "serviceAlias" : "falcon-overlay-service",
         "serviceContext" : "falcon-overlay-service",
         "serviceName" : "ServiceRegistryDataObjectService",
         "serviceType" : "DataService",
         "superTypes" : [],
         "type" : "LeafService",
         "version" : 1
      },
      {
         "createdByUsername" : "AnonymousSystemUser",
         "createdDate" : 1664478658956,
         "dataObjectAuthorizationToken" : {
            "canUserDelete" : true,
            "canUserEdit" : true,
            "nonEditableFields" : [],
            "nonViewableFields" : []
         },
         "displayLabel" : "DataRegistryService",
         "edition" : 1,
         "healthCheckURI" : "http://data-registry-service.default/health",
         "id" : "DataRegistryService",
         "lastModifiedByUsername" : "AnonymousSystemUser",
         "lastModifiedDate" : 1664478658956,
         "leafVersion" : "3.5",
         "qualifiedId" : "/DataRegistryService",
         "qualifiedType" : "/LeafService",
         "rootId" : "DataRegistryService",
         "rootType" : "LeafService",
         "serviceAlias" : "DataRegistryService",
         "serviceContext" : "data-registry-service",
         "serviceName" : "ServiceRegistryDataObjectService",
         "serviceType" : "DataObjectService",
         "superTypes" : [],
         "type" : "LeafService",
         "version" : 1
      },
      {
         "allowsLiveMetadataEditing" : false,
         "allowsNonLiveMetadataEditing" : false,
         "allowsRuleEditing" : true,
         "allowsRuleViewing" : true,
         "createdByUsername" : "AnonymousSystemUser",
         "createdDate" : 1664479098014,
         "dataObjectAuthorizationToken" : {
            "canUserDelete" : true,
            "canUserEdit" : true,
            "nonEditableFields" : [],
            "nonViewableFields" : []
         },
         "displayLabel" : "RulesEditorToolService",
         "edition" : 1,
         "healthCheckURI" : "http://rules-editor-tool-service.default/health",
         "id" : "RulesEditorToolService",
         "lastModifiedByUsername" : "AnonymousSystemUser",
         "lastModifiedDate" : 1664479098014,
         "leafVersion" : "3.5",
         "qualifiedId" : "/RulesEditorToolService",
         "qualifiedType" : "/LeafService",
         "rootId" : "RulesEditorToolService",
         "rootType" : "LeafService",
         "serviceAlias" : "RulesEditorToolService",
         "serviceContext" : "rules-editor-tool-service",
         "serviceName" : "ServiceRegistryDataObjectService",
         "serviceType" : "DataService",
         "superTypes" : [],
         "type" : "LeafService",
         "version" : 1
      },
      {
         "allowsRuleEditing" : true,
         "allowsRuleViewing" : true,
         "createdByUsername" : "AnonymousSystemUser",
         "createdDate" : 1664481096598,
         "dataObjectAuthorizationToken" : {
            "canUserDelete" : true,
            "canUserEdit" : true,
            "nonEditableFields" : [],
            "nonViewableFields" : []
         },
         "displayLabel" : "com.leidos.falcon.target.definition.TargetDefinitionServiceApplication",
         "edition" : 1,
         "eventServiceURI" : "/websocket-service/web-socket/event-query",
         "healthCheckURI" : "http://target-definition-service.default/health",
         "id" : "com.leidos.falcon.target.definition.TargetDefinitionServiceApplication",
         "lastModifiedByUsername" : "AnonymousSystemUser",
         "lastModifiedDate" : 1664481096598,
         "leafVersion" : "3.5",
         "qualifiedId" : "/com.leidos.falcon.target.definition.TargetDefinitionServiceApplication",
         "qualifiedType" : "/LeafService",
         "rootId" : "com.leidos.falcon.target.definition.TargetDefinitionServiceApplication",
         "rootType" : "LeafService",
         "serviceAlias" : "com.leidos.falcon.target.definition.TargetDefinitionServiceApplication",
         "serviceContext" : "target-definition-service",
         "serviceName" : "ServiceRegistryDataObjectService",
         "serviceType" : "DataService",
         "superTypes" : [],
         "type" : "LeafService",
         "version" : 1
      },
      {
         "allowsRuleEditing" : true,
         "allowsRuleViewing" : true,
         "createdByUsername" : "AnonymousSystemUser",
         "createdDate" : 1664540741900,
         "dataObjectAuthorizationToken" : {
            "canUserDelete" : true,
            "canUserEdit" : true,
            "nonEditableFields" : [],
            "nonViewableFields" : []
         },
         "displayLabel" : "OpenskyService",
         "edition" : 1,
         "eventServiceURI" : "/websocket-service/web-socket/event-query",
         "healthCheckURI" : "http://opensky-service.default/health",
         "id" : "OpenskyService",
         "lastModifiedByUsername" : "AnonymousSystemUser",
         "lastModifiedDate" : 1664540741900,
         "leafVersion" : "3.5",
         "qualifiedId" : "/OpenskyService",
         "qualifiedType" : "/LeafService",
         "rootId" : "OpenskyService",
         "rootType" : "LeafService",
         "serviceAlias" : "OpenskyService",
         "serviceContext" : "opensky-service",
         "serviceName" : "ServiceRegistryDataObjectService",
         "serviceType" : "FunctionalService",
         "superTypes" : [],
         "type" : "LeafService",
         "version" : 1
      }
   ]
}