FieldMetadata format attribute support

ryan.c.scott · July 7, 2021, 8:32pm

We have a String field for email.
According to this post, the regex is:

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

Does LEAF currently support something like this? It looks like it uses a few of the more intricate regex operands.

mark.a.alvaro · July 7, 2021, 8:35pm

I’ve read many times that regexes are not the best way to validate email addresses including in that post. Definitely LEAF has no concept yet of an EmailAddressFieldMetadata or an out-of-the-box regex or validation method. You can certainly try that regex in a StringField’s format, which if Java supports it, should be fine as far as our backend is concerned. JS may also try to validate formats client-side (@caleb.m.mchenry ), in which case you’d need to know if they support it as well. Of course you are definitely free to try, but I’m not sure I’d suggest using that regex over a validation library.

ryan.c.scott · July 8, 2021, 4:00pm

[Fatal Error] :57:46: The entity name must immediately follow the '&' in the entity reference.
Exception in thread "main" java.lang.RuntimeException: Failed to load resource with name: /beanMetadata/request/Request.xml
	at com.leidos.leaf.beans.parsers.loader.ResourceLoader.wrapErrorForLoadByName(ResourceLoader.java:184)
	at com.leidos.leaf.beans.parsers.loader.ResourceLoader.lambda$loadResourcesByName$9(ResourceLoader.java:178)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableOnErrorNext$OnErrorNextSubscriber.onError(FlowableOnErrorNext.java:94)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.onError(FlowableUsing.java:124)
	at io.reactivex.rxjava3.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onError(SingleToFlowable.java:67)
	at io.reactivex.rxjava3.internal.operators.single.SingleMap$MapSingleObserver.onError(SingleMap.java:70)
	at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:47)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
	at io.reactivex.rxjava3.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:35)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
	at io.reactivex.rxjava3.internal.operators.single.SingleToFlowable.subscribeActual(SingleToFlowable.java:37)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15696)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing.subscribeActual(FlowableUsing.java:73)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableOnErrorNext.subscribeActual(FlowableOnErrorNext.java:39)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableDoOnEach.subscribeActual(FlowableDoOnEach.java:50)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15696)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:162)
	at io.reactivex.rxjava3.internal.operators.parallel.ParallelPeek$ParallelPeekSubscriber.onNext(ParallelPeek.java:156)
	at io.reactivex.rxjava3.internal.operators.parallel.ParallelRunOn$RunOnSubscriber.run(ParallelRunOn.java:273)
	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.xml.sax.SAXParseException; lineNumber: 57; columnNumber: 46; The entity name must immediately follow the '&' in the entity reference.
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at com.leidos.leaf.beans.parsers.JaxBUnexpectedAttributesValidator.validateUnexpectedAttributes(JaxBUnexpectedAttributesValidator.java:73)
	at com.leidos.leaf.beans.parsers.BaseJaxBParser.lambda$doParse$1(BaseJaxBParser.java:111)
	at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
	... 23 more

ryan.c.scott · July 8, 2021, 4:09pm

format="^[a-z0-9!#$%&'*+/=?^_{|}~-]$"`
It seems to not like &

mark.a.alvaro · July 8, 2021, 4:24pm

XML has special rules for what characters can appear literally in attribute string values. You’ll likely need to figure out what to escape/write using unicode or whatever it is style.

david.a.riggs · July 8, 2021, 4:40pm

Ampersand must be escaped as & when used in an XML attribute value:

format="^[a-z0-9!#$%&amp;'*+/=?^_`{|}~-]$"

See: Extensible Markup Language (XML) 1.0 (Fifth Edition)

ryan.c.scott · July 8, 2021, 4:52pm

Thanks Mark and David. One last bit, any way to get past the max length?

Caused by: com.leidos.leaf.beans.exception.beanmetadata.BeanMetadataValidationException: 2 validation error(s) detected.

<Detected in com.leidos.leaf.metadata.service.validation.beanmetadata.BeanMetadataValidationService.validateAttributeLength() at line: 384>
In Bean (Request-1) on Field (requestorEmail), For attribute (format), value length (426) is greater than maxLength (255) for value: (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

mark.a.alvaro · July 8, 2021, 4:57pm

No but you could register a custom ValidationString in your BeanMetadata to do that validation instead.

https://docs.leidos-leaf.com/stable/3.2/2.0-leaf-concepts/2.2-metadata-driven-development/beans/bean-metadata/custom-and-cross-attribute-validation/

I don’t think it’s likely that LEAF will change the max length allowed for formats, which could require migration from downstream teams to alter tables, and would seem to encourage regex’s greater than 255 characters being used which seems like a maintenance nightmare.

Again, though, all of that said, a ValidationString’s actual validation string can be 4000 characters. So you could register a static method to check that regex and then use that static method in a BeanMetadata’s ValidationString to check the email address.

mark.a.alvaro · July 8, 2021, 4:59pm

That said, feel free to put a request in to change the max allowable length for format, and it definitely can at least be considered if the workaround won’t work for you.

david.a.riggs · July 8, 2021, 5:26pm

I’ll respectfully offer the opinion that trying to validate every dark corner of the various email-related RFCs with a regular expression is a losing battle, and you’re still likely to wind up frustrating users by being over-strict and denying a legitimate address.

While I can’t predict your specific use cases, at the end of the day the reason we persist an email address is to ensure we can successfully email the user, and a regular expression doesn’t buy you that. Sending an actual email with a unique confirmation link for the user to receive and click is the only way to guarantee that they’re on the receiving end of the (valid!) address.

A minimal regex can help prevent syntax typos and obvious non-email-addresses, while the actual email round trip is the true litmus test. Something as simple as /^\S+@\S+\.\S+$/ can be sufficient for that purpose.

And before you ask, I’ll say that LEAF doesn’t have this email-roundtrip functionality built in, sorry!