Consumer Data Right (CDR) — Lessons Learned from a Data Holder’s Conformance Test Suite (CTS) experience -Part 2
Manglu Balasubramanian
This article is part of the series of articles on Lessons Learned from a Data Holder’s CTS experience. The first article in this series is available here.
Note: Refer to the CDR Series section (at the bottom of this page) for links to other articles about Consumer Data Right.
The readers can look at the links in the reference section to learn more about Conformance Test Suite (CTS) and its role in the CDR ecosystem.
This article starts with the improvements that the CTS team made during the last few days/weeks (since the first article was published).
Improvement #1: Visibility of Request and Response
The update to CTS now introduces a Show Details link alongside the test steps.
This show details link provides the user information on both the request and response objects. This provides a quick and clear insight into the happenings. With the previous version(s), the users had to examine their logs and narrow them to the time window of test execution before the required information could be located for analysis and troubleshooting.
“Copy to Clipboard” is a default functionality that is available in most sites. It is good to see this addition available in CTS Portal.
The current copy functionality has its flaws as well! It introduces several new lines (as seen in Figure 3 above). The baseline approach for the copy functionality should be WYSIWYG (What You See Is What You Get)
Is there still room for improvement in this space?
(a) Removing the control characters in this Detailed view would be welcome. Though the required information is available, the control characters are an eye-sore and distraction.
(b) The Copy functionality introduces additional new lines. The new lines are annoying, and they should be removed.
(b) The Copy functionality includes the line numbers. Users would like to use pretty-print their JSON for better readability and unless the line numbers are removed the JSON formatters are not going to be of any help. If the line numbers are not copied when the users click on Copy that would be helpful to the users.
Now, let’s switch gears to the lessons learned section!!
Lesson #4: Awareness of CTS Versions and its implications
CTS has a few versions available, and it is important that the CDR participants are aware of which version they are currently executing their test scenarios.
Refer to the CTS Version History for information on what is covered in each version of the test suite.
There are a several obligations for a Data Holder for the July 2021 release. Some of these obligations are covered in CTS 3.1.0 test scenarios . They are not covered in the earlier versions such as say CTS 2.0.0.
If a Data Holder executes an earlier version of CTS (e.g., 2.0.0 or 2.1.0), then they will miss out on these additional test scenarios. Of course, the Data Holder’s testing processes would identify if there were any issues with their compliance obligations. However, testing them in an environment such as CTS can only be beneficial and ensures that things won’t break when the participant goes live in the CDR Register.
It is important to highlight that CTS does not cover all the obligations of the participant and the onus is on the CDR participants to ensure that their testing regime has the rigours and practices in place to ensure compliance.
What could we have done better ?
We were assigned a CTS version (2.0.0) and did not keep a close eye on the evolution of the test suite. We should have proactively sought to be on the right CTS version to ensure that we have good coverage of our Data Holder Solution.
What could the CTS Team have done better ?
(a) If a version is assigned to a CDR Participant and they haven’t executed any tests when a new CTS version is available, the CDR Participant should be notified and upgraded to the newer version with their consent.
(b) Newer versions should be communicated to the CDR Participants via emails and other mechanisms besides the newsletters. The CTS/Onboarding representatives should actively engage with the participant and give them adequate information and guide them to the right version of CTS.
(c) Displaying the version of the CTS in the CTS Portal. The users may or may not be aware of the version of CTS and showing this prominently can only be beneficial (as there are at least 4 active versions in play currently 2.0,0, 2.1.0, 3.0.0 and 3.1.0).
(d) In the current environment, we have CDR participants who are completing CTS in any of these 4 versions that are currently in-play. Reducing the surface to just a “n” and “n-1” CTS version (i.e., 3.1.0 and 3.0.0) could be an approach to adopt/embrace.
Lesson #5: CTS specific code-changes are required which must be rolled back for Production release
The interpretation by the CTS team in at least one instance appears to be different from what the Data Holders have interpreted.
“x-fapi-auth-date” is defined as a required header for all resource calls (customer present and unattended).
However, the CTS test case for Get Accounts does not send this header as part of its request. If a Data holder had implemented based on the spec definition (such as what we did) then the CTS reports this as a failed test scenario.
Sample request from CTS for this scenario is shown below. It sends only the Authorization and the x-v headers.
{
“RequestType”:”Outbound”,
“Request”:{
“Method”:”GET”,
“Url”:”https://dataholder.com.au/cds-au/v1/banking/accounts",
“Headers”:{
“Authorization”:[
“Bearer 7TokenABC”
],
“x-v”:[
“1”
]
}
},
“Response”:{
....
}
}What could we have done better ?
Absolutely nothing.
What could the CTS Team have done better ?
(a) CTS team would have observed this issue with other Data Holders and a knowledge article and changes (if any ) should be published clarifying things if the Data Holders have mis-interpreted this requirement. This would have saved a lot of time (and energy) for the Data holder teams.
Lesson #6: Using two test versions (One for Compliance followed by a Voluntary test)
Most of us have heard this phrase: “One can’t have their cake and eat it too”. However, this idiom isn’t true with respect to CTS. Participants can be at a CTS Version (e.g., 2.0.0) for compliance purposes and then use a newer CTS Version (e.g., 3.1.0) for voluntary test.
Completing the compliance in 2.0.0 allows us to get the production certificates and proceed with the rest of the on-boarding activities. At the same time, we can test our solution with CTS 3.1.0 before we head to our production on the 1st of July 2021.
CDR Participants are allowed to use the CTS environment in an on-going fashion even after go-live which helps them to validate their solution changes actively in their Sofware Development Life Cycle.
What could we have done better ?
Nothing really. We were lucky to be in the first non-major ADI CTS version which has fewer test scenarios compared to the newer versions.
What could the CTS Team have done better ?
(a) CTS team could improve their communication about Voluntary CTS testing. Information and articles on how participants (both the incumbent CDR Participants as well as new participants) can take advantage of Voluntary CTS would be beneficial to the CDR community.
Lesson #7: Obtaining Production Certs after 80% success with CTS
The older version of the Participant on-boarding guide (Version1.1 Dec 2020) stated two different positions about when a participant can request a Production Certificate.
In Page 12 (7. On-boarding process — Step-by-step instructions) , it states the following:
For example, if your production environment has been provisioned you may provide your production details or request a production certificate in the Participant Portal before completing the Conformance Test Suite (CTS).
While Page 31 (10. Appendix B: Testing guidance) states
All testing activities should be completed before the production certificate is requested (7.9).
The newer version of this Onboarding guide (Version 1.2, June 2021) removes this ambiguity by removing the section from Appendix B.
A Participant cannot request for a Production Certificate at any time (of their choosing) before completing CTS. They are only allowed to make the request for Production Certificate(s) after they pass 80% of their CTS test scenarios
What could we have done better ?
(a) Keeping a close eye on the newsletters to see if newer versions of the on-boarding guide are available. It is however difficult to read through the entire document and see which changes are applicable/relevant to us!
What could the On-boarding Team have done better ?
(a) Update the On-boarding document to clearly articulate that 80% of the test scenarios must be completed successfully so that it is clear to the participants.
(b) Highlight the key differences between versions of the On-boarding guide so that the changes are available at a quick glance to the participants.
Conclusion
CTS has been an interesting experience for our team (and other Data Holders based on informal conversations). It did help us identify a few issues/defects in our solution which weren’t identified in our internal testing.
Open ID Foundation has a Conformance Suite which covers OpenID Connect, FAPI & FAPI-CIBA Profiles, including Open Banking UK & Australia CDR security conformance certification. Data holders can and should take advantage of this Suite alongside the mandatory CTS to ensure that their solution is robust and compliant.
I am sure that the CTS and ACCC On-boarding teams will take on board some of the suggestions listed in this article to improve the experiences of CDR participants in future CTS activities
Hopefully the other Data holders in the CDR eco-system find these lessons useful.
Acknowledgements
I would like to thank the ACCC CTS team for their support and assistance during our CTS activities.
References:
- CDR Conformance Test Suite
- Official CDR Website
- CDR Support Portal
- July 2020 CDR Ecosystem Production Incidents
- CDR Participant Test Strategy
- CDR Participant On Boarding Guide V1.2
- Open ID Conformance Suite
CDR Series:
- CDR — What’s it all about?
- CDR — Who are the Intermediaries, Why are they needed and What services do they provide?
- CDR — What’s in it for me? Queries from Jim (a typical consumer)
- CDR — Consent and it’s characteristics!!
- (CDR) — Lessons Learned from a Data Holder’s Conformance Test Suite (CTS) experience -Part 1