# Four data challenges to overcome before you’re MiFID II compliant
Author:  Pal Sinha, Barnali 
Author URL: https://financedigest.com/author/pal-sinha-barnali
Published: 2017-11-18
Category: FINANCE
Category URL: https://financedigest.com/category/finance
Meta Title: MiFID II Compliance: Challenges and Solutions for Financial
Meta Description: Learn about the challenges and requirements of the Markets in Financial Instruments Directive II (MiFID II), and why many institutions are struggling to comply.
URL: https://financedigest.com/four-data-challenges-to-overcome-before-youre-mifid-ii-complianthtml

![undefined](https://prod.superblogcdn.com/site_cuid_cm5qst7v3003gwirgwqtxn8i8/images/fd181117-2-1736843231391-compressed.jpg)

_By **Daniel Roberts**, principal sales engineer,_ [MarkLogic](http://www.marklogic.com/)

Right now, financial market institutions are gearing up to meet the Markets in Financial Instruments Directive II (MiFID II) deadline of January 2018. MiFID II is not only another reporting requirement, it is a regulation that involves technical, political and economic requirements being put in place – requirements that run to literally thousands of pages. The long-term consequences are not entirely knowable at this time – but it is clear that non-EU firms will be impacted as well when doing business with EU entities.

The key issue is that these comprehensive regulations (with all their vagaries) and the threat of massive fines for non-compliance are creating [fault lines](https://www.financedigest.com/analysis-crisis-radar-falls-on-fault-lines-in-europes-commercial-property.html "Analysis-Crisis radar falls on fault lines in Europe’s commercial property") among executives, bank risk officers and the developers tasked with complying.

As regulators bid to create more open and transparent financial markets by tackling the “ [under-regulated and opaque aspects of the financial system](https://www.ft.com/content/ae935520-96ff-11e7-b83c-9588e51488a0)”, which will likely generate tens of terabytes of increasingly complex data, many institutions are reportedly a long way behind schedule in their efforts to meet this regulation.

This is in part because MiFID II is much larger in scope compared to the preceding regulation. It requires a different approach towards leveraging data in operational and transactional transparency to ensure investor protection and a fair and uniform functioning of the European financial [marketplace](https://www.financedigest.com/why-embedded-finance-will-be-integral-in-the-future-of-b2b-marketplaces.html "Why embedded finance will be integral in the future of B2B Marketplaces").

However, addressing MiFID II is a massive data integration effort, which most financial services [firms](https://www.financedigest.com/digital-marketing-for-finance-and-insurance-firms-is-changing-are-you-ready.html "Digital marketing for finance and insurance firms is changing — Are you ready?") are not yet geared up for. In terms of data governance and compliance readiness, this regulation will impact transaction [reporting of all financial instruments traded in Europe](https://www.financedigest.com/transforming-vat-how-is-europe-modernising-tax-reporting.html "Transforming VAT: How is Europe modernising tax reporting?"), post-trade transparency, and real-time data delivery. It will also require better execution with a specific need to be able to reconstruct past events and provide all types of communications related to transactions. Many financial institutions currently have more than 20,000 relational databases within their organisation, this causes huge data fragmentation and silo problems. Often what we [see when these organisations approach a new data regulation like MiFID II is that their solution is to turn](https://www.financedigest.com/in-u-turn-iea-sees-world-weathering-lost-russian-oil-supply.html "In U-turn, IEA sees world weathering lost Russian oil supply") to yet another relational database, which further exacerbates the problem.

Patching holes won’t work. In the relational database world, current data challenges can be addressed, but thereafter organisations are required to [predict the future](https://www.financedigest.com/covid-19-impact-on-attitude-and-heading-reference-systems-market-volume-analysis-future-prediction-industry-overview-and-forecast-2028.html "Covid-19 Impact On Attitude and Heading Reference Systems Market | Volume, Analysis, Future Prediction, Industry Overview And Forecast 2028") to keep up with the changing nature of their data. This is a flawed approach; relational databases don’t cope well in environments where data structure is endlessly changing. If these organisations could effectively get a 360-degree view of their data and implement a robust data governance strategy, regulatory reporting would be a breeze, yet most organisations fail to realise this panacea.

Many of their challenges will [revolve around](https://www.financedigest.com/the-soft-skeletal-implants-market-revolve-around-digitally-driven-innovation.html "The Soft Skeletal Implants Market revolve around digitally driven innovation") specific requirements in MiFID on reporting and data delivery. These include:

**Governance**

[Banks have to be able to generate the right information for the regulator](https://www.financedigest.com/be-simple-in-bank-capital-bank-of-england-tells-global-regulators.html "Be simple in bank capital, Bank of England tells global regulators"), and that creates the need for a compliance library and logic that can be reused. With many consumers of information internally and externally, logic that defines fields for reporting is often different across reports and data silos. As such, data integration is therefore difficult with a relational database, as everything is focused on the syntax not the semantics – i.e. the capability to discover the facts and relationships in data, and understand the context of those facts. In a SQL database, there is no semantic mean defined for data items across the organisation and, consequently, data governance is often limited to it’s difficult to merge data for accurate, [timely reporting](https://www.financedigest.com/credit-suisse-executives-reassure-investors-after-cds-spike-financial-times-reports.html "Credit Suisse executives reassure investors after CDS spike, Financial Times reports"). Rather than encourage an army of business analysts to re-process information for each data delivery, a better [approach is to store and re-use methods from one central repository and associate both to the data sets](https://www.financedigest.com/uks-finance-minister-sets-out-new-economic-approach-to-finance-bosses.html "UK’s finance minister sets out new economic approach to finance bosses") and results. The methods and code can then be updated over time independently of data updates.

**Transaction Reporting**

For transactional reporting, MiFID II requires financial firms to send information for all eligible trades within a regulated [market](https://www.financedigest.com/u-s-markets-regulator-takes-aim-at-coinbase-lending-product.html "U.S. markets regulator takes aim at Coinbase lending product") to a regulator as soon as practicable. This means that trades need to be reported to an authorised reporting mechanism (ARM), which is responsible for packaging up the information and providing it to the local competent authority.

Siloed controls and inflexible legacy technologies, are, in most cases, unable to handle multiple sources of data, and the number of fields for transaction reporting has significantly increased. Traditional relational databases struggle to process the data when some information doesn’t fit the required fields. That data will be rejected, resulting in inaccuracy of reportable data. The possibility to ingest the data as-is and reconcile the data in place represents significant advantages and expedites the route to compliance.

**[Post Trade](https://www.financedigest.com/euro-zone-posts-record-trade-deficit-in-august-amid-soaring-energy-prices.html "Euro zone posts record trade deficit in August amid soaring energy prices") Transparency**

In a world of [post trade](https://www.financedigest.com/uk-housing-boom-may-derail-post-brexit-trade-dreams.html "UK housing boom may derail post-Brexit trade dreams") transparency, trades are sent to the ARM in real-time through an authorised publishing authority (APA). These authorities are responsible for making public trades related to bonds, [structured finance products](https://www.financedigest.com/strong-performance-of-investec-lowes-structured-product-maturities.html "Strong performance of Investec / Lowes structured product maturities"), emission allowances, and derivatives. This is a tough requirement to satisfy.

With MiFID II, [companies must demonstrate effective oversight and control](https://www.financedigest.com/hospital-acquired-infection-control-market-global-leading-companies-analysis-revenue-trends-and-forecasts-2027.html "Hospital Acquired Infection Control Market Global Leading Companies Analysis, Revenue, Trends and Forecasts 2027") over policies and procedures which govern all communications. Moreover, there is a requirement to [supply](https://www.financedigest.com/russias-putin-indias-modi-to-discuss-trade-food-supplies-on-friday.html "Russia’s Putin, India’s Modi to discuss trade, food supplies on Friday") regulators with communications associated with a specific trade and being able to reconstruct history of trade cycle events. Discovery or knowing what information you knew and when and how you knew it has changed over time. Now, a bitemporal view – meaning the ability to track when events occurred (valid time) in combination with when they were recorded (system time), for auditing and tracking – of data lineage should be a critical component of your regulatory reporting infrastructure.

Retail banks are very familiar with the issue of discovery, but with many messaging and data formats the ability to identify information across not just business silos but information silos (email, messages, texts, office documents etc.) is a serious challenge for investment banks. Traditional databases store unstructured data as CLOBs which, as a unit of information, is not sufficiently granular and cannot resolve complex search and discovery questions to prove how you knew the information you knew and when. Even after the search problem is solved, managing this information as part of an information life cycle management (ILM) process becomes singularly [important to prevent operational costs](https://www.financedigest.com/german-gas-import-costs-surge-153-in-first-half-as-imports-drop-24.html "German gas import costs surge 153% in first half as imports drop 24%") escalating.

**KPI Reporting**

To safeguard against Algo [trading as well as trading beyond limitations and to ensure investor](https://www.financedigest.com/how-to-begin-investor-trading-in-australia.html "How to begin investor trading in Australia") protections against inducements, organisations will need to generate reports on various Key Performance Indicators (KPIs). As part of the mandate to be able to reconstruct past events and unwind any trade with communications, a second timeline will be imposed to answer the question, “What did we know when the trade was made and how can we prove it?”

Each of these challenges will [impact every aspect of a bank:](https://www.financedigest.com/digitisation-of-banking-the-impact-of-digital-money.html "Digitisation of Banking: the impact of digital money") operations, trading and technology. The conventional wisdom will be to think “data warehouse,” “reporting solutions” and “ETL” when trying to tackle these problems, but this won’t address the underlying challenge that ETL and traditional [technology provide](https://www.financedigest.com/inmobi-wins-gold-silver-bronze-mma-smarties-mena-awards-named-technology-provider-of-the-year-for-fourth-consecutive-year.html "InMobi Wins Gold, Silver, Bronze MMA SMARTIES MENA Awards, Named “Technology Provider of the Year” for Fourth Consecutive Year") the agility needed for organisations to change in time for January 2018.

Furthermore, organisations that view MiFID II as merely a compliance exercise will cement its failure. It is vital that financial firms establish a sound application framework that will not only model and bend to meet MiFID II’s requirement, but which will also act as a single [reporting platform to help them meet any regulatory need in the future](https://www.financedigest.com/chromoendoscopy-agents-market-2031-report-details-the-future-development-manufacturers-trends-share-size-and-forecast.html "Chromoendoscopy Agents Market 2031 Report Details the Future Development, Manufacturers, Trends, Share, Size and Forecast"), be it MiFID II, the Dodd-Frank Act or EU GDPR.

Solving these data [challenges](https://www.financedigest.com/meeting-the-data-challenge-in-financial-services.html "Meeting the data challenge in financial services") requires a database that empowers you to integrate all of your data with minimal disruption to your business. We recommend a [design approach for reporting solutions](https://www.financedigest.com/product-design-verification-and-validation-solution-market-2021-present-scenario-and-growth-prospects-2025.html "Product Design Verification And Validation Solution Market 2021 | Present Scenario and Growth Prospects 2025") that ensures agility and flexibility. The solution should deliver a regulatory [reporting platform that incorporates best practices and operational effectiveness and allow for adaptive growth](https://www.financedigest.com/amino-acids-market-trends-size-share-growth-forecast-report-2025.html "Amino Acids Market Trends, Size, Share, Growth Forecast Report 2025") in scope and scale. The design goal should not be to remedy one-off [reporting requests but to build in a capability to respond to emerging](https://www.financedigest.com/n-korea-reports-first-covid-outbreak-orders-lockdown-in-gravest-emergency.html "N.Korea reports first COVID outbreak, orders lockdown in “gravest emergency”") requirements with relative ease and cost efficiency.

As several financial firms have already discovered, there is an easier way to bring all these data silos together. Using an operational data hub or trade store approach – built on a flexible, enterprise-grade NoSQL database with [integrated Google-like search – can pay](https://www.financedigest.com/impact-com-closes-2021-with-new-google-pay-integration-150m-in-funding-and-51-increase-in-clients-yoy.html "impact.com closes 2021 with new Google Pay integration, 0M in funding and 51% increase in clients YoY") dividends for data challenges where the data and requests from regulators change over time.

[ABN AMRO](https://www.financedigest.com/abn-amros-q3-profit-jumps-as-interest-income-recovers.html "ABN Amro’s Q3 profit jumps as interest income recovers") is using the next-gen MarkLogic database platform to bring vast amounts of unstructured and structured trade data into one central, easily manageable operational trade data store. With a consistent, transparent record of every order and trade event, ABN AMRO is able to comply with internal and external reporting requirements in a fast and [flexible manner now as well as in the future](https://www.financedigest.com/the-future-of-payments-flexibility-regulation-and-bespoke-models.html "The Future of Payments: Flexibility, Regulation, and Bespoke Models").

The [bottom line](https://www.financedigest.com/how-high-performers-can-increase-a-bottom-line.html "How high performers can increase a bottom line") is that MiFID II requires new organisational processes and thinking, research and development in how to support it, and a departure from rigid technologies that make adaption costly and likely, not even feasible.


---
This blog is powered by Superblog. Visit https://superblog.ai to know more.
---