4 minute read
Data intelligence is the ability to transform massive data sets into intelligent data insights using semantics, machine learning, and artificial intelligence technologies.
Such processes typically improve the services and investments that corporations have done in their data.
There are various kinds of data sets that corporations have within their ecosystem.
Some of the data sets are structured, some are unstructured, for example word documents, PDF documents, PowerPoint spreadsheets – all of these are classified as unstructured data.
Circa 80% of the world’s data right now is unstructured. In addition, there are structured databases like relational databases where data sets are organised in a more normalised format.
Data intelligence starts with organising your data, then classifying it and after that extracting information from that data or rather data points which give new insights or meaning to your data.
This kind of information extraction is done using various machine learning models or fuzzy matching using custom ontologies and taxonomies.
Why is data intelligence particularly important in the content of regulations and compliance policies?
There are various use cases that corporations are trying to solve in a compliance and regulatory environments.
Regulated industries have heavy reporting needs before compliance bodies. Typically, there are 4 use cases that are common in regulatory environment:
- Facilitating e-discovery and reporting on what they knew when they knew it, or how they arrived at a decision at a given time. This also relates to time-stamping data.
- Enabling regulatory compliance. For example, for GDPR there may be a request to remove all personally identifiable information the organisation doesn’t have consent for.
- Ensuring automation, for example for automated the report creation
- Increasing visibility, for example on product performance or post market surveillance.
There are various ways to make use of data-driven intelligence applications.
They include trader surveillance platforms, digital publishing platforms, various semantically linked regulatory compliance applications, cognitive search analytics platforms, clinical research data hubs.
For example, the EPA – the Environmental Protection Agency in the agro-chemical space – has come out with the EPA 2035 goal to stop et all mammalian testing.
And it becomes more and more important to have these kind of intelligent data platforms built so that companies are able to comply with such regulations.
How do we help our customers achieve data intelligence?
Datavid as a company is focused on building data intelligent platforms.
We have a lot of tools as well as services that we have built, which makes it easier for organisations to collate their data, extract information from their data, then organise and store the data in proper indexes, on top of which you can build custom visualisations tailored to specific needs.
This approach improves the quality of data so that users are able to search and find this information using their domain vocabularies, as well as share this information with downstream analytics applications or reporting applications.
Essentially it makes the data more structured, which then allows one to easily report on the data to regulatory bodies.
We have expertise in proprietary and open source NLP software in text extraction and table extraction.
We have built connectors for common repositories like SharePoint and common BLOB storage like S3, which also enables more downstream use cases like data science and data analytics on extracted data.
Our expertise in working with unstructured documents and multi-model databases gives us an edge and ability to transform unstructured data into a normalised structured format, with section identification, entity extraction and enrichment and searchability.
This approach allows us to build a data-centric application quickly.
What are the practicalities of working on a data intelligence project with customers? What is a typical composition of a team working on a project? What are the timelines?
Because we have already created some reusable microservices in this area enabling teams to gather structured and unstructured data and put it into a multi-model database, we really expedite the timeline.
We have an orchestration engine and expertise in AWS serverless technologies as well as multi-model NoSQL database like MarkLogic which allows data to be loaded as-is and then converted into a normalised format.
Integrating with underlying data repos itself might take two, three months, but we already have pre built microservices in this area, which we can do within few days.
The timeline agets reduced by at least 50-60%.
With a small agile team of one BA, one QA, and 2 developers we can really get the whole timeline composed from a one- to two-year project down to three to a six-months project.
What are the regulations we currently help our customers comply with?
As mentioned previously, there are few use cases for data intelligent platforms.
One, of course, is Discovery.
Once you’ve indexed the data and you have enriched it, you’re able to discover and draw better meaning out of it.
And that’s also a very important use case for compliance in R&D industries where there is a lot of research and development. In the Life Sciences and Pharma we have built applications and reporting that helps in EPA and MDR regulations.
For example, for MDR which came into force in May 2021, medical device manufacturing must adhere to strict guidelines to ensure their products are safe to use.
To do this they need to do post market reporting in a more granular and automated way. This is where data intelligence platforms help in expediting the business needs around reporting.
There are other examples around GDPR and FCA.
How do we help customers accelerate time to delivery and time to value?
For this purpose we have reusable microservices to collate and organise data.
We have Alpha version of our platform Rover already available.
If you were to summarise three main take-aways from this interview, what would they be?
It is important to draw a conclusion on why Data Intelligence Platforms are so important and they can help with, including:
- Building new e-commerce applications which open new revenue streams
- Ensuring regulatory compliance and reporting needs.
- Providing substantial benefits and value by improving your workforce’s efficiency and productivity.
I would encourage you to talk to a Datavid Consultant and we can help!
You can also watch a recording of our webinar which served as the basis for this interview at this link.
Frequently asked questions
