5 minute read
Large companies and organisations generate massive amounts of data under different domains during their operations.
And if they have been in the business for some time, it probably means that their business data faces some degree of information silos.
Information silos can occur when different components of the same business have data that is incompatible, hence the organisation is unable to make proper use of all the data they have.
Long running companies find the need to solve this information silo problem and bring together data from their different departments to make meaningful conclusions.
Building interfaces on top of a MarkLogic backend
Marklogic database systems have very powerful features to ingest, process and transform data from a variety of sources and make them readily available for business needs.
There is a lot to like and talk about how MarkLogic solves the information silos problem. Let's focus on the UI, which comes after the fundamental solution of bringing data together in a compatible manner from all the different sources.
Let’s look into how proper user interface development can enable more capabilities from the business data brought together from various sources by exploring a solution we developed.
A real-world example of consolidated data consumption
Once we have got to the stage where we have consolidated the data from information silos under a single roof, comes the part about consuming or using it meaningfully.
This is the part where humans interact with the previously unavailable information and try to make use of it for some benefit.
We’ve dealt with such a solution for a client where we had to bring in product data together from various systems and consolidate the results to be reviewed by subject matter experts (SMEs) to be ready for consumption by other systems.
Data coming from different systems rarely has the same schema, structure or even the same information. Here, it was crucial that the end result would output more accurate and complete data than all of the inputs.
The solution had to be a source of truth about their product data and should have reliable information, which will then feed into a variety of applications.
This requirement meant that we needed to develop a solution to view, review and correct the resulting data manually by experts in their respective fields.
Building a UI for data-intensive applications
We needed a good UI where users could access the data for viewing, experts could come in and review and modify the values which they find wrong, and the people reviewing could benefit from the context of where this data came from in their decision making process.
The solution was to have an interface which would present all the different versions of data that came from the different sources and then the result of all that combined.
We developed a tabular UI in Angular leveraging its reactive form capabilities and reusable components to provide the experts with a UI having enough context for them to analyse the quality of the resulting data and to make edits if necessary.
The interface has a flexible layout with different columns for instances of data from each source and a form input for the resulting data, where the expert may edit information.
Designing a user-friendly data reviewing experience
Even though the user is supposed to be a subject matter expert, the UI was made to be easy to use.
Their expertise is required to know which of the sources provides better quality data and to find out possible low quality or erroneous data. The UI shows visually which source it selected for each of its data points to form the final result.
If the user feels like a different source provides better value for the same data point, they have the option to simply click on that item to have it populated in the result.
If the data points coming from each system are different or incomplete, the resulting data can also have different number and type of data points depending on the available data sources from which it was consolidated.
This means a very flexible form input which can grow, shrink, or transform to match the structure of the item which it currently shows.
Why use Angular for data-intensive projects?
We used Angular UI as the frontend interface technology to generate these context views and the form to match the number and structure of the data points coming in.
The solution was not just a single entity of product data from different sources.
The review process involves interconnected data (all consolidated from different systems), where the reviewer would be able to navigate to the usage data of the product and also to a detailed view with the products’ internal details of constituents, substances, etc.
The review UI adapted to this navigational behaviour where the user would start with some product details and then move on to review its usage data for various applications and also to its internal constituent material information.
Another interesting fact about solving the information silos is that—once the data comes together—the result is a rise of use cases and applications for the data itself.
Enabling these use cases is the key reason to invest in such a solution.
How relationships connect the dots between each data point
Our project also had the potential to show users the relationships between data points for them to draw more insights, and also to help as a reference when developing new products.
They needed to see how all these data points relate to one another, and to have a visual representation of the products, its uses, its constituent materials and internal details, and how this would relate to various different information pieces which can be tied to that product.
A graphical user interface was proposed as a solution where a directed graph with nodes and edges would represent each data entity related to a product and the user could get to know more details by interacting with nodes of that graph.
We used an angular wrapper of Apache E-Charts library to provide the graph interface customising it extensively to meet the requirements.
The users could tap on the nodes to see more information on a details panel to one side and had the option to selectively display the relationships they were interested in only.
Users could use this view to have a more insightful understanding of the data and help them visualise the relationship between various data points.
Deriving new product information from existing data
Once the previous features were delivered, a new requirement came up.
We now had a library of more complete and accurate data than all the different sources that existed previously, so to create new product information, the client wanted to use this system.
In this case, the data entry process could go through the same review system we built, enabling them to compare the newly-created dataset with existing ones from other sources.
We designed an easy to use UI which enabled the data entry to match the rich data schema that was evolved from combining data points from all those sources.
The main challenge was to keep it simple enough while providing all the flexibility and capability needed to logically lay out the three data entities we had by now:
- Product details;
- Product Constituent material details, and;
- Usage details.
All these three data structures were presented in an intuitive way, with features like cloning and search and filter to reduce the manual labour of data entry.
We had the requirement to populate the possible values of some data points from a reliable repository so that there is less chance of manual error. All of these features were delivered by leveraging Angular’s components, modularity, and reactive dynamic forms.
Silos aren’t just a data problem
The project started as a solution to information silos, and while MarkLogic features solved the initial and fundamental problem, the user interface was just as important in enabling interaction with the newly available information.
The project has grown in its capability in an agile way thanks to the effective use of features provided by Angular, with good design and planning.
Users wanted more usability on top of the data and our project delivered all these requirements while retaining a simple UI.
The team at Datavid was able to create new solutions with UI for the project by seamlessly developing capabilities needed via the MarkLogic backend.
Having the right data in place is only half the story. Enabling humans to make sense of it and driving new capabilities from it is why we started this journey.
Frequently asked questions
