Bigeye

Leading the discovery and design of Bigeye's new Lineage product.

Bigeye is a data observability company that gives data quality engineering teams the ability to measure, improve and communicate data quality. Data pipelines are complex and it can be difficult to quickly gain a big picture understanding. Users need lineage to understand relationships between the data in their system.

Bigeye partnered with Underbelly to have our team lead the discovery and design for a multifaceted lineage feature that would differentiate Bigeye. The lineage feature would help Bigeye users to see their data relationships, identify the root causes, and understand the impact of data quality relationships.

Data lineage helps ensure that accurate, complete, and trustworthy data is being used to drive business decisions.

Business goals

To differentiate themselves in the data tool market, Bigeye needed to create a foundation for lineage that could be scaled to provide more targeted support for data issue resolution workflows. With Bigeye, our team created three main separate lineage features: catalog lineage, root cause analysis, and impact analysis. Catalog lineage would stem the tide of sales losses where lineage was cited as a deciding factor while root cause analysis and impact analysis would provide users with better, more targeted issue resolution recommendations.

Problem statement

How might we clearly show users the origin of a data issue?

Looking upstream in a data pipeline in root cause analysis, alerting issues may only be symptoms to the core problem. Sometimes the root cause of the problem stems from a dataset that is upstream in the lineage.

How might we clearly show the impact of an issue on affected downstream processes and datasets?

Looking downstream in impact analysis, when multiple pipelines and data issues are detected, determining which issues to address first requires understanding of what downstream elements are potentially impacted.

Solution

We performed discovery by conducting competitive analyses and talking with existing Bigeye customers. We then designed catalog lineage, root cause analysis, impact analysis, BI (business intelligence) tool integration, and column-level lineage features, all while getting frequent feedback from customers. To facilitate handoff, we created detailed design annotations and videos. Through the engagement, we collaborated with project managers, front-end and back-end engineers, product and brand design teams, and sales to ensure the lineage feature would align with the existing Bigeye product and differentiate it from competitors.

The unique approach we implemented for designing complex lineage is to show just the right amount of information to help the user understand a data issue while also giving a clear recommendation on what action to take next.

In root cause analysis, the lineage graph presents recommendations and navigation links to what Bigeye infers are the root causes of a particular issue. The user can jump to those root causes and tackle the rest of the root cause analysis from there with confidence that resolving issues there will be the most impactful.

Results

The lineage feature not only aligns with the existing Bigeye product but also elevates Bigeye above competitors to offer a unique take on lineage - giving users more targeted issue resolution recommendations. Lineage brought significant business value to Bigeye and helped differentiate it as a leader in lineage visualizations in the market.

Additionally, our team hit all project deadlines. We were able to adapt to working in the complicated data space to deliver a feature that has already brought new customers to Bigeye.

Insights

With every lineage feature and iteration, we considered how scaling would affect the lineage look and function and how users interact with it. To make large pipelines more readable and easier to view, we introduced zooming features, clear hover interactions, and stoplight colors to represent table health.

We collaborated with engineering teams to develop a system of accessibility features that support screen readers and allow users to tab through lineage while keeping the graph’s integrity. Building for accessibility will make product analytics more consistent.

No items found.

Credits

No items found.

Let's talk

about your big idea.

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form.