Printer Friendly Version

Processing Dark Data

While dark data is an invaluable addition to an intranet it doesn't come without its challenges. Since dark data exists outside the corporate intranet, and often outside the knowledge of intranet owners, finding and porting this content to an intranet can be quite time-consuming.

There are two major issues associated with the consolidation of dark data into an intranet: Discovery and integration.

Discovery
Discovery is the process of finding and aggregating all applicable dark data to be included in an organization's intranet. But this process isn't an exact science. And it can be further complicated by content owners who, for various reasons, aren't willing to share their pool of information. In these cases dark data isn't hidden by obscurity, but rather deliberately concealed.

Although organizations with large amounts of heterogeneous content types can install a search appliance such as Google's Mini (for small- to medium-sized businesses) or Search Appliance (for large businesses), it does nothing to address dark data that exists in the form of hardcopies or user knowledge that has no corporeal home.

In order to maximize your chances of uncovering useful dark data, you need to run an information audit (the issue of running information audits will be explored in greater detail in an upcoming article). But don't try to create a single group to perform this audit. Instead, have individual intranet content owners conduct the audit from within their department or workgroup. Those who know their content best should be the ones responsible for the audits.

But the true key to dark data discovery lies in users' perception of the intranet. When users begin to see it as a productive, long-term business tool and not a flavor of the day, they will be more likely to share knowledge and bring some of this dark data to light. Without this cooperation your chances of uncovering dark data will be greatly diminished.

Integration
Once dark data has been discovered, your biggest decision will be what action to take upon this content — to decide the extent of your integration process. Will all of your heterogeneous content types be left in their native formats and simply linked to in your intranet or will this content be converted to one standard Web-based format? There's a case to be made for both.

Leaving all dark data in their native formats is certainly the quickest method, but you might be left with content inconsistencies in the long run. Intranets provide the entire user community with read-only content — much like Internet content. It's the content owners' responsibility to manage and update their intranet content. If dark data were to be left in its native format, say an Excel spreadsheet, users would most likely have to download the file from the intranet for local viewing. When this happens, there's a danger that the original file will be modified and re-circulated into the organization's information stream. The file could be changed and e-mailed from user to user — each making their own sets of modifications until there are dozens of copies. And when everyone is done with their copy, there will be dramatic differences from those files and the original "production" version sitting on the intranet.

Converting dark data to a standard intranet format is ideal but can become very effort intensive — especially if the format is not consistent with intranet content standards. Dark data contained within databases, applications, or in hardcopies can prove to be particularly difficult to convert. Dark data integration can include manually reformatting content, integrating applications with your intranet, and digitizing hardcopy documents via OCR (optical character recognition).

Closing Thoughts

The value of dark data isn't in question, but the trick is in finding it. It can be hidden in small corners of the company or long forgotten in some dusty old file server. And with all the value they can provide to the entire user community, dark data remains only accessible by a privileged minority — if even that.

Unfortunately, there's no quick fix or silver bullet. The actions required to find and integrate dark data into an intranet is dependent on the amount and complexity of this content. If you have a sense that you're in the possession of large quantities of usable dark data, focus your attention on finding this content rather than re-inventing the wheel.

But however you decide to approach this process of discovery and integration, the most important consideration when dealing with dark data is not to allow it to turn into a black hole, never to be seen again.