IBIS: A Powerful, Drupal-based Biosurveillance System

Ben PritchettI'm very excited about Joshua Lee's talk on the Drupal-powered International Biosecurity Intelligence System (IBIS) at DrupalCon 2015. Though I'm no biosecurity expert, the aggregation methods and process workflow for gathering biosecurity information is relevant to many industries. In his talk, the technology for creating this data aggregation system will be covered, as well as how the Drupal community can both benefit and contribute to this project.

To give a quick background of how IBIS works, the system grabs articles from different sources, which are either trusted or need human verification. These articles, once verified by a human or being from a trusted resource, can then be brought into the IBIS network for distribution to IBIS members. The human intervention on this system is kept low to allow for information to be reported quickly to biosecurity experts for decision making.

I sat down with Joshua Lee to fill in some of the blanks and give us a more detailed background of IBIS.

Q: What were some of the issues you encountered when developing this system? Do you think these were unique to the workflow needed for biosecurity reporting?

The most significant challenge I encountered when developing the system was building a high performance website. As a news aggregator, IBIS is collecting about 10,000 news articles using the Google API and 5,000 research and industrial articles from different RSS feeds every day. Most of my time was spent on improving the algorithm and server architecture to make sure the site performs well when it is collecting the news articles. This would be a common challenge for any sort of news aggregator.

The animal related biosecurity articles are mainly discovered using a combination of the host, pest and disease names. This requires the IBIS query builder to compose a google search query using a special syntax. Because the search pattern is unique for IBIS, it stops the technology from being used in other industries. My current research is to make the IBIS search process generic so the IBIS algorithm can be adopted widely in other areas.

Joshua LeeQ: How has open source enabled the creation of IBIS? When looking at content management platforms, why was Drupal chosen?

The Australian Government Department of Agriculture decided to use open source technology to build IBIS from the beginning. IBIS means to benefit not only the Australian government but also the global biosecurity community. Increasing numbers of organizations in different countries have been using IBIS for different purposes. Open source allows anyone in the community to contribute to IBIS which also opens the possibilities for how IBIS can be used.

Drupal was selected because it's one of the most popular open source CMS platforms and has a growing community. IBIS will be developed by different contractors and companies, and Drupal will minimize any transition period by also providing solid support from the community.

Q: What considerations were there when determining whether a process should be automated or handled by an Article Evaluator/Search Editor?

IBIS is designed to bring as many relevant articles as possible for the users to analyze. The article editor only needs to manually submit an article to the system when the article cannot be detected by IBIS.

The article evaluator's role is to trash the noise and promote the articles they think may be important for the subscribed users. This process needs scientific researchers with biosecurity background to be able to identify the valuable articles.

Q: Is there a plan to package the technology IBIS uses (as a Drupal module or quickstart, for example), so a company or organization needing this workflow can easily use this technology as well?

There have been other organizations from different industries showing their interests in the technology IBIS is using. We are happy to share the codebase and the algorithm to the community. In doing so, we may also benefit from community contributions. Knowing that IBIS specific functions and structure need to be removed in order to create a generic system, we have plans to research the possibility of converting IBIS into a Drupal distribution or a set of Drupal modules. We will publish the modules on drupal.org when the work is finished.

Q: What additional automation is being planned for IBIS?

The next stage of IBIS development will focus on the data analysis and the social media news aggregating functions. Existing IBIS subscribers benefit from the daily digest. We plan to bring subscribers into the site by creating the visualization data analysis tool for users with different interests. The tool will be able to generate diagrams to show the context between pest and diseases in different countries. A social media aggregator will become optional for users who are interested in detecting trending topics on different social media.