Infinit.e - developing core infrastructure

Overview

Note that this section describes the not-common activity of building and modifying the core platform, not the more common activity of creating plugins of various types - for details on the latter activity, click here.

All of the code and libraries used in the Infinit.e platform is available under the Affero GPL (a few utilities are licensed under Apache 2.0, see the individual headers for more details), hosted on Github:

• Some install scripts and binaries needed before Infinit.e is installed, the Infinit.e Pre-requisites.
• The code, libraries and install scripts for Infinit.e itself.

It is important to note that the Infinit.e Open Source software is currently developed under the Cathedral Model, which is to say that Ikanow will develop the majority of the software in between releases, and will be responsible for all of it. That said, we welcome patches and community contributions and will endeavor to build them into the next release whenever they have value for the community.

Before distributing modified software, please check the terms of the open source licenses to double check you are compliant with any restrictions and requirements.

Getting the source code

The links are above, simply fork a new branch from github (useful web page discussing the "fork and pull" development model).

Setting up a development environment

This section describes how to set up a Windows or Linux or Mac development environment for Infinit.e:

• Install a JDK, eg 6u22+ (we currently use 6u30)
• Install Eclipse J2EE
• (Additional step for developing widgets - see subsection below)
• Eclipse tomcat plugin (google for "sysdeo tomcat plugin"), v321, unpack directly into plugins directory
• tomcat 6.0 (need to specify TOMCAT_HOME in eclipse tomcat plug-in configuration AND make sure that the Tomcat Version is set to 6.X in Preferences -> Tomcat within Eclipse)
  
  • We don't normally use tomcat as the debug platform for the API, so this is now mainly only necessary to get some libraries that are dependencies in the API server (see below)
• Get branches of the following projects, from Github:
  • deploy/infinit.e-config.rpm (for config files - necessary to run the code, not to compile it)
    • Then fill in the "infinite.api.properties.TEMPLATE" and "infinite.service.properties.TEMPLATE" - see installation guide for more details.
  • core/infinit.e.api.server (server for the API)
  • core/infinit.e.core.server (server for harvesting and custom processing)
  • core/infinit.e.data_model (contains all the object format specifications as Java beans ("pojos"), plus libraries to transform them between API, data store, and index)
  • core/infinit.e.harvest.library (all the logic for harvesting, extraction, and enrichment)
  • core/infinit.e.generic.library (all the logic for generic statistical enrichment of the data)
  • core/infinit.e.custom.library (all the logic for custom processing)
  • utility/infinit.e.elasticsearch.driver (utility project to abstract away the elasticsearch libraries)
• (Redhat etc) To run ANT from the command line requires "yum install ant" and "yum install ant-junit"
• To run a local instance of the REST API (requires all the non "GUI only" libraries), navigate to infinit.e.api/WEB-INF/src/com.ikanow.infinit.e.rest.application and run "StandaloneRestletApp.java" as Java applet
  • the command line should point to the "infinite-home/config" directory in the "infinit.e-config.rpm" project (or a copy).
  • Logging:
    • It may be necessary to set the "log4j.appender.DAILY.File" variable in the "log4j.api.properties" file for file logging to work.
    • Adding "STDOUT" to the "log4j.rootLogger" comma-separated variable will log to the eclipse console, which is useful
• It is also possible to run the API in tomcat mode (this is not normally necessary):
  • Change tomcat lib path to use necessary libraries: In eclipse click Window -> Preferences.  Then Tomcat -> Advanced.  Select all the libraries and data_model (see below).
  • If you need to reference local config files (running tomcat locally you will)  Change the file in infinit.e.api/WEB-INF/web.xml to from /opt/infinit-home/config to your config file location (e.g. c:/whatever/config)
  • (Later versions of tomcat (6.0.33+?) have removed the "manager" role and replaced it with "manager-gui, manager-script, manager-jmx, manager-status", so it is necessary to edit "TOMCAT_HOME/conf/tomcat-users.xml" and edit the user XML block, replacing the former with the latter)

Additional step for developing widgets or modifying the GUI framework

To develop widgets or modify the GUI framework, the following steps are necessary separately to the above guide:

• For GUI debugging, windows only: Flash Builder Debugger (scroll to bottom. Version Tagged "For IE" installs to internet explorer. Others install to all other browsers), a commercial component (we use 4.6.0).
  • It is also necessary to download and install version 4.5.1A of the Flex SDK (which is an open source library)
  • Note even without the commercial component, the GUI code can be built using the ant script plus the Flex SDK
    • (our software should work with 4.5.0+, but the checked in flex properties files point to 4.5.1A, so it's possible but a slight pain to work with different versions - for Linux-only installing, it shouldn't matter at all though, you specify the location of the Flex SDK as FLEX_HOME, eg "ant -DFLEX_HOME=/home/user/flex_sdk")
• Other projects that need to be downloaded in eclipse:
  • For GUI framework development:
    • visualization/frameworks/flex/infinit.e.flex.client (GUI only - can only be run on Windows, though can be built on linux using the provided ant script)
    • visualization/frameworks/flex/infinit.e.flex.client.library (GUI only - can only be run on Windows, though can be built on linux using the provided ant script)
    • visualization/frameworks/flex/infinit.e.widget.library (GUI only - can only be run on Windows, though can be built on linux using the provided ant script)
  • For widget development:
    • None, just follow the instructions here.
      • (of course you can check out widgets from "visualization/widgets/flex" to look at examples or edit existing widgets to change/enhance their functionality - note however that most of the widgets use the commercial iLog/Elixir licenses, which aren't included in the software distribution - trial versions can be obtained from here, together with details for purchasing the full commercial licenses.
        • The widgets using iLog are: InfiniteMapQuestWidget - only for the heatmap - , InfiniteSentimentWidget, InfiniteSignificanceWidget, InfiniteTimelineWidget, InfiniteTimelineEventWidget)
• To run a local instance of the GUI (requires the "GUI only" projects), navigate to "infinit.e.flex.client/src/index.xml" and run as web application
  • To point to anywhere other than the cloud GUI (eg to a local API, or to the Beta cloud build), it is necessary to create "html-template/AppConstants.js" from the ".TEMPLATE" file, and modify the "url" variable in the "getEndPointUrl()" function (And possibly "getMapLicenseKey()" though not if switching to localhost). Note that the getMapLicenseKey() can contain any string (when using the Open Source version of MapQuest - for the Licensed version it must be an API key obtained from MapQuest).

Building the JARs

All of the Java projects can be built by running the default ant target from the top level. 

The projects must be built in the following order:

• infinit.e.data_model
• infinit.e.harvest.library
• infinit.e.generic.library
• infinit.e.custom.library
• infinit.e.core.server and infinit.e.api.server

The "infinit.e.builder" project has an ant script whose default target will build all the JARs and also the RPMs. (And also the GUI, which will fail if the Flex SDK isn't installed, or if iLog/Elixir isn't installed - so in practice you will want to comment a load of projects out).

Building the install RPMs

Once all the JARs have been built, the RPMS can be built in any order by running ant at the top level of the projects (in install order):

• infinit.e-config.rpm
• infinit.e-index-engine.rpm
• infinit.e-db-instance.rpm
• infinit.e-processing-engine.rpm
• infinit.e-interface-engine.rpm

As above, note that the "infinit.e.builder" project has an ant script whose default target will build all the JARs and also the RPMs. (And also the GUI, which will fail if the Flex SDK isn't installed, or if iLog/Elixir isn't installed - so in practice you will want to comment a load of projects out).

Testing changes

Building a good JUnit-based test framework is on our roadmap, but it's not easy because of the number of external components (databases, Lucene indexes, etc) needed for many basic use cases. Therefore it is unlikely to happen until we find a willing intern!

In the meantime, the following test code exists:

• infinit.e.api.server: a mix of manual/semi-automatic test scripts are maintained in "infinit.e.api.server/TestScripts", they require a database to be populated.
• infinit.e.harvest.library, infinit.e.generic.library: manual "by eye" test code in "./test/TestCode.java" - some of which could easily be converted into automatic JUnit format, other bits require a database to be populated.

Contributing code back

Simply initiate a pull request (useful web page discussing the "fork and pull" development mode).

Note that, like most other commercial companies who publish their software as open source, we require contributor agreements (personal or corporate) to be in place before we will accept contributions back into the branch.