IoT Interoperability for Big Data: What Exactly?

The importance of IoT interoperability is widely acknowledged and promoted.  The McKinsey report estimates that achieving interoperability would unlock an additional 40% of IoT market value.  However, there is usually little specificity as to what type of interoperability is being discussed and what it is it useful for when achieved.   There are two important flavors of IoT data interoperability illustrated in the adjoining figure.  We argue that aggregation or service-level level interoperability across domains and specifications is the one necessary to enable useful big-data IoT aggregations.  In contrast, most active IoT standards definitions are focusing on M2M, intra-domain interoperability.

IoT Data Interoperability: What Exactly?

In IoT system hierarchies, such as the one depicted in the figure, there are at least two very important levels of data interoperability:

  • Machine-to-machine (M2M), device-level interoperability
  • Aggregation level interoperability, as provided to cloud services by data queries and APIs

Current IoT Standards Direction: M2M Interoperability, Intra-domain

Device-level M2M interoperability is the one most commonly talked about and it is the target of most IoT standards under development, including OCF (formerly OIC, now merged with AllJoyn and UPnP), W3C WoT, IPSO, OneM2M.  For devices to interoperate, such standards – in addition to data models – often define or assume communication protocols, security, discovery, and sometimes add configuration and device management.  Interoperability is often claimed, but at this level it means that devices, such as home appliances, from different vendors will interoperate if all of them implement correctly the same specification or run some common middleware.  Similar functionality may be achieved by installing proprietary, vendor-specific middleware on all target devices, albeit without the benefit of a public specification and with the ensuing vendor lock in.  Interoperability at M2M level is typically intra-specification and intra-domain, i.e. it can be achieved among a group of devices with compliant implementations of a common specification.

The problem in practice is that there are many different IoT standards proposals with incompatible data models and definitions, usually aimed at rather narrowly focused domains – such as (many flavors of) industrial and consumer.  While IoT is poised to thrive by using internet connectivity and much of its design and technology, unfortunately so far it has failed to create a moral equivalent of HTML for sensor data.

Big-data IoT Aggregations: Inter-domain, Inter-specification Interoperability

To increase the value of analytics and AI, big-data aggregations need to include large and diverse sets of IoT data that typically span a multitude of devices across different domains.  In practice, that typically implies the use of multiple different IoT standards mixed with proprietary and legacy data and meta-data formats that are thus inter-domain and inter-specification.

As an illustration, consider a smart city that plans to use IoT technology to coordinate and optimize data across its many disparate IoT and legacy systems such as transportation, lighting, energy, building management, water supply.  Or a holistic factory-optimization project that needs to stitch together IoT sensor data with a variety of different and likely incompatible data formats provided by vendor-specific equipment into a common sensor database for predictive analytics.

These requirements can be addressed by providing data interoperability at the aggregation or cloud service level, shown in the figure at the layer above the domain-specific or federated sensor database(s). Service-level data interoperability implies providing a common data “meta” model or annotation for data queries used by services.  It provides the ability to query aggregate data and meta-data sets and obtain results in a common form, regardless of differences in formats used to originally encode data when captured at the edge. IIC refers to this as conceptual interoperability [IIC IOT ref arch, p68], i.e. representation of information in a form whose meaning is independent of the application generating or using it.


A significant benefit of IoT data interoperability is the ability to create large, useful aggregations of sensor data for post-processing such as data mining, analytics, machine learning, and AI.  It is well known that effectiveness of all those techniques increases with the size and diversity of data sets. At present, IoT data are fragmented and locked in silos due to incompatibility of formats in proprietary platforms and in numerous evolving standards that focus on limited domains.  Another significant benefit of an architected approach to interoperability is the ability to incorporate volumes of existing sensor data being produced by legacy systems, including proprietary Supervisory Control and Data Acquisition (SCADA) systems, Building Management Systems (BMS), energy utilities, and various automation and manufacturing systems using legacy standards such as BACnet and Modbus.

In addition to increasing the IoT data-set size and diversity, the availability of a commonly understood format enables creation and use of portable IoT applications and services – such as analytics and AI – and significantly accelerates variety and usefulness of offering by following the proven Internet playbook.  In the absence of service-level data interoperability, which is the case today, IoT services and applications tend to be custom tailored to specific systems and data formats, which makes it difficult and prohibitively time consuming to port them to another system or cloud hosting service.  This situation makes today’s complex IoT installations look more like the proprietary and brittle SCADA systems of yore. To proliferate at Internet velocity and scale, IoT data need to be made easy to aggregate with portable apps and services to operate on them – both made possible by service-level interoperability.

How to Get There?

Aggregation- or service-level interoperability requires a common methodology for trans-specification data and meta-data annotation or modeling.  There is no current specification for that, but some standards bodies – like IPSO, with encouraging interest from others – are beginning to work on semantic interoperability across domains and specifications.  Its requirements and properties will be discussed in another post.  However, its feasibility and an implementation option were demonstrated by our interoperability POC. It provides a real-time translation of sensor data encoded in IPSO, OCF, and Haystack into a common semantically interoperable form.


Haystack 2017 Keynote by Milan Milenkovic, IoTsense

Gave keynote at Haystack Connect 2017 biennial conference in Tampa, FL on May 9. (First keynote speaker at two Haystack conferences in a row.)  Title “IoT Semantic Interoperability and Project Haystack: Beginning of a Beautiful Friendship”.

Haystack 2017 Keynote

The primary intent was to promote a call to action to Haystack community to participate in defining cross-standards interoperable data formats in collaboration with other standards bodies.  Key topics covered:

  • What is semantic interoperability and why it is important?
  • Overview and positioning of standards that include sensor data and meta-data models
  • Conjecture that it is no longer likely for a dominant standard to emerge, so go for interoperability
  • Interoperability POC as a feasibility proof
  • Next steps – technical and standards collaboration

Judging by a number of side conversations with participants, audience reception was very favorable. Haystack principals expressed additional interest and support in a private meeting, we plan to follow up with other standards. Link to presentation.

IoT Data Interoperability POC: a Pragmatic Feasibility Proof

Introduction and problem statement

The importance of IoT interoperability is widely acknowledged, touted by standards and vendors, and often discussed by industry analysts.  The McKinsey report “Unlocking the Potential of the Internet of Things” estimates that achieving interoperability in IoT would unlock an additional 40% of market value. So, how can semantic IoT data interoperability be accomplished in practice?

It does not seem likely to happen through standards – there are too many competing ones in IoT data space, fragmented and conflicting in some cases, with no discernible dominant player. They are mostly focused on intra-specification interoperability among compliant devices, but not across specifications and domains where the estimated 40% of the additional value is.

An alternative top-down approach would be to try to structure a meta ontology from the existing standards definitions.  This tends to be rather cumbersome in practice and it ran into problems with semantic web efforts. It is too slow to cope with evolving IoT standards and the explosive growth of types of IoT devices.

We explored an alternative pragmatic approach to cross-standard interoperability by selecting some representative IoT standards and prototyping a translation into a common interoperable data format. This proved to be feasible and practical, as well as instructional, and we are in the process of expanding and promoting it as an added activity to leading IoT standards bodies.

Implementation: data interop POC

To explore the feasibility and complexity of IoT data interoperability in practice, we created an interoperability proof of concept (POC).  The basic idea was to mimic real-world IoT diversity by having a collection of sensors whose data are encoded in different standards and converted in real time to a common format to be consumed by IoT applications and services in the cloud.

The setup consists of 3 gateways acting as sensor data aggregators (Intel Galileo boards), each with a set of 5 sensors that measure same physical entities – air temperature, (simulated) chilled-water temperature, humidity, CO2, and electrical current flow.  Sensor selection was driven by an actual smart-building engagement and use case. Specific sensor types were chosen for their wide use in different domains, such as industrial, weather, commercial buildings, and homes.

IoT Data Interop POC at IOTSWC 2016

The code running on each gateway samples sensor data periodically, encodes it in one of three popular IoT standards – IPSO, OCF and Haystack respectively – and sends data to the fourth gateway that is acting as the format-conversion unit.  The format-translation gateway receives data encoded in specific standards and converts/translates them to a common interoperable data format which is the same regardless of differences in the input data, thus providing consistent interface to cloud services, such as analytics or ML.

IoT Data Interop POC System View

Note that the format-conversion function could be placed anywhere in the data path before the data-retrieval APIs, e.g. as a container service running in a gateway, fog node or cloud.  In our implementation, we placed this function in a separate larger gateway to make the demo self-contained.  We also used that gateway to host and drive the POC UI (live link) function.

IoT Data Interop POC UI

The reading highlighted in the figure shows the IPSO encoding for a temperature sensor whose ID is gipso/3303/1.  Other encodings of the equivalent sensor, OCF, Haystack and BMP are greyed out.  While there are obvious differences in encodings between standards, the interop format is the same for all three, with only sensor IDs being unique as they refer to different physical sensors attached to different gateways.

Conclusions and next steps

The POC demonstrated that our ground-up approach to translating individual standard definitions is feasible and practical.   In retrospect, this stands to reason because all standards are modeling the same physical entities, real world “objects”, thus resulting in (mostly) comparable abstract object models. Coding of the translation function was further simplified by the fact that all three standards provide variants of the popular JSON [key, value] form of data serialization.

On the negative side, we experienced the drawbacks of non-isomorphic and rigid modeling of smart objects in several standards.  In some cases, we were forced to provide superfluous data fields to satisfy arbitrarily clustered resource properties, in others we were unable to express the available numerical sensor readings and had to use Boolean values instead.  We’ll communicate these findings and other insights gained from the POC to the related standards bodies.

We also implemented an expanded version of the POC with the additional proprietary sensor data-format to test the effectiveness of the proposed approach beyond standards.  The outcome was successful.

Being positively and strongly encouraged by this experience, we intend to work with several standards bodies to promote adding of semantic data interoperability to their charter and deliverables.


This work was partially funded by Intel corporation. POC concept, architecture and system design by IoTsense, implementation HTEC group.

[contact-form][contact-field label=’Name’ type=’name’ required=’1’/][contact-field label=’Email’ type=’email’ required=’1’/][contact-field label=’Website’ type=’url’/][contact-field label=’Comment’ type=’textarea’ required=’1’/][/contact-form]

IoT Data Interoperability Demo at IOTSWC 2016, Oct 25-27, Barcelona


We are having IoT data interoperability demo to show feasibility of real-time translation of sensor data from different standards into a common format to be consumed by cloud services. The purpose is to enable useful aggregations of large IoT data sets that portable services – such as analytics and ML – can operate on.

Assuming that no single standard for IoT sensor data format will emerge to rule them all, we are focusing on the next best thing – data interoperability across IoT standards and domains. The demo consists of three gateways collecting data from three sets of sensors measuring the same set of physical phenomena but encoded in three different standards – IPSO, OCF, and Haystack – converted to a common, interoperable format. Come see us at Intel booth at IOTSWC in Barcelona, Oct 25-27, 2016.

A Case for IoT Interoperable Sensor Data and Meta-Data Formats

Previously published paper in ACM Ubiquity Symposium, synopsis

While much attention in the Internet of things/web of things (IoT/WoT) community has been focused on building sensing systems and backing cloud infrastructure, enabling third-party applications and services that can operate across domains and across devices has not been given much consideration.  The challenge for the community is to devise standards and practices that enable integration of data from sensors across devices, users, and domains to enable new types of applications and services that facilitate much more comprehensive understanding and quantitative insights into the world around us.

The target is to achieve semantic or, as IIC refers to it, conceptual interoperability [IIC IOT ref arch, p68], i.e. represent information in a form whose meaning is independent of the application generating or using it.  When supported, semantic interoperability achieves two important objectives (1) enables service-level integration of IoT end-to-end systems constructed using components from different vendors, such as a variety gateways running different middleware, with a third-party cloud for data storage, processed by analytics from independent vendors, and  (2) it allows aggregation of  data from different domains, such as disparate systems in smart cities, to allow for holistic management and – more importantly – to enable big data by virtue of creation of large data sets that are understandable, and thus usable, to analytics and other services.

Useful instantiations of multi-domain IoT systems can provide significant new business and innovation opportunities for new services. In order to fulfil that promise, IoT/WoT systems need to be designed to support some level of commonality by defining interoperable sensor data and meta-data formats, naming, taxonomy and possibly ontology.  This paper sketches some use cases that motivates this need and outlines a possible path to get there.

Full paper can be found at the link below

A Case for IoT Interoperable Sensor Data and Meta-Data Formats