iSENSE

Communications and Information Technology Ontario (CITO)

QNX Software Systems

INCO Mines

Co-investigator: Prof. Dimitris Makrakis

Description of the Research

Introduction and Overview.

The rise of Internet technologies and standards in the recent past has caused a very rapid convergence of computing and communications technologies. IBM Global Services make the following observations [1; p.29] about the so far four major movements in the computing industry: the first era was the central computing, or mainframe, era that span the years 1950 to 1980. The second was the PC era, which began in the 1980s. The third was the network era, which really started in the 1990s with the rapid growth on the Internet and computer networking. The network era of computing promises dramatic changes in the nature of work and commerce, and in the structure of the industry. While this era is still unfolding, a fourth era is foreseen emerging, once all technologies of the networked computing era have stabilized and have been integrated into the majority of businesses and households. This will be the era of Pervasive Computing  where the emphasis shifts from information and communication technologies to information and communication per se [1; p.30]. Computers will become pervasive technology, that is, a technology more noticeable by its absence than its presence, [2].

Pervasive computing will give us tools to manage information easily. Information is the new currency of the global economy. We increasingly rely on the electronic creation, storage, and transmittal of personal, financial, and other confidential information, and demand the highest security for all these transactions. We require complete access to time-sensitive data, regardless of physical location. We expect devices -- personal digital assistants, mobile phones, office PCs, sensors, appliances and home entertainment systems -- to access that information and work together in one seamless, integrated system. Pervasive computing can help us manage information quickly, efficiently, and effortlessly.

Research reports on pervasive computing is only now emerging in refereed publications, at least to our knowledge.  IBM is a major industrial player in pervasive computing [9, 10] and they have recently announced strategic alliances with many companies, among which the Ottawa-based QNX Software Systems Ltd. A major project (Oxygen) has recently (April 1999) been initiated by the MIT Laboratory for Computer Science [4]. Oxygen is an integrated collection of eight new technologies: handheld PDAs, wall and trunk computers, a novel net, speech understanding, knowledge access, collaboration, automation and customization. Stanford University [3] is another early contributor.

The first mass-produced pervasive computing devices are starting to appear. The Clarion AutoPC [5] provides an efficient, reliable and secure integrated communications, computing, navigation, car control and entertainment system. The NCR Microwave Oven/Home Banking Terminal [6] and the Electrolux Internet Connected Screen Fridge [7], allow effortless home management. A good example scenario is given in [3]. Opening the fridge to take out a soda, you notice that there is only one left. The "smart" fridge has recorded that and adds an action item on your shopping list. The next day, as you drive home from work, the GPS-enabled AutoPC in your car, previously informed by your fridge that purchases need to be made, signals that you are near a supermarket. As you cruise the isles of the supermarket, wearing your augmented-reality goggles and your wearable computer [8], a soda supply triggers an object recognition program and an alarm reminds you to buy soda. The same of course could be done by your pocket Personal Digital Assistant (PDA) sensing the presence of the soda supply.

Another pervasive computing example was given by IBM Chairman Gerstner in [9]: "Think about driving down the autobahn. Your intelligent car develops an engine problem, but instead of flashing you a warning light it sends a message directly to the manufacturer over a wireless connection to the network. The manufacturer's systems diagnose the problem and transmit a fix back to the electronics complex in your car. In fact, that corrective fix is transmitted to all models everywhere in the world, without ever having to notify the owners. .... Instant information on performance is captured and sent immediately into product development and manufacturing. Continuous feedback loop, continuous improvement, resulting in better cars. Good for consumers".

Electric motors are an example of pervasive technology [2]. The average US home contains two dozen or more electric motors. A multitude of sensors are gathering the information needed to control them. As all these are buried inside many appliances (vacuum cleaners, microwave ovens, refrigerators, VCRs, etc.) we have difficulty identifying them and we don't even care where and how many they are. In the future, the same will be true with computers, most of which will be hidden in information appliances. These new appliances are "smart devices" embedded with microprocessors that allow users to plug into intelligent networks and gain direct, simple, and secure access to both relevant information and services. These devices are as simple to use as calculators, telephones or kitchen toasters. Pervasive Computing envisions the "networked home" where domestic devices can interact seamlessly with each other and with in-home and external networks. Using the existing home infrastructure based on open industry standards, a person will be able to integrate the home network with external networks to easily manage home devices, both locally and remotely.

Digital and more recently computer-based instrumentation and measurement technologies have evolved over the last 30 years in order to provide an interface to the physical world for many computer based industrial control applications, [11], [12], [13], [14], [19].  The advent of the pervasive computing provides new incentives for the development of even more efficient and versatile sensing technologies. New sensors are needed to measure a greater diversity of parameters in order to provide a wider, more dynamic and more information-rich window on the environment.

Recent progress in computer and VLSI technologies allow the use of complex signal and image processing, system identification, modelling, control, and AI algorithms, as well as user friendly virtual environments for the development of an ever growing diversity of real-time intelligent sensing applications ranging from Computer Integrated Manufacturing (CIM) to smart homes and offices.  These new developments point to the emergence of a new type of intelligent control based on a multisensory perception of the state of the controlled process and its environment [15], [16]. These intelligent model-based adaptive controllers provide an intelligent connection of the perception to action to achieve specified goals in complex changing environmental situations.  World models, built and maintained from information gathered by a multitude of sensors, provide a common abstract representation of the state of the environment. At the perception level, the world model is analyzed to infer relationships between different objects and assess the consequences of the controller's actions.

The perception capability is an AI-oriented active investigatory window providing global information that reduces the uncertainties about the physical state of the controlled process and its environment [20].  The only a priori defined constraint in this case is what parameters are to be measured without necessarily knowing where or when or even if  they will occur.  The question of how multiple sensor data are integrated in world models adds a new dimension to what can be called the perception paradigm.

More human-like intelligence based on neural networks, fuzzy logic, and expert systems will be incorporated on a wider scale in the new generation of instruments and measurement methods.  Virtual instruments will evolve to become even more user-friendly by providing interactive sensor-based virtual environments for telepresence operations, where human operators will be integrated in hybrid man-machine control loops as ultimate strategic-level decision makers [17], [18].

As their perception ability grows thanks to intelligent sensing the information appliances are evolving into  sensor-based  intelligent  appliances representing the next evolutionary stage for the pervasive computing paradigm. These intelligent appliances will behave like pervasive intelligent autonomous robotic agents providing a seamless intelligent connection of the perception to action, [15].

While the smart networked home is a very good showcase for information appliances, the intelligent sensing agents and sensor-based appliances will allow the pervasive technology idea to spread to other areas of human activity as for instance the mining and manufacturing industries, security industry, transportation, training and health etc.  It will not be exaggerate to claim that this technology, when integrated in the emerging global information infrastructure, will have a profound impact on our personal and professional activities, and open business opportunities, of similar or even higher scale to what we are experiencing presently with Internet.
 

Description of the research in detail

Early digital and computer-based instrumentation architectures and communications standards as HP-IB (IEEE 488) represented embryonic smart sensing solutions supporting the first generation of computer based industrial applications.   Microprocessor controlled sensors and virtual instrumentation integra tion environments such as LabView together with the wireless and internet communications have then allowed to develop a larger variety of embedded industrial applications.  The advent of pervasive computing marks an urgent need for a new generation of intelligent sensing agents and sensor-based appliances and related resource management environments for a broader area of applications involving loosely coupled, event-driven, heterogeneous intelligent sensing agents and appliances.

1. Development of intelligent sensing agents and sensor-based intelligent appliances for pervasive computing applications in heterogeneous environments.
Intelligent sensing agents will be developed as autonomous agents carrying out task-directed active investigation of specific environment parameters.  Reliable communication modalities will allow them to cooperate in order to monitor the multi-parameter state of large systems. Intelligent sensing functions will also be embedded together with computing and actuation functions in intelligent appliances for pervasive computing applications.
Model based sensor architectures for both propriceptors, i.e., sensors monitoring the internal state of an information appliance, and exteroceptors, i.e., sensors monitoring the state of the environment outside the information appliance, using sensor- and world-models will provide superior modularity, interchangeability (plug and play) and transparence allowing for easier sensor fusion and knowledge extraction. Sensor models will be used to describe how sensors observe parametrically represented objects and Bayesian probability will be used to represent sensor models. Statistical as well as fuzzy and neural network methods will be studied for sensor fusion.
Intelligent task-directed information gathering features will be studied to allow for a more elastic and efficient use of the inherently limited sensing and processing capabilities of each agent. Each task a sensing agent has to carry out determines the nature and the level of the information that is actually needed. Sensing agents should be able of selective environment perception that focuses on parameters important for the specific task at hand and avoid wasting effort to process irrelevant data. A task-specific decision making process will guide the incremental refinement of the environment model.
Each intelligent sensor will be developed as an intelligent agent able to learn/adapt, able to deal with multiple redundant communication carriers (intranet, internet, power lines, wireless, infrared,..).

2.   Development of versatile man-machine sensing interfaces allowing messaging for real-time control and/or communication between sensors and human operators/users.
Research in the field of computer-human interfaces (CHI) has been extensive. As the era of pervasive computing commences, portable wireless PDAs or wearable computers will be widely used. New CHI paradigms (e.g., retinal projection) and designs are thus called for and this research will be addressing them. Design of versatile man-machine interfaces allowing messaging for real-time control and/or communication between sensors and human operators/users will be investigated. Voice recognition and synthesis for iSENSE PDAs will be researched. Our recent experience in using Distributed Virtual Environments in CHI design, with applications to e-commerce and industrial tele-training, will be an asset in this research. An example application will be telecontrol in a hazardous mining environment.

3.  Design of temporal synchronization protocols among sensors, actuators and/or human operators.
Multimedia synchronization typically refers to the temporal relations among media such as video, audio, graphics, images, text, etc, produced during the authoring of a multimedia document. Temporal asynchronies are due to network and devices delay, delay  variation, buffer sizes, available bandwidth. This research will extend recent results in media synchronization protocols [23, 27], designed for compensation of media asynchronies of the order of 100 msec or higher, to much stricter temporal constraints of the order of a few msec or microseconds, pertinent to sensors and actuators relationships. An example in computer-integrated manufacturing is the actuation of a robot-controlled welding operation on an assembly line upon a sensor command.

4. Design of networking technologies for the support of pervasive intelligent sensing agents and appliances.
As a very large number of devices  will be serviced through the wireless and wire-line global networks infrastructure, existing technologies will be rendered inefficient; new solutions have to be invented. The nature of pervasive computing and services devices require that the developed architectures should distributed rather than centralized. Our work will take into account both the developing PAN (Personal Area Networks) (IEEE 802.15), media access control and existing Local and Wide Area Network Standards (e.g. IEEE 802.11), Internet Protocols (Mobile IP, IPv6, RTP/RTCP, RSVP, XTP etc.). However, we expect that the size and complexity of the problem would require the development of new technologies and standards. The objective of our research will be to develop Distributed Networks Architectures (DNA) suitable for the support of pervasive computing at a large scale. Our work will address wired and wireless networking issues and development of cost-effective solutions for environments where deployment of advanced networking infrastructure could be unjustifiably costly. Our solutions will address both Internet and Intranet architectures.  The following activities will be part of our research:
· Accurate modeling of tele-traffic produced by such devices and assessment of network impairments on the performance
Accurate knowledge of the new traffic dynamics is essential, since successful design of networking technologies and applications requires accurate knowledge of tele-traffic behaviour.
· Development of “service” admission control and connection establishment policies, as well as resource allocation and resource adaptation algorithms for the support of pervasive devices.
Pervasive devices will require the service of a highly inhomogeneous global network.  Our work will be focused around the Internet protocol suite (Mobile IP and Cellular IP, IPv6, RTP/RTCP, RSVP) and address deployment over heterogeneous networks.  It will also address the realities of distributed network architectures and the need for balanced loading of the network based resources, while the QoS of the applications/services is safeguarded.
· Development of Quality of Service capable, error-resilient and resource allocation-efficient multiple access protocols for the efficient transportation of sensor traffic.
The bandwidth and resource limitations of the wireless medium require that that information content is “compressed” as much as possible, in order to “consume” the least amount of resource possible.  However, such low redundancy makes the information vulnerable, especially in an error-prone environment such as wireless channels and networks. In our work, the resource mapping policies between the wired and wireless network and the protocol interoperability issues will be addressed.
· Development of “intelligent networking” infrastructure and definition of suitable architectures of distributed nature.
The proposed solutions will be capable of: a) servicing the processing requirements of pervasive devices, b) provide them with full mobility support and c) guarantee their operation in a seamless fashion.   This requires extension of the “Intelligent Networking” concepts to the Internet and Wireless Internet.
· Development of cost-effective network solutions for environments where there is no advanced networking infrastructure deployment.
We plan to develop suitable technologies for establishing inexpensive LANs for pervasive computing and intelligent sensing applications. Various avenues, such as use of the power line network will be investigated.

5.   Agent-based framework allowing management of heterogeneous functions for a large diversity of distributed intelligent sensors and pervasive computing appliances.
Pervasive computing environments involve both human-machine and machine-machine interaction and cooperation paradigms. The research will concentrate on machine-machine aspects. We are all familiar with human-to-human communication and cooperation, which require a common language and an underlying system of shared knowledge and common values.  In order to achieve a similar degree of machine-machine interaction and cooperation, we will develop a framework allowing management of heterogeneous functions and knowledge for a large diversity of pervasive computing devices. This framework will address functionality at a level of abstraction higher than that of the classical communication network protocols, and even than that of coordination frameworks such as ODP (Open Distributed Processing) and CORBA (Common Object Request Broker Architecture), which provide mainly distribution transparency. The heterogeneous pervasive computing devices cannot realistically be expected to have a common language and understanding of each other's functions. Accordingly, the proposed functionality management framework will investigate integration mechanisms allowing different domain-specific devices to communicate, understand each other and collaborate toward common goals.  In order to provide a flexible extensible open framework, we will develop methods by which different devices will exchange the grammars describing their domain-specific languages and learn to understand each other. This way, the devices will be able to advertise their own functions, search and discover providers of required services, and express their needs in a collaborative environment.

6.  Case study:
Environment monitoring, telesensing and remote-controlled robots are currently developed for hazardous operating environments: fire, explosive, chemical, nuclear, mine fields, underwater, as well as space. As the complexity of tasks increase and the need for real time reactive behaviour becomes  critical, the robots will require to be capable of teleautonomous behavior.

We will design a pervasive computing system using intelligent sensing agents and sensor-based   appliances, i.e. intelligent sensing robotic agents,  for multi-parameter monitoring and  real-time telecontrol in a hazardous mining environment (INCO).  This pervasive computing environment will be supported by a real-time resource management system using the QNX real-time operating system
Essential for the control of the robotic agents is an embedded perception system able to timely and correctly assess the state of the robot's physical environment.  The use of multiple sensors is beneficial in improving the accuracy, the cost and robustness of the perception process.  Many of these applications are impossible to fully automate and remote human operator expertise is still needed to carry out tasks requiring a higher level of intelligence. A proper control of these remote operations cannot be accomplished without some telepresence capability allowing the human teleoperator to experience the feeling that he/she is in the remote environment.
The specific objectives are: (i) the development of intelligent exteroceptor sensing agents and teleautonomous robotic agents, (ii) the design of a model-based multi-sensor fusion system able to integrate a variety of sensors that cover all four phases in the environment perception process: far away, near to,  touching, and manipulation, (iii) the study of new task-directed sensor fusion and learning methods for an active perception which will allow the robot to gather information by interaction with the environment (iv) to provide human oriented geometric-  and force-domain telepresence feedback for teleoperation, (iv) design of a redundant multi-carrier communication system for the sensing and robotic agents.
Machine learning techniques will be studied for the real-time reactive behavior of the sensing and robotic agents. A variety of reinforcement learning paradigms using learning classifier, neural networks, fuzzy logic, and Q-learning, as well as the evolutionary learning paradigm will be studied to implement basic reactive behaviors that accomplish simple tasks. In order to find more efficient solutions to the complex tasks we will study the combination of the intrinsic reactive-behavior with higher-order world model representations of the environment.
In terms of networking technology, the activities described earlier will be pursued. Prototypes will be developed and integrated in the test-beds at U of O and at INCO’s research facilities.  INCO will be making available to us their research facilities in Copper Cliff, Ontario for testing our developments.  We will have access to the underground wireless and wired underground networks and the high speed network that is used to transport information between their research facility and the headquarters.
 
 

General references:

1. "Assessing the Strategic Value of Information Technology", Research Report, The Economist Intelligence Unit Ltd and IBM Global Services, N.Y., 1999
2. J. Birnbaum, "Pervasive Information Systems", Communications of the ACM, Vol. 40, No.2, Feb. 1997, pp. 40-41
3. A.C. Huang, B. Ling, S. Ponnekanti, and A. Fox. "Pervasive Computing: What Is It Good For?"  Workshop on Mobile Data Management (MobiDE) in conjunction with  ACM MobiCom 99 (to appear).
4. M.L.Dertouzos, "The Future of Computing", Scientific American, 52, Aug. 1999, pp.52-55; also "MIT Oxygen Project"  http://www.lcs.mit.edu/anniv/press/oxygen040799
5. Clarion Corp., "Auto PC ," http://www.autopc.com/walkthrough/communication/, 1999
6. NCR Corp., "NCR Microwave Oven / Home Banking,"  http://www.wired.com/news/news/technology/story/14949.html, 1999
7. Electrolux Inc., "Electrolux screen Fridge", http://www.wired.com/news/news/email/explode-infobeat/technology/story/%17894.html, 1999
8.  S. Mann, "Wearable Computing; A first Step Toward Personal Imaging", IEEE Computer, 30 (2), Feb. 1997
9. L. Gerstner, "Pervasive Computing" , Keynote  -CeBIT 98,  http://195.27.241.3/news/news1/ns-3975.html
10. IBM, "What is Pervasive Computing", http://www-3.ibm.com/pvc/pervasive.html
11. M. Hardwick, D.L. Spooner, T. Rando, K.C. Morris, "Data Protocols for the Industrial Virtual Enterprise," IEEE Internet Computing, vol. 1, no. 1, Jan-Feb.  1997, pp. 20-29.
12. R. Itschner, C. Pommerell, M. Rutishausser, "Glass: Remote Monitoring of Embedded Systems in Power Engineering," IEEE Internet Computing, vol. 2, no. 3, May-June. 1998, pp. 46-52.
13. P. Dabke, "Enterprise Integration via CORBA-Based Information Agents", IEEE Internet Computing, vol. 3, no. 5, Sept-Oct. 1999, pp. 49-57.
14. C.M. Pancerella, N.M. Berry, "Adding Intelligent Agents to Existing EI Frameworks," IEEE Internet Computing, vol. 3, no. 5, Sept-Oct. 1999, pp. 60-61.
15. A. Brooks, L.A. Stein, "Building Brains for Bodies", Autonomous Robots, Vol. 1, No.1, pp. 7-25, 1994.
16. B.V. Dasarathy, "Sensor Fusion Potential Exploitation – Innovative Architectures and Illustrative Applications," Proc. IEEE, Vol. 85, No. 1, pp. 24-38, Jan. 1997.
17. R.W. Picard, "Human-Computer Coupling," Proc. IEEE, Vol. 86, No. 8, pp. 1803-1807, Aug. 1998.
18. S. Mann, "Humanistic Computing: "WearComp" as a New Framework and Application for Intelligent Signal Processing," Proc. IEEE, Vol. 86, No. 11, pp. 2123-2151, Nov. 1998.
19. R. Sasdelli, C. Muscas, L. Peretto, "A VI-Based Measurement System for Sharing the Customer and Supply Responsibility for Harmonic Distortion," IEEE Trans. Instrum. Meas., Vol. 49, No. 5, pp. 1335-1340, Oct. 1998.
20. K. Kariya, S. Takayama, H. Kochi “Laboratory Model for Thinking of Measurement Science,” in Instrumentation and Measurement Technology and Applications, (E.M. Petriu – Edit.), pp. 13-18,  IEEE Press, New York, 1998.
21. A.A. Platonov, J. Szabatin “Analog-Digital System for Adaptive Measurements and Parameter Estimation of Noisy Processes,” in Instrumentation and Measurement Technology and Applications, (E.M. Petriu – Edit.), pp. 259-268,  IEEE Press, New York, 1998.
22. E. Steenput, Y. Rolain “Auto-Consistent Mathematical Environment for Measurement Software Development,” in Instrumentation and Measurement Technology and Applications, (E.M. Petriu – Edit.), pp.280-285,  IEEE Press, New York, 1998.
 

MILESTONES

Dec. 2000 Development of first Intelligent Sensor Architecture completed; Assessment of temporal synchronization requirements in iSENSE competed;  Study of existing PDA interfaces and capabilities for pervasive computing completed; Preliminary design of wireless or integrated networking architecture completed.

Mar. 2001  Development of first new synchronization protocol completed; First man-machine interface, using voice recognition, completed; Tele-traffic modeling of interconnected devices completed; new wireless protocols study completed;  Case study for mining application initiated.

Dec. 2001 Intelligent Network activities initiated; research on presence, position tracking sensors, interfaces, retina projections initiated; Design of new synchronization protocol for large distributed environments completed.

Mar. 2002 Implementation and performance evaluation of distributed synchronization protocol completed; Implementation and evaluation of new CHI completed; Intelligent Networking research completed; case study completed and final demo of complete iSENSE prototype test bed.