Introduction to KnowNow Products
This document covers the following topics:
- KnowNow
Architecture
- Architectural
Components
- Communication
Protocol
- Concepts
- KnowNow Event
Router
- Microservers
- Glossary of
Terms
Overview
KnowNow Architecture
KnowNow is a software platform for integrating applications in a simple, incremental, and cost-effective manner. KnowNow combines publish-subscribe event delivery with internet-style routing to bring reliable, asynchronous communications to applications over the Internet. In this model, applications " publish " information, in the form of " events " and these events are categorized by " topics ."
Other applications " subscribe " to these topics and receive the events' information as it occurs. For example, an investment bank might run a publish-subscribe system with a separate topic for each stock traded on the NASDAQ and NYSE. Information related to WalMart (WMT), such as ticker prices, breaking news, new research reports, etc. is published to the "WalMart" topic, and all traders who subscribe to this topic receive the various pieces of information (i.e. events) as they are published.
This section describes the KnowNow architecture, components, and operations.
Architectural
Components
The KnowNow architecture comprises of four components, each connected by HTTP as the native communication protocol:
Event Routers
KnowNow Event Routers manage topics, events, and subscribers. Users and applications publish events to Event Routers, and the Routers then deliver these events to subscribers. Unlike traditional publish-subscribe middleware, KnowNow Event Routers work in a loosely coupled, decentralized manner. A Router can manage its own local subscriber population, and a Router can be a subscriber to another Router's topic set. This design enables Routers to scale over the Internet. Just as the Internet is a "network of networks," KnowNow Event Routers can scale to meet Internet-level needs.
Event Routers support a flexible method of topic subscriptions. As in any publish-subscribe system, a topic subscriber can be an end user or an application. All users or applications that subscribe to a topic receive events on that topic. However, KnowNow provides additional capabilities that support a great deal more flexibility and power.
In KnowNow technology, a topic can be a subscriber to another topic. For example, an investment bank might create a "Retailing Index" topic that subscribes to individual company topics such as WalMart, KMart, Target, etc. As a subscriber to each of these, the Retailing Index topic now provides a dynamically aggregated information stream. Users can subscribe to this topic, filter it, etc. for on-the-fly data aggregation. Also, because the Event Routers can subscribe to each others topics, corporations can share information and distribute it to their local users, without sharing these users' information or requiring a single large namespace.
In many ways, the Event Router looks and acts like a standard HTTP Web server. Multiple Routers can be clustered for failover and scalability. Topics are exposed as standard URLs, and the Router uses HTTP as it's native communication protocol. Publishing and subscribing are actually performed with HTTP POST and GET. This enables applications that have an HTTP interface to publish events to the KnowNow Event Router without having to make any modifications at all; an HTTP POST to the correct topic's URL is all that is required.
KnowNow events can be flexibly typed, and can contain XML, SOAP, or other message format.
Modules
KnowNow Event Routers can accept Modules, via an API, that perform a variety of functions such as data transformation, content filtering, authentication, and authorization. KnowNow Router ships with a set of basic modules to perform these functions:
· Regular Expression (Regex) Filter: The Regex filter can be attached to a topic or a route and filter events based on their kn_payload content with a regular expression that has been assigned to the filter. The module can handle the following four actions for events that match the regular filter--Reroute, deny, allow, and transform. For more information, refer to the Regular Expression Filter Module Sample documentation.
· SQL Authentication Module: The SQL Authorization (sqlauth) module is an example of an authentication module that authenticates login information against an Oracle database. For more information, refer to the SQL Authentication Module documentation.
Through these Modules, KnowNow becomes an extensible platform for real-time data interchange among decentralized applications.
Microservers
Microservers are optional, client-side components that manage persistent, bi-directional connections between an application and an Event Router. They are designed to be non-intrusive and lightweight; the JavaScript Microserver works with "zero-install" and is automatically downloaded into the browser without local installation or modification. By holding open a connection and periodically sending small "heartbeat" packets, the Microserver delivers bi-directional communication over HTTP 1.1.
Beyond the persistent connection, Microservers also perform event delivery functions such as auto-checkpointing and offline queuing. Auto-checkpointing ensures that events are delivered and received reliably, and that duplicates are automatically removed. Offline queuing enables publishers and subscribers to continue working when network connections are not available. If an application is publishing events and a connection is not available, the Microserver simply queues the events locally, then transparently delivers these events when a connection is re-established.
In fact, the Microserver is optional. Some applications do not require a persistent connection; they simply want to publish events. Since a publish in KnowNow happens via a HTTP POST, applications can communicate with Event Routers without Microservers, and without any modifications. Conversely, the Router can publish to an external URL, so an application can subscribe without a Microserver. In doing so, the application does not receive persistent connections, offline queuing, or checkpointing; and in some cases this is an acceptable trade-off.
Microservers are available for a variety of platforms and languages, including JavaScript, Python, Java, C, and ActiveX.
Gateways
Gateways connect the KnowNow platform with other infrastructure technologies, such as message queuing, email, or database systems. KnowNow Gateways provide an easy way to extend these enterprise technologies across the Web and to the end-user's desktop. For example, an organization my use IBM MQSeries? to perform message delivery between several internal systems. Through the KnowNow MQSeries Gateway, this organization can pass messages flowing over MQSeries through the firewall and over the Web to business partners. These partners need not have MQSeries installed. Alternatively, one of these partners may have a different internal messaging system, such as Tibco Rendezvous. By using the TIB Gateway, this company can receive its partner's MQSeries events, over the Web, and have these flow through its Tibco bus.
Enterprise middleware such as MQSeries and Rendezvous are often used to pass information between internal systems but not to end-users. KnowNow can extend the data in these systems to users' desktops in a non-intrusive manner, via the JavaScript Microserver (for browser-based access) or the Excel Microserver (for spreadsheet access).
Communication Protocol
KnowNow uses HTTP 1.1 as its native communication protocol. HTTPS for secure communications is also supported natively across all components. Using HTTP as the communication protocol allows KnowNow to work automatically through firewalls, without modification.
KnowNow's simple API (publish(), subscribe(), and unsubscribe()) and use of HTTP for communication gives developers a very easy way to build network-enabled applications without understanding network protocols, error handling, etc. For example, adding the ActiveX Microserver to a Visual Basic application enables the application to send and receive data over the Internet without special networking code.
Up-to-Date, Reliable Delivery
The KnowNow components described above communicate dynamically to deliver information as events happen. The persistent, bi-directional connections between Microservers and Event Routers allow events to be sent proactively to subscribers without polling or page refreshes. To ensure reliability of dynamic delivery, KnowNow implements store-and-forward messaging between publishers and subscribers.
Store-and-forward is a well-known means of providing reliability over a network where connections may be dropped and restored. At each step in the delivery chain, KnowNow provides an automatic means of holding and resending an event until it can be delivered successfully to the appropriate endpoints.
Summary
KnowNow's architecture provides reliable, asynchronous, and loosely coupled communication over the Web. By using URLs for topic names and HTTP as the communication protocol, KnowNow allows applications that speak HTTP to publish events without modification. Topic routing and flexible subscriptions enable dynamic data aggregation and syndication, without knowing all endpoints or user domains. Finally, by riding atop the existing technologies that most companies already have in place, KnowNow makes integration fast, incremental, and cost-effective.
KnowNow's flexible architecture makes it possible to create integrations that were not economically feasible before.
Concepts
This chapter gives a detail description of the KN Event Router, Router terminology, and the Microservers.
KnowNow Event
Router
The KnowNow Event Router accepts events as input, then routes them by sending them to other destinations. This is comparable to an IP router: it accepts packets from one network and sends them to a specified destination. An IP router interconnects networks; a KnowNow Event Router interconnects applications. Applications publish events to the Router, and the Router copies those events to other applications that subscribe to them. Currently, the KnowNow Event Router accepts published events via HTTP/HTTPS as well as subscription requests via HTTP/HTTPS; other protocols are possible but have not yet been implemented.
What is an Event?
An Event is the basic packet of information that the Router knows how to deal with. KnowNow Events are made up of key/value pairs (e.g. `price: $59.03'). Applications can use any key/value pairs that make sense for their application; KnowNow reserves all headers beginning with `kn_' for system usage.
A typical event might look like the following:
This event contains 3 headers defined by the application (price, qty, and symbol) and two kn_ headers that have special meanings. For more information on other headers that have special meaning to the Event Router, see See What is an Event Header?.
kn_id holds a unique identifier for this event. Applications can generate their own id's (by including a kn_id header and value when the event is published to the Router) or can rely on the Router to generate them; if an event is published to a Router without a kn_id field , the Router will automatically generate and apply a unique kn_id header .
kn_time_t contains the time (measured in seconds since Jan. 1, 1970) this event was first published. Here again, the Router will apply this header to any event it receives which does not already contain a kn_time_t field. In the example above, the event was first published on Thursday, July 26, 2001 at 11:42:23am.
price , qty , and symbol are application-defined headers that have meaning to the publisher and to the subscriber.
|
|
The KnowNow Event Router
does not place any restrictions on these headers; non-ASCII data need only be
encoded in UTF-8 format before transmission into the Router. |
What is an Event Header?
A number of headers are predefined for use with the KnowNow Event System. Some of these headers are generated by the Router, others are supplied if missing, and still others are simply recommendations made to applications developers for conventions to be used in their applications.
In general, KnowNow reserves the right to use headers beginning with the prefix kn_ for its own use; application-defined headers should not use this naming convention.
What is a Topic?
Every time an event is published to the Router, it must be published to a topic. Think of a topic as the address or channel to which an event gets published (topics are also important for subscriptions as you'll see below). Every topic on a KnowNow Event Router is fully addressable as a URI (Uniform Resource Identifier). For instance, the topic named / stockQuotes on the Router running on gravitar.bigcompany.com can be addressed as:
http://gravitar.bigcompany.com/kn/stockQuotes
Much like a standard web server manages a hierarchy of URLs, the Router manages a hierarchy of topics. So an application might use / stockQuotes/nasdaq for publishing a feed of Nasdaq price updates, / stockQuotes/nyse for publishing NYSE updates, etc.
|
|
If an event is published to /stockQuotes , only subscribers to that topic will be sent a copy of the event. Subscribers to / stockQuotes/nyse will not receive the event. |
The topic hierarchy is initially empty when you start up a brand new Router, and topics get created on the fly as events are published into them.
What is a Route?
One of the most powerful things the KnowNow Event Router allows you to do is set up routes out of a topic. When an event is published into a topic, the Router automatically forwards that event along all the routes out of the topic. Routes can be made to any URI (currently only HTTP and HTTPS protocols are supported). This means, for instance, that you could create a route from the topic / stockQuotes/nyse to a CGI script or servlet on another machine (e.g. a route to http://weblogic.bigcompany.com/servlet / nyseListener ). Whenever an event is published into /stockQuotes/nyse , the Router will make an HTTP POST request to the URI referenced in the route, sending the event as part of the request. Since the destination of a route can be any URI, routes can be made between topics on the same KnowNow Event Router, different Routers, or from the Router to any application that can receive an HTTP POST request. In addition, you can create an unlimited number of routes out of a given topic, making fan-out of events easy.
Metadata Subtopics
The Router maintains metadata subtopics for every topic in its hierarchy. These are called kn_routes subtopics and kn_subtopics . To illustrate what these topics do and how they can be used, here is an example topic hierarchy:
/stockQuotes/nyse/kn_subtopics
/stockQuotes/nasdaq/kn_subtopics
kn_subtopics topics contain an event for each of the subtopics of the given topic. So, for instance, /stockQuotes/kn_subtopics contains an event for /stockQuotes/nyse , one for /stockQuotes/nasdaq , and one for /stockQuotes/kn_routes . The Router will publish an event into the kn_subtopics topic whenever a subtopic gets created; thus if the subtopic /stockQuotes/ftse were to be created, a new event would be published to / stockQuotes/kn_subtopics referring to the new subtopic.
Similarly, kn_routes topics contain an event for each of the routes out of a given topic. For instance, if a route were created from /stockQuotes/nasdaq to a new topic called /stockQuotes/all , the Router would publish an event into /stockQuotes/nasdaq/kn_routes with the full URI of the destination topic (in this case http://router.bigcompany.com/kn/stockQuotes/all ). Just as with kn_subtopics , it is possible to subscribe to kn_routes topics; applications can then receive notification whenever a new route is created out of a given topic. Again, just as with kn_subtopics , only the Router may publish to a kn_routes topic.
Subscriptions vs. Routes, and Journal Topics
The KnowNow Event Router does not distinguish between a subscriber to a topic and a route out of that topic. In fact, since the Router only routes to URIs, what is referred to as a subscription in another Publish and Subscribe system is just a route out of a topic in the KnowNow Event Router. The KnowNow Event Router can route an event to a subscription, to a topic (dynamic aggregation), and even route to another Router (data distribution). At the very simplest conceptual level, what the Router does when it receives an event to publish is as follows:
· Determine to which topic the event is destined
· Create that topic if it does not already exist (and add a kn_subtopics entry to the new topics' parent topic)
· Deliver the event into the topic
· For each entry in the topic's kn_routes subtopic, send the event to the URI specified in the route. Current Event Routers only support HTTP and HTTPS protocol URIs; the Router sends the event data as an HTTP POST to the specified location.
KnowNow's Microservers allow applications to maintain a persistent connection to the Router, so that when an application subscribes to (creates a route out of) a particular topic, any events published to that topic are sent down to the application over a live TCP connection instead of the Router making an HTTP POST to the application. Most applications cannot receive a POST from the Router, so they use the Microserver as a connection to the outside world. Microservers accomplish this through the use of a third (and final) type of special topic known as a journal topic. A journal topic is in essence a URL that's accessible on the Internet. This allows an application to access the journal topic and receive data (events) from the Router.
A journal topic is any topic whose final path element is kn_journal . For example,
/who/steve/kn_journal is a valid journal topic, while /who/stevekn_journal is not.
Journal topics are special in that:
· they don't have subtopics (no kn_subtopics )
· they cannot have routes out of them ( no kn_routes subtopic ).
Journal topics can only be used as the target of a route from another topic (e.g. a route from /stockQuotes/nasdaq to /who/steve/kn_journal ). Journal topics are the endpoints to which Microservers can connect to create tunnels, the persistent TCP connections to the Router.
When an application initializes a Microserver, the Microserver connects to the Router (using an HTTP GET request) and creates a tunnel from a random journal topic name. Some Microservers use a naming convention for their journal topics such that they all begin with /who , but this is not required by the Router. As long as the journal topic ends with /kn_journal , the Router will create the journal and associated tunnel.
Later, when the application asks the Microserver to subscribe to a given topic (e.g. subscribe to /stockQuotes/nyse ), the Microserver actually creates a route out of /stockQuotes/nyse to the journal topic associated with its connection. When an event is later published to /stockQuotes/nyse , it will be forwarded along the route to the users' journal, and from there sent down the tunnel to the Microserver.
The Router currently supports two kinds of tunnels:
Microservers select the kind of tunnel to create when they connect to the Router. At this time, all Microservers use simple tunnel format to communicate with the router, with the exception of the Javascript Microserver.
Finally, journal topics support one extra feature--checkpointing. All events delivered to a journal topic are modified to include a kn_route_checkpoint header. The value of this header is a unique identifier for the event that can be used when reconnecting to a pre-existing journal. In the case where a Microserver's tunnel connection is broken for some reason, it can reconnect to its journal topic and provide the last checkpoint id it successfully received. The Router then sends all the events since that checkpoint down the tunnel, making it possible for the application to be sure it has received all its events despite the reconnect.
Event lifetimes and Initial Route Population
When an event is delivered to a topic, it is stored inside the topic and then forwarded along any routes out of the topic. Unlike other Publish and Subscribe systems, the Router allows events to live beyond the time of their initial publishing. Events can include a kn_expires header that sets the time the event will expire. Events published to the Router that do not include this header are subject to a default expiration time, configurable in the Router's runtime configuration. (The shipping default of this parameter is currently 3600 seconds, or one hour).
When a route is created out of a topic, the route creation request can include parameters for initial route population, which requests the Router to forward existing non-expired events along the route immediately. Route requests can ask the Router to deliver the last n events (known as do_max_n ) or all the events seen within the previous n seconds (known as do_max_age ).
Status Events
When processing certain requests, the Router generates events (known predictably as `status events') to describe the outcome of these requests. These status events are normally written as the response to the inbound HTTP request to the Router, but can be redirected to any URI (including other topics on the same Router or external applications). See the HTTP API reference for more information on the status events generated by various requests.
Permissions
Permissions in the KnowNow Event Router allow an administrator to control who can publish to a topic, who can subscribe to a topic, and who can create a subtopic of a topic. Authentication of users is handled in standard HTTP authentication fashion; See the Router's Getting Started guide for configuration details for the Router and RFC 1945 (HTTP) for more information on HTTP authentication.
Router Utilities
The following utilities are installed with the KnowNow Event Router:
For more information on the utilities, see the corresponding utility documentation.
Microservers
KnowNow Microservers hold open persistent HTTP or HTTPS connections between a Router and any application running in Java?, C, Windows or Web Browser environments. Microservers should be used when applications need to receive events and it is inconvenient or impossible for them to accept KnowNow's native event transport, HTTP POST. Normally this happens when the subscribing application is behind a firewall or NAT, or when an HTTP server is not available in, or appropriate for, the application environment.
KnowNow Microservers are available, but not always needed, as KnowNow's Web-native architecture makes it simple for any system to communicate with the KnowNow Event Router using HTTP POST. However, Microservers are employed in situations where it is less than ideal to send or receive HTTP POST. For example, HTTP/1.1 implementation is not conveniently accessible in the integration environment; or a KnowNow Event Router needs to be connected from behind a NAT Router, a firewall or other network configuration that would not allow inbound HTTP connections; or perhaps developers want to take advantage of a simple API for publish-and-subscribe instead of writing to the HTTP API.
In any of these cases, KnowNow Microservers present a local API for router topic management, giving developers a convenient and easy way to send and receive events to the Event Router from the native application environment, with full functionality to publish, subscribe, and unsubscribe.
The KnowNow Event Router is shipped with the JavaScript Microserver. Additionally, the following Microservers are also available:
JavaScript Microserver
The KnowNow JavaScript Microserver is software that frees up Web browsers from the sluggishness of the Web's usual request/reply communication process, to open real-time, interactive data channels between browsers and applications, all with zero installation to the browser. The following browsers and platforms support the KnowNow JavaScript Microserver:
· Internet Explorer 4.0 on Microsoft Windows 95®, Windows 98®, Windows NT® 4.0
· Internet Explorer 5.0/5.01/5.5 on Microsoft Windows 98®, Windows NT® 4.0, and Windows 2000®
· Netscape Navigator 4.76 on Microsoft Windows 98®, Windows NT® 4.0, Windows 2000®, Sun Solaris? 2.8
For more information on the JavaScript Microserver and its API, please refer to KnowNow Event Router Programmer's Guide.
Windows Microserver
The KnowNow Windows Microserver allows any Windows application to send and receive KnowNow events. In addition to the standard Microserver functionality, the Windows Microserver can present ActiveX and COM interfaces to the KnowNow architecture, as well as locally queue outbound events if the network or Router is unavailable. The Windows Microserver is supported on Microsoft Windows 98®, Windows NT® 4.0, and Windows 2000®. It allows for integration with VB/VBA/Win32 and IIS. For more information on the Windows Microserver and its API, please refer to the KnowNow Microserver for Windows Programmer's Guide.
Java Microserver
The KnowNow Java Microserver is supported on Java? 1.2 virtual machines and in the J2EE? framework. It can be used to connect Java applications, servlets, EJBs, and other types of Java environments. On top of the standard Microserver functionality, the Java Microserver offers JMS pubs/sub API support in BEA Weblogic? 6.0 and IBM Websphere 4.0 application servers. For more information on the Java Microserver, please refer to the KnowNow Microserver for Java Programmer's Guide.
C Microserver
The KnowNow C Microserver is supported on Solaris? 2.8 as a shared object library and allows developers working in C or C++ environments to take advantage of Microserver features. For more information on the C Microserver, please refer to the KnowNow Microserver for C Programmer's Guide.