Scalability, Performance and Availability

Does the Container take care of Distributed Garbage Collection?

Distributed garbage collection is a generally bad idea. EJB explicitly avoids DGC by leveraging good ideas, like containers, transactions, passivation, and server-side timeouts.

Our implementation of RMI-over-IIOP provides RMI signatures on the client side, but uses POA on the server side. As such, it uses CORBA object-activation which, for good reason, has never required DGC.

DGC is a non-issue if you take activation into account. You can encapsulate all info that identifies a bean in an IOR. Since the EJBObject can be activated any time a request comes in, and since all required state is part of the IOR, you don't need the EJBObject to sit around occupying memory. Further, since the server reserves the right to deactivate this object at will, there's no need for GC or for Java 2 weak references or whatever.

What is the overhead of intra-VM bean calls compared to "normal" Java method invocations?

Note that a container does have "real" work to do, and a call to a local bean will be expected to be slower than a normal call. But there are many optimizations and configurable options that reduce the hit. In terms of the actual overhead:

marshalling: unless you specify -DEJBCopyArgs, there is none for intra-bean in-process calls.

Corba invocation: there is only one ORB for a set of in-process beans. Visibroker is optimized to handle in- process calls much more efficiently than remote calls. Hence there is no overhead for intra-VM bean invocations.

transactions: Make sure only a single transaction is started per RPC, if possible. Make sure your access to the in-process beans are not starting their own transactions. Typically, all your beans should have transaction-mode "Required". This mode will cause a transaction to be begun when the call first enters the container, and the same transaction to be reused by all calls made within the container. Typically, this is what you want. If each access to your entity beans is occurring in a new transaction, you need to load in the state of the bean for each call. Run all the accesses in a single transaction, and you go from 1000s of JDBC calls, to 1 or two.

security: unless you have specified security, no ACL is being done.

reflection: this is not an issue, since the Container is designed to do all reflection at initialization time only.

You can use the EJBTimers or EJBDetailTimers flags to see exactly what is happening in the container.

How does the Visibroker POA architecture help in scalability

Inprise EJB container has the most sophisticated implementation of an object adapter that is available in the market. One of the main reasons for that is the use of POA infrastructure which gives two distinct advantages to our container that no other product can offer...

1. Parallelized access to Entity Beans. Other EJB containers have to "serialize" the call to the Entity bean which affects the scalability and performance. Without a POA like infrastructure your Entity Beans WILL NOT BE ABLE to service multiple client invocations simultaneously. That is a serious limitation if you are looking to service millions of concurrent transactions.

2. Passivation and Reactivation of stateful session beans. Most of the implementations out there do the Passivation correctly but have to keep some pointer information in memory to be able to reactivate the session bean. We have 0 (zero) bytes in memory for each passivated session bean. This is a HUGE advantage since at some point other EJB containers will reach their memory limit but with Inprise EJB container you are not limited.

What other Visibroker ORB features ensure the scalability and reliability of EJB applications?

- Thread Management

- Connection Management, Connection and Request Timeouts, Connection Caching

- Unconstrained Partitioning

- OAD for high availability

How is the Naming Service used to provide load balancing?

The Clustering feature of the Naming Service needs to be turned on. It is as simple as setting the following property (either in the property file or as a -D switch)
vbroker.naming.propBindOn=1
What happens then is that all object registrations of the same name in the NS gets "clustered" so that successive lookups or a particular name round robin among the cluster members. This way we get load balancing at the time of home lookups.

Failover Support in the Inprise EJB Container

You can register the same sets of beans in each container to get a homogeneous clustering capability, or you can partition your beans across various containers to get a heterogeneous clustering capability.

You should then be able to kill instances of the Containers and your clients will fail over to the replicas. Your stateless SBs and entity beans would fail over immediately. You can configure your stateful SBs to fail over, by sharing a Stateful Session Storage Service among them. To do this, run a single instance of the container with only the -jss flag, and don't specify the -jss flag to all the other containers. This will centralize stateful session state in that single jss instance. This should allow fail over of stateful SBs, as well.

Failover is supported by leveraging the Naming Service and the osagent. We support failover of all types of EJBs. To summarize the behavior:

1) Stateless Session Beans. If you run multiple containers (e.g., VMs) with the same Stateless SB, they will act as fail-over replicas of each other. So call (1) to a Stateless SB will go to VM 1. If VM 1 shuts down, and the SB is installed in VM 2, then call 2 will go to VM 2. The client will be unaware of the fail-over. Since the beans are stateless, this is correct behavior.

2) Entity Beans. From the Container standpoint, Entity Beans are similar to Stateless Session Beans, except that before being able to use a replica entity, it needs to be loaded in. The current transaction will fail, as it should. Both the resource and the synchronization objects associated with the original (failed) entity in VM 1 will be unavailable, and the transaction manager, presuming rollback (as per the OTS/JTS specification) will roll back the current transaction. However, subsequent transactions will simply fail-over to use the replica entity in VM 2, and will continue correctly.

3) Stateful Session Beans. Stateful SB are passivated into a JDatastore database, which is available via an IDL interface. This Session Storage service can either run in process, or out of process. If run out of process, multiple Containers can share a single Session Storage service. So, let's assume we have two Containers (one in VM 1, and one in VM 2) running replicated Stateful Session Beans, and the Session Storage service running in VM 3. Let's say a client creates a shopping cart (a Stateful SB) in VM 1. The shopping cart will be automatically passivated every 5 seconds (a tuneable parameter). Then let's say that the client puts a book into the shopping cart, and then thinks for a while. The shopping cart will be stored persistently within 5 seconds in the Session Storage. Now, lets say VM 1 crashes. The shopping cart EJBObject will automatically fail-over to VM 2. This container will see that it doesn't have the state for the user's shopping cart activated, and it will load the state from the storage server (VM 3). The client will now continue using the shopping cart with its contents intact.

This implementation has some problems:

* What happens to state that changed between the last passivation and the VM crash?
It is lost. By default, you can lose up to 5 seconds of work. i.e., you can lose all the items you put in your shopping cart within the last 5 seconds. This doesn't seem like a big problem. If need be, you can set passivation to occur more frequently.

* What happens if VM 3 crashes?
Your shopping cart will be unavailable until the Session Storage service is brought back up. Once the service is back up, you are back to where you were before VM 3 crashed. This is really the same issue that you would have if any database failed. Note, however, that since the Session Storage API is IDL, it is possible for you to write your own highly-available service, possibly built on Oracle, or something extremely robust. That being said, note that it is in fact less likely that VM 3 crash, than either VM 1 or VM 2, because VM 3 is not running user code. VM 1 and VM 2 are both running Containers with user-code in it. VM 3 is only running JDatastore and the ORB, which both tend to be quite stable, as far as Java code is concerned.

Note: that this implementation does not imply that every bean gets published to the OSAgent. This would be a horrendous (albeit typcial) use of the OSAgent. Rather, services representing the beans get published (one per type of bean). This is accomplished via some POA/VB4 magic under the covers. It is conceptually similar to the VB3 notion of service activators, but considerably more sophisticated. Only the bean's POA is registered.

Also, if a large number of clients are all using a primary container, which goes down, all the clients will have to fail over to the replica simultaneously. This might be painful. That being said, if you have a "a large number of clients", then typically you will be running through some sort of concentrator (probably a servlet in a Web Server) and in reality there is only a small number of clients.

Are you assuming that the beans to which you hope to fail-over, have already been created in the other container?

No assumption beyond those inherent in the EJB spec. is being made:

1) For stateless session beans, the container "magically" has as many beans available as needed. They don't need to be pre-created.

2) For entity beans, the container loads in a bean as needed. Since they already exist in the database, they don't need to be pre-created.

3) For stateful session beans, yes, the object must be created before it is used (as per the EJB specification), and after a timeout it will be stored in a shared repository. Once the bean is created, and passivated, it can be activated into any other container sharing the repository. It does not need to be pre-created in the other container.

How should I tune the Container JVM for maximum performance?

There are a couple of VM flags that need to be set for reasonable performance.

The main issue is that VBJ based programs require a good deal of memory to run well. Unfortunately, the default VM configuration does not provide a good deal of memory. Thus, the VM spends a lot of its time (> 80% in some cases) garbage collecting. By increasing the default memory size, and by decreasing the amount of memory VBJ uses, things run significantly faster.

For anyone doing performance testing, we would recommend the following flags:

Servers: -Xms32m -Xmx32m -Dvbroker.orb.streamChunkSize=1024
Clients: -Xms16m -Xmx16m -Dvbroker.orb.streamChunkSize=1024

These flags go between the driver (e.g., "java" or "vbj") and the name of the Java main class. These flags have the following meanings:

-Xms32m         Set the default heap size to 32 Mb
-Xmx32m          Set the maximum heap size to 32 Mb
-Dvbroker.orb.streamChunkSize=1024 
                Make the default buffer size 1024 bytes
Increasing the heap size is useful because your program can run for longer between GCs. It is much more efficient for the GC to make big runs infrequently than to make small runs frequently, due to the significant overhead of doing GC.

Enabling verbose GC (using -vertbosegc) is useful to see what is going on. It will give you an indication of how much time your program is doing GC. For example, if you don't specify the heap size and go with the default setting, GC'ing is done every 100-120 ms. This is rediculous since it takes about 80-90 ms for the GC to start and stop (and hence your program runs 3-4 times as slow). You would like to see the GC running at most once every 5 seconds, which was achieved on a particular machine with the above heap size settings, on Solaris using JDK 1.2. Your mileage will vary on different OSes and VMs.

Changing the default chunk size for streams (e.g., buffers) is useful as a performance tweak at the ORB level where buffers won't burn through the heap as quickly.