Oracle Coherence Cache error on Weblogic 12c : could not establish a connection error .

We were deploying our coherence cache client to Weblogic 12c server, while the solution was working perfectly fine in the local tomcat server it was giving deployment error on the Weblogic server.

 

The error was com.tangosol.net.messaging.ConnectionException: could not establish a connection to one of the following addresses: [x.x.x.x:9099]; make sure the "remote-addresses" configuration element contains an address and port of a running TcpAcceptor

 I verified that the host and port are open from my machine by telnet the host on 9099 port.We searched the web and oracle support forums but couldn’t find any cause of this problem. This prompted to write this blog so that it can be helpful to others.

While taking another look at the stack trace, I noticed a class cast exception at the bottom of the stacktrace.

Caused by: java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to java.lang.Integer
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.Acceptor.negotiateProtocols(Acceptor.CDB:20)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.Acceptor$MessageFactory$OpenConnectionRequest.onRun(Acceptor.CDB:39)
        ... 8 common frames omit

This gave me a clue that this is another class loader issues with Weblogic. Weblogic server comes with a whole bunch of its libraries / jar which it loads at server start up and if any of those jar clashes with your application jar it can cause weird behaviors.

In our application we were using the coherence-3.7.1.12.jar whereas Weblogic 12c was loading  coherence.jar Version 12.1.3.0.0. Once the root cause was established the fix was very simple.

I updated the weblogic.xml to force weblogic to use the class loaded from the web-app instead of from server.

xml version="1.0" encoding="UTF-8"?> <wls:weblogic-web-app xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.4/weblogic-web-app.xsd">     <wls:weblogic-version>12.1.1</wls:weblogic-version>     <wls:context-root>myapp</wls:context-root>     <wls:container-descriptor>         <wls:prefer-application-packages>              <wls:package-name>com.tangosol.*</wls:package-name>                </wls:prefer-application-packages>     </wls:container-descriptor> </wls:weblogic-web-app>

 Stacktrace of the error :

com.tangosol.net.messaging.ConnectionException: could not establish a connection to one of the following addresses: [10.64.17.208:9099]; make sure the "remote-addresses" configuration element contains an address and port of a running TcpAcceptor
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.initiator.TcpInitiator.openConnection(TcpInitiator.CDB:129)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.Initiator.ensureConnection(Initiator.CDB:11)
        at com.tangosol.coherence.component.net.extend.remoteService.RemoteCacheService.openChannel(RemoteCacheService.CDB:7)
        at com.tangosol.coherence.component.net.extend.RemoteService.doStart(RemoteService.CDB:11)
        at com.tangosol.coherence.component.net.extend.RemoteService.start(RemoteService.CDB:5)
        at com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:57)
        at com.tangosol.coherence.component.util.safeService.SafeCacheService.startService(SafeCacheService.CDB:5)
        at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:27)
        at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)
        at com.tangosol.net.ExtensibleConfigurableCacheFactory.startService(ExtensibleConfigurableCacheFactory.java:687)
        at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureService(ExtensibleConfigurableCacheFactory.java:605) ~
        at com.tangosol.coherence.config.scheme.AbstractCachingScheme.realizeCache(AbstractCachingScheme.java:54)
        at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureCache(ExtensibleConfigurableCacheFactory.java:242)
        at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:205)
        at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:182)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1506)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:313)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1127)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1051)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:949)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:530)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1202)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) 
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) 
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) 
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) 
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) 
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:762) 
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) 
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) 
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:109) 
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:691) 
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:321) 
        at org.springframework.boot.context.web.SpringBootServletInitializer.run(SpringBootServletInitializer.java:117) 
        at org.springframework.boot.context.web.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:108) 
        at org.springframework.boot.context.web.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:68) 
        at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:175) 
        at weblogic.servlet.internal.WebAppServletContext.initContainerInitializer(WebAppServletContext.java:1394) 
        at weblogic.servlet.internal.WebAppServletContext.initContainerInitializers(WebAppServletContext.java:1331) 
        at weblogic.servlet.internal.WebAppServletContext.initContainerInitializers(WebAppServletContext.java:1317) 
        at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1834) 
        at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:2876) 
        at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1661) 
        at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:823) 
        at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:360) 
        at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:356) 
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) 
        at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:138) 
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124) 
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:216) 
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:211) 
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) 
        at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:73) 
        at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:24) 
        at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:729) 
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) 
        at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:258) 
        at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:48) 
        at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:165) 
        at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:80) 
        at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:226) 
        at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:418) 
        at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:51) 
        at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:200) 
        at weblogic.management.deploy.internal.AppTransition$2.transitionApp(AppTransition.java:30) 
        at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240) 
        at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:169) 
        at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:123) 
        at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:210) 
        at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:118) 
        at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78) 
        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_31]
        at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_31]
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017) 
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388) 
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430) 
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456) 
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225) 
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82) 
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488) 
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98) 
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606) 
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77) 
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231) 
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254) 
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413) 
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456) 
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225) 
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82) 
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488) 
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98) 
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87) 
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162) 
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147) 
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548) 
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311) 
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) 
Caused by: com.tangosol.net.messaging.ConnectionException: connection rejected
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.Acceptor$MessageFactory$OpenConnectionRequest.onRun(Acceptor.CDB:66)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor$MessageFactory$OpenConnectionRequest.onRun(TcpAcceptor.CDB:13) 
        at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.onMessage(Peer.CDB:1)
        at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:52)
        at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.onNotify(Peer.CDB:103)
        at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
        at java.lang.Thread.run(Thread.java:662) ~[na:1.8.0_31]
Caused by: java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to java.lang.Integer
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.Acceptor.negotiateProtocols(Acceptor.CDB:20)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.Acceptor$MessageFactory$OpenConnectionRequest.onRun(Acceptor.CDB:39)
        ... 8 common frames omitted