10.2.0.2 Listener bug, new child listener process forked

Today I solved a nasty intermittent problem we’d been having on one of our Production servers so I thought I’d blog about it here in case it saves someone else some pain. The symptoms were

1) No-one was able to connect to the database.

2) There were two listener processes running, which was apparent from a ps-ef|grep tns

oracle 2647896 1495262  1 May 18 -  0:00 /oem/oracle/product/10.2/bin/tnslsnr LISTENER 
						-inherit 
oracle 1495262       1  1 May 13 -  2:32 /oem/oracle/product/10.2/bin/tnslsnr LISTENER 
						-inherit

(Note the different start times and the later process was definitely not started manually.)

This had happened once or twice and we found that when we killed the child listener (note that one listener’s parent process ID is the process id of the other) everything worked correctly.

This time I had a good look through the listener.log and noticed these warning messages related to the ONS service, even though we’re not using RAC.

WARNING: Subscription for node down event still pending

It took a while to track it down, but it turns out that this is a known, but unpublished bug number 4518443 in 10.2.0.2 (and no doubt previous versions), fixed in 10.2.0.3. For those of you with Metalink accounts, the relevant note is here. You can either apply patch 4518443 or, as a temporary workaround, disable that service by adding the following to your listener.ora configuration and restarting the listener.

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

16 comments

  4. Once again, I forgot to mention the O/S – AIX 5.3. However, as the Metalink note says

    This problem can occur on any platform.

    All new connections via TNS listener hang, no errors reported

    Reply

    1. No RAC… so was something else listening on port 6101? Or was the listener configured to look for ONS on a different port, on which some different app was running? Seems like I’ve heard that Oracle Application Server can cause this problem; were that related to the problem?

      Reply

      2. even without RAC or iAS, if you use ASM, an ONS will be running and therefore you can face this problem.

        Reply

        1. Good point, Martin, thanks.

          Because I rarely get to work with ASM at the moment, I wouldn’t have thought of this.

          Reply

  6. Good Article Doug,

    I ran into this problem, saw your article and helped me to resolve the issue.
    Thanks for sharing the information.

    Reply

  8. In 10.2.0.4 (EE on hp-ux, at least) it still leaves copious warning messages in non-rac listener.log.

    SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF (or appropriate listener name) in listener.ora fixes.

    I’m not using iAS, RAC or ASM, but those messages are still annoying. It’s seems the targets.xml file used by dbconsole defines which listener is used, simply using dbconsole is using enough of ONS to create these warning messages.

    Reply

  9. I ran into the same problem at yesterday
    midnight on our production server and the way I fixed it was same as Doug’s. I want to figure out why the second listener would come out. I search from google and get the snswer from your blog. Now I know it is not people manually started process, it is a bug in 10.2.0.2 and lower release (we are using 10.2.0.1).

    Thanks for the Doug’s good blog and all other people’s helpful information related this issue.

    Reply

  11. Hi This is chirdeep from India, blog is really very much helpful. We actually have a same scenario but our database version is of 9.2.0.8. Hence if some one could help it would be very much helpful.

    Regards,
    Chirdeep

    Reply

Leave a comment

Your email address will not be published. Required fields are marked *