It was last day for Anil at our office. I am really going to miss this guy. I think I complemented this guy very well. It's very important to have such a partner specially when working in Product support environment.
Anil, Thank you very much for all the things you did for me and for all the times we have worked together. I will always miss you.
All the best in your next endeavors.
PS: I will learn to play snookers and lets play someday when I visit your place.
Friday, July 20, 2007
JVM Crash
In my view JVM crash is the most dreaded problem that could ever happen to an App server on production machine. Unfortunately, one of our clients production machine has been crashing regularly with a certain version of JDK at high concurrency, We have looked at few crash reports and determined that the crash appears to be happening in one particular piece of code.
Our client being Premium partner with Sun was able to take it up with Sun Support Team. They needed some info from our team as well. So a conference call was setup.
This is how it went on
Sun: We have looked into the crash reports. But would like to know if there is any more info you could provide.
Me: Sure( I ended up saying the changes that we made).
Sun: Anything else?
Me: We might have lot of things to say. But what is that you are looking for?
Sun: I am just trying to get more info on the problem as there seems to be nothing in the logs. How did you determine that it was a problem with JDK?
Me: It crashed and produced a crash report which we shared with you, You being the developers for this JDK should be able to say more about the crash and why t happened
Sun: I do not see any specific info from the logs that you have sent
Me: But we see that there is this crash that always happens in a compiler thread.
Sun: Oh okay, that is good info. Let me forward this to my analysis team. But can you tell me where you got this info from?
Me: Did you happen to have a look at the crash logs? It says so in the logs.
After a day Sun team has come up with the outcome of the analysis. I thought it was impressive. The outcome was that there was a StackOverFlow during class-native compilation(Thanks to Rajiv for taking pains to explain me about this compilation).
There were few params that were suggested. One of them was to increase the compilerthreadstacksize. We have tried with few options 1024,2048 but to no avail. We had to go back to Sun team to report about our unsuccessful attempts. So there was one more angle that was brought into the picture. There might be some recursion in the code due to which the stackoverflow was happening. Well that sounded logical to me. But where was this happening? Since the current stacktrace in the jvm crash reported at jvm.dll, I am convinced to believe that it was happening somewhere in the native code of JVM. But the Sun team had to differ here. We wanted to know how we can check where this is happening.(All these were through email correspondenses).
Next day there was an email from the Sun team in which they have provided one way to check where the StackOverFlow was happening.
"We would like you to capture a thread dump before the crash so that we can analyze the issue. For accuracy, it would be really good if you can capture at least 3-4 thread dumps."
Whoa!!! How do I capture a thread dump before the JVM crash? I need some real Oracle to help me out in predicting the time of crash so that I can capture the thread dump before the crash!!
Well said SUN!!!!
Our client being Premium partner with Sun was able to take it up with Sun Support Team. They needed some info from our team as well. So a conference call was setup.
This is how it went on
Sun: We have looked into the crash reports. But would like to know if there is any more info you could provide.
Me: Sure( I ended up saying the changes that we made).
Sun: Anything else?
Me: We might have lot of things to say. But what is that you are looking for?
Sun: I am just trying to get more info on the problem as there seems to be nothing in the logs. How did you determine that it was a problem with JDK?
Me: It crashed and produced a crash report which we shared with you, You being the developers for this JDK should be able to say more about the crash and why t happened
Sun: I do not see any specific info from the logs that you have sent
Me: But we see that there is this crash that always happens in a compiler thread.
Sun: Oh okay, that is good info. Let me forward this to my analysis team. But can you tell me where you got this info from?
Me: Did you happen to have a look at the crash logs? It says so in the logs.
After a day Sun team has come up with the outcome of the analysis. I thought it was impressive. The outcome was that there was a StackOverFlow during class-native compilation(Thanks to Rajiv for taking pains to explain me about this compilation).
There were few params that were suggested. One of them was to increase the compilerthreadstacksize. We have tried with few options 1024,2048 but to no avail. We had to go back to Sun team to report about our unsuccessful attempts. So there was one more angle that was brought into the picture. There might be some recursion in the code due to which the stackoverflow was happening. Well that sounded logical to me. But where was this happening? Since the current stacktrace in the jvm crash reported at jvm.dll, I am convinced to believe that it was happening somewhere in the native code of JVM. But the Sun team had to differ here. We wanted to know how we can check where this is happening.(All these were through email correspondenses).
Next day there was an email from the Sun team in which they have provided one way to check where the StackOverFlow was happening.
"We would like you to capture a thread dump before the crash so that we can analyze the issue. For accuracy, it would be really good if you can capture at least 3-4 thread dumps."
Whoa!!! How do I capture a thread dump before the JVM crash? I need some real Oracle to help me out in predicting the time of crash so that I can capture the thread dump before the crash!!
Well said SUN!!!!
Thursday, July 19, 2007
HP ServiceGuard
Been busy last couple of weeks. Some really nice things happened during these weeks. We had an opportunity of integrating Pramati Server with HP Serviceguard. We had one of our customer who was looking for clustering solutions. We have offered Pramati Cluster, which offers fail over, and load balancing. The End User had HP machines for production enviroment. With these HP machines he happened to purchase the Serviceguard framework which manages the fail over mechanism and manages switching of IP address( virtual IP address). We had setups with OS level clustering such as Windows Clustering, Sun Clustering. However, these happened to manage the things at a machine level. Well it really depended on how you are trying to configure it. Generally these run with Active-Active or Active-Passive configurations. Active-Active means that both the machines are in active state and the data replication happens on both machines and the load is balanced between the two machines. This is achieved by using a Virtual IP address that forwards the traffic to back end machines. In Active-Passive configuration, only one machine is active at any time and all the traffic that hits virtual IP address is routed to the active machine.
However, the HP Serviceguard was managing the thing at package level. For this service any application registered with it is a package and it manages the package between the cluster nodes. That is to say App server can be running on one cluster machine and Database on the other. These two are independent and could be running anywhere on the cluster machines. With this background, I assume it is now safe to go into the details of what happened during this integration.
The End User has called up asking for few queries on how Pramati Server can be fit into the picture. Pramati Server has clustering solution which works independent of OS level clustering. We have proposed the same. However since there is no single point of entry for the traffic for cluster nodes, we were left with either using a loadbalancer or leveraging on the HP Serviceguard framework to manage the traffic routing. The Application vendor was in favor of leveraging on the existing HP Serviceguard framework. Hence, there were series of conference calls setup with HP implementation team, the Application vendor and us.
This is where the real fun has begun.
The following is the snapshot of the conversation that took place between the HP implementation guy at the clients place and me:
With all introductions done…
Me: How does this HP Serviceguard thing work? ( Though I have done some ground work on the HP Serviceguard thing, couldnot find any relevant docs on how the applications should interact with it).
HP: The HP Serviceguard has to register your application and a virtual IP address configured for your application.
Me: Okay, how do we register the application with HP Service?
HP: You will have to provide us with few scripts using which we would register your app into HP Service.
Me: (I was happy to hear this. Good just few scripts and its all done). Okay, what would these scripts be and what is the desired functionality of the scripts.
HP: I am not really sure, but all I know is that you will have to provide me few scripts.
Me: ( What the !!!!). Okay, if you can tell us what these scripts should be doing, we might quickly put up few scripts for the desired functionality.
HP: ( He repeats the same thing). I am not really sure, but all I know is that you will have to provide me few scripts.
Me: ( Now I am beginning to worry. This is not going to end soon). Okay, then who would know about what kind of scripts are required?
HP: The HP Serviceguard team would know.
Me: Are you from the HP team or a reseller of the product?
HP: I am from HP team, but from implementation team. So I do not know what kind of scripts. All I know that is few scripts are supposed to be provided by you.
Me: (Does any shell script do? Such as the one to display simple helloworld on the Console?) Okay, can you give me numbers of your HP Service team so that I can talk to them?
HP: You wouldn’t be able to talk to them directly without any case id.
Me: Okay, can we create a case for this and then talk to them.
HP: Sure, we should be able to do that. Shoot across an email on the info required and I will get back to you.
So I shot across an email to this guy and waited for a day. Nothing happened on it. So decided to call up and check what’s happening:
Me: Looks like we haven’t got a reply from your team. Since we have logged a case, can we call them up and check with them?
HP: Yes, but I do not have the numbers for the HP Serviceguard team.
Me: Okay, how do we get this?
HP: Can you call up HP Sales team and check with them?
Me: (Sigh….) Okay, I will call them up and check.
Now I call up HP sales team.
Me: Hi, this is ….. We have one customer who is interested in integrating our App server with one of your product. HP Serviceguard. We have few clarifications. Can you help us?
HPSales: I can provide you with HP Support number who should be able to help you
Me: Great.
I call up this number
Me: ( I ended up speaking few minutes about the current situation and what we are looking for).
HP Support: Sure Mr Naveen. Before we can start with any of your queries, can I have the serial number of the machines?
Me: Sure, we have few HP machines at our place. So will the number from any of them do?
Here comes the ace..
HP Support: No, the serial numbers should be of the machines on which HP Serviceguard framework was purchased.
Me: Okay I will get back with these numbers
Now I call up this HP guy at the clients place and ask him for the numbers. I asked him if he can give me the serial numbers. For some strange reasons he was reluctant to give me these numbers.
Finally with some intervention from Application Vendor and End User, we could get a sample script that was used to integrate MySql with HP Serviceguard. So we just mimicked these scripts and provided them to this HP implementation guy. After a day we got a call from our App Vendor saying all went well and Pramati Server has been registered with the scripts provided by us. One more happy customer.
But I really feel that HP Serviceguard is the one that provides clustering solution, they are supposed to have some documentation on what is required from applications such as App servers, database etc. It should have published its API if any and should be a part of the software that they sell. I wonder why it is not the case.
Saturday, June 30, 2007
Wednesday, June 27, 2007
Session Stealing-2
Contnuation of the previous posting..
My apologies for the abrupt ending of my previous post. Something important came up due to which I had to end it abruptly.
Session hijacking is generally crafted using the following methodologies:
1. Request-Response Sniffing
2. Cross site scripting
Well, the first one can be prevented when the whole session is handled through https. However, if part of the session is handled through http and is switched over to https, then the sniffer would be able to pick up the session id transferred in the http session. To avoid this Pramati Server uses a special cookie in addition to the sessionid cookie. This pair is validated when trying to access the https pages. As the second cookie is set via https, the sniffer would not be able to view it easily. When the sniffer/hijacker sends a https session without the secret cookie, the server would understand that this is not from the authenticated user and hence will deny the response.
Regarding second, you should check if the server is immune to XSS (cross site scripting) vulnerabilities.
My apologies for the abrupt ending of my previous post. Something important came up due to which I had to end it abruptly.
Session hijacking is generally crafted using the following methodologies:
1. Request-Response Sniffing
2. Cross site scripting
Well, the first one can be prevented when the whole session is handled through https. However, if part of the session is handled through http and is switched over to https, then the sniffer would be able to pick up the session id transferred in the http session. To avoid this Pramati Server uses a special cookie in addition to the sessionid cookie. This pair is validated when trying to access the https pages. As the second cookie is set via https, the sniffer would not be able to view it easily. When the sniffer/hijacker sends a https session without the secret cookie, the server would understand that this is not from the authenticated user and hence will deny the response.
Regarding second, you should check if the server is immune to XSS (cross site scripting) vulnerabilities.
Session stealing
You have heard about username/password, identity stealing. Did you ever hear about session stealing/hijacking? Session stealing is the act of taking control of the user session after having obtained/generated authenticated session id. Following is a bit intro on session and session id.
HTTP is stateless protocol. To maintain the state of the logged in users and identify them, the servers depend on the session ids. Session is a series of interactions between two end points( in this case server and client) that happens during the span of single connection. Session ID is a random alphanumeric string that a web server assigns a specific user for the duration of that visit. Once the user is logged into the web site/application a session is created for that user and the server hands out the session id to the browser when sending the first response. The browser would send this Session ID to the server on all the subsequent requests. As long as the user makes the requests from the same browser without closing and reopening it, the web site would not ask for the login information. This is coz,the server/application validates the session id received from the browser and would check if the user with that session id is logged in.
Now that we know what session and session id is let us move on to how it is transmitted between the server and the browser. One of the most used method is to set the session id as cookie on the browser JSESSIONID in case of J2EE and ASPSESSIONID for .NET servers. If you have any tool such as IEHttpHeaders for IE or LiveHttpHeaders for Firefox, you would be able to see something similar to this in the response from the server.
Status=OK - 200
Date=Wed, 27 Jun 2007 11:13:45 GMT
Content-Type=text/html
Set-Cookie=JSESSIONID=978704440835854248; Path=/
X-Cache=MISS from HYD-MDU-CACHE2
Via=1.0 HYD-MDU-CACHE2:515 (squid/2.6.STABLE12)
Connection=close
What you are seeing here is the Session ID Cookie and the value of the session ID. Anyone sniffing on the network packets between your server and you would be able to easily flick this info. Now once he has that session id, he would send the request to the server with the session id along with the request (You can use Tamperdata extenstion of Firefox to do this)
Now you would start thinking. This guy has some random number generated by the server and that is passed between my browser and server. So what? Just to remind you, this session id is not just another alphanumeric string, as far as the application is considered, this your passport to the application unfortunately a passport without photo on it. Any one who has this session id can get the server tricked into believe that it is YOU who is talking to the server. It is equivalent to some one flicking off your passport and presenting himself as you(Remember no photo on it). Now when application believes that some one else is you, then it would allow that person to do what you would be able to do! Let me put it in few steps
1. You open your bank site and go to the login page and login.
2. Once you are successfully logged in, the server would redirect you to your account details and setting a session id cookie in your browser.
3. When you make any request in any of the bank site, the intelligent browser would send the session id to server along with the new request.
4. The server would verify this id and see that you are already logged on. Hence no more login requests.
5. Now lets say there is a guy in the middle who has been sniffing the requests and responses between your machine and the server. He would be able to see the Session ID cookie that’s shared with you. Now he would pick up the same session id and send it over to the server. Since the id is shared between server and only you, the server would be under the impression that you are the one who is talking to server but it is actually not!
6. Now the guy in the middle would make a request for harmless page with your session id sent along to the server.
7. Server would verify the session id and see that you are already logged in and hence would present the harmless page to the guy in between.
8. From here, he would be able to navigate to your account page effortlessly and view details or do what fancies him at that time.
More about this in my next blog.
Sunday, June 17, 2007
Look who's doing it!!
What if govt flouts pollution control rules? In the pic is one of the waste dumping yards used by GMCH( Greater Muncipal Corporation of Hyderabad). Every day hundreds of trucks dump the garbage collected from all over the city and it is burnt in the evening. Look at the rising smoke. If something goes wrong we have govt to complain to. Now where do we go??
Subscribe to:
Posts (Atom)
