tag:blogger.com,1999:blog-8405107760807432973.post1227336553526918679..comments2024-01-04T09:55:32.459-08:00Comments on Java Concurrency (&c): Lightweight Asynchronous Sampling ProfilerJeremy Mansonhttp://www.blogger.com/profile/04241094734813086257noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-8405107760807432973.post-49792519257441692812022-09-01T22:26:09.309-07:002022-09-01T22:26:09.309-07:00@zhaoxi you can't call JNI functions - or many...@zhaoxi you can't call JNI functions - or many other functions - in signal handlers. Signal handlers triggered by a SIGPROF can execute asynchronously - that is, at any time in the execution of a program. So, every function you call in a signal handler has to be able to be called no matter what the main program is doing. We call that property "async safety" or "reentrant&Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-78700770899369742282022-08-30T02:26:00.935-07:002022-08-30T02:26:00.935-07:00Hi Jeremy, I'm still waiting for your explanat...Hi Jeremy, I'm still waiting for your explanation of why things like printf and JNI/JVMTI invocations crash VM...<br /><br />I've developed a profiler with asyncGetCallTrace which is triggered by SIGPROF just like yours. But I implemented some JNI function like FindClass and GetStaticMethodID in signal handler. It turns out that the target program crashed once the profiler attached. What Zhaoxihttps://www.blogger.com/profile/03673217838748765165noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-23774192000960172562017-02-15T13:52:54.919-08:002017-02-15T13:52:54.919-08:00Not sure you can do that with JVMTI. One thing to...Not sure you can do that with JVMTI. One thing to do would be to use bytecode rewriting to add exception handlers that store relevant local information. I believe that Quasar does something like that for implementing continuations.Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-93474945714402932017-02-15T04:41:33.237-08:002017-02-15T04:41:33.237-08:00I want to store the values of all the state variab...I want to store the values of all the state variables whenever any exception is generating in a java program. I am current trying to find a way with jvmti. Can you help me out with something ?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-76011876915005044072013-09-23T21:47:52.522-07:002013-09-23T21:47:52.522-07:00@comain:
1) Thanks.
2) Having two arrays and a ...@comain: <br /><br />1) Thanks.<br /><br />2) Having two arrays and a CAS is the right kind of thing. You have to figure out what to do with the first array once you've switched to the second, because you have to decode it synchronously.<br /><br />3) We solved the native method issue internally by adjusting the JVM to return the PCs for native frames. That's not such a great solution Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-11788505959434659932013-09-23T20:02:00.980-07:002013-09-23T20:02:00.980-07:00Jeremy, good job.
Regarding the incremental part, ...Jeremy, good job.<br />Regarding the incremental part, I think we can use two buffers and an atomic indicator. Let the async writer to swap the buffers periodically. Do you think that will work?<br /><br />Also, there is no native stack, do you have any hint on implementing this?comainhttps://www.blogger.com/profile/03282744418949682458noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-48542169645076654032013-09-09T23:15:42.046-07:002013-09-09T23:15:42.046-07:00I recall asking a kernel expert about that at one ...I recall asking a kernel expert about that at one point, but I don't recall what the answer was. It was certainly good enough for us at Google.Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-21798888951807750752013-09-09T22:42:52.577-07:002013-09-09T22:42:52.577-07:00That makes sense. But I was wondering about distri...That makes sense. But I was wondering about distribution across cores on multicore CPUs, with JVM threads potentially biased towards or bound to a specific CPU.<br />Arnohttps://www.blogger.com/profile/10504698344833366308noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-68041361945366492682013-09-09T21:09:05.909-07:002013-09-09T21:09:05.909-07:00@Arno - Whether this works depends on the OS. It ...@Arno - Whether this works depends on the OS. It works just fine on Linux, where the OS sends the signal to whatever happens to be on the processor at the time the signal fires. OS X always sends the signal to the main thread, which is annoying - I have a workaround, but I haven't had a chance to debug it yet.<br />Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-48358687493358637452013-09-09T14:53:46.226-07:002013-09-09T14:53:46.226-07:00Great stuff, thanks for sharing!
One thing is bugg...Great stuff, thanks for sharing!<br><br />One thing is bugging me though: Profiler::Handle seems to sample just the thread that happened to be interrupted by the signal. Am I missing something? Is there some underlying mechanism at the OS level that distributes the signals evenly across the JVM's threads?Arnohttps://www.blogger.com/profile/10504698344833366308noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-13184138034643197482013-09-04T23:12:30.148-07:002013-09-04T23:12:30.148-07:00@Zoltan - I'm not likely to have time to do it...@Zoltan - I'm not likely to have time to do it, so anyone else around here who wants to do it should just let me know.Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-69693547139441352412013-09-04T19:20:22.872-07:002013-09-04T19:20:22.872-07:00if somebody would write some code to summarize the...if somebody would write some code to summarize the stack samples into the protobuffer format: http://code.google.com/p/spf4j/source/browse/trunk/spf4j-core/src/main/proto/proto_sample_nodes.proto, they could be visualized with the spf4j UI: http://code.google.com/p/spf4j/downloads/list... I am throwing this idea out in case somebody has more time than I do...Zoltan Farkashttps://www.blogger.com/profile/13403082247059421847noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-78653639716699852282013-09-01T10:50:57.994-07:002013-09-01T10:50:57.994-07:00Good work!
This might complement the tools that I...Good work! <br />This might complement the tools that I developed for continuous profiling (www.spf4j.org). Serializing the stack samples to protobuf format supported by spf4j is all that needs to be done... <br />I have recently made some progress in visualizing the samples and improved on brendan greg's flame graphs(http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/) see detail in my Zoltan Farkashttp://blogs.zoltran.comnoreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-3479375231737046602013-08-26T13:40:07.598-07:002013-08-26T13:40:07.598-07:00@burtonator: I'm with you all the way up to &q...@burtonator: I'm with you all the way up to "build up a profile on a production application". That sounds like some kind of summarization (into call tree, graph, whatever). My take is that the summarization is where the rich information in the samples themselves is lost. If the goal is to find bottlenecks, the high measurement precision you get with lots of samples is being bought Mike Dunlaveyhttps://www.blogger.com/profile/16657552696382525525noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-58016488874398072602013-08-25T11:22:34.517-07:002013-08-25T11:22:34.517-07:00@burtonator - what you propose is very similar to ...@burtonator - what you propose is very similar to how we use this technology at Google.Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-77579040058271743842013-08-25T10:32:08.814-07:002013-08-25T10:32:08.814-07:00This is really interesting and I might end up divi...This is really interesting and I might end up diving in here and seeing what I can do to improve this.<br /><br />I've wanted this for a LONG time.<br /><br />My thinking is something along the lines of continuous integration and continuous delivery and I'm been cautiously referring to it as continuous profiling.<br /><br />Basically the idea that you asynchronously gather stack traces burtonatorhttps://www.blogger.com/profile/08049781702053733725noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-82887358112829138942013-08-22T18:55:09.426-07:002013-08-22T18:55:09.426-07:00This comment has been removed by a blog administrator.Cesarhttp://bestrowingmachinereviews.us/2013/08/15/how-to-use-a-rower-1/noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-30405357912375629512013-07-26T10:15:30.344-07:002013-07-26T10:15:30.344-07:00I've updated the post with an explanation of w...I've updated the post with an explanation of why things like printf and JNI/JVMTI invocations crash your programs.Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-3864593156908598332013-07-26T09:38:51.498-07:002013-07-26T09:38:51.498-07:00@Romain: can you send me the patch via email? Blo...@Romain: can you send me the patch via email? Blogger thought the the > and < chars in your post were HTML tags, and stripped them out.<br /><br />I'll update the post with more detail on what "asynchronous" implies in terms of what you can invoke and what you can't, since that seems to be a source of confusion.Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-52720992150534387622013-07-26T09:01:15.118-07:002013-07-26T09:01:15.118-07:00Hi Jeremy,
Thanks for sharing the code.
I have be...Hi Jeremy,<br /><br />Thanks for sharing the code.<br />I have been trying to implement a real time version of it, but encountered an issue : calling jvmti->GetMethodName in the handler of the SIGPROF signal ends up triggering a segmentation fault. Any leads on why it is safe to call it on vm death, but not while executing?Anonymoushttps://www.blogger.com/profile/01311785395632708833noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-77965417271093206032013-07-26T07:51:58.484-07:002013-07-26T07:51:58.484-07:00> I've only tested on Linux (Ubuntu Precise...> I've only tested on Linux (Ubuntu Precise). A mac port<br />> may require various patches. In fact, I have no idea if<br />> it will even work!<br />> <br />> Having said that, I've fixed that particular error, so <br />> feel free to try again.<br /><br />Sorry for the late reply - no notification:(<br /><br />Anyway, thanks, I did actually get further on my own to Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-76158707875963064912013-07-26T05:27:52.626-07:002013-07-26T05:27:52.626-07:00Hi,
Thx for this it looks interesting !
I've...Hi, <br /><br />Thx for this it looks interesting !<br />I've compiled it and passed it as a Java option, but it looks like in my case stack traces are only gathered when the VM is shut down. Until the VM shutdown is requested, the stacks.txt file stays empty. Is there an other way to trigger the stack collection ? I've tried with OpenJDK 1.7.0.25 provided as a Fedora RPM.<br /><br />I Romain Ghttps://www.blogger.com/profile/10879424013679277912noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-21673709792111757412013-07-24T12:50:06.104-07:002013-07-24T12:50:06.104-07:00I just posted a comment on your 2010 post. I'm...I just posted a comment on your 2010 post. I'm glad you are doing stack sampling. I've had a lot to say on this: http://scicomp.stackexchange.com/a/2719/1262<br />It isn't that a lot of stack samples are needed. It's that you only need a small number, but the ones you do get need to be examined carefully, not just blindly summarized.Mike Dunlaveyhttps://www.blogger.com/profile/16657552696382525525noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-41647743239527294432013-07-16T13:37:44.133-07:002013-07-16T13:37:44.133-07:00Hi Oleg! Thanks for doing it, because it saved us...Hi Oleg! Thanks for doing it, because it saved us an enormous amount of time and effort. We should talk some time...Jeremy Mansonhttps://www.blogger.com/profile/04241094734813086257noreply@blogger.comtag:blogger.com,1999:blog-8405107760807432973.post-3874224723976389382013-07-16T13:25:40.331-07:002013-07-16T13:25:40.331-07:00Hi Jeremy,
It's good to see somebody has figu...Hi Jeremy,<br /><br />It's good to see somebody has figured AGCT out. I'm the engineer behind its use in Sun Studio Analyzer. Though that is now a thing of the past. One tidbit: having been implemented on top of essential low level mechanisms in HotSpot, AGCT becomes naturally available on every platform HS is ported to.Anonymoushttps://www.blogger.com/profile/07916070952536241199noreply@blogger.com