I ask because we just did that, and none of the .class files in the directory I mentioned were recreated. Earlier today, when we tested a zmmailboxdctl stop/start. it seems some of the class files were recreated. But again, not the displayMessage_tag.class file which as you can see from the directory listing below is still the file we restored from backup earlier today.
We opened a support case earlier today on this BTW, so I'm not trying to "double-dip" as it were. But if you do have some info that could help Jason Bryan, I'm sure the both of us would be appreciative.
drwxr-x--- 2 zimbra zimbra 4096 Apr 14 15:29 .
drwxr-x--- 11 zimbra zimbra 4096 Apr 14 15:29 ..
-rw-r----- 1 zimbra zimbra 20870 Apr 14 14:12 attachments_tag.class
-rw-r----- 1 zimbra zimbra 11418 Apr 14 14:12 body_tag.class
-rw-r----- 1 zimbra zimbra 63704 Apr 6 21:38 displayMessage_tag.class
-rw-r----- 1 zimbra zimbra 22391 Apr 14 14:12 messageAction_tag$messageAction_tagHelper.class
-rw-r----- 1 zimbra zimbra 49064 Apr 14 14:12 messageAction_tag.class
-rw-r----- 1 zimbra zimbra 7700 Apr 14 14:12 messageIframe_tag.class
-rw-r----- 1 zimbra zimbra 2485 Apr 14 15:29 messageListViewToolbar_tag$messageListViewToolbar_ tagHelper.class
-rw-r----- 1 zimbra zimbra 34692 Apr 14 15:29 messageListViewToolbar_tag.class
-rw-r----- 1 zimbra zimbra 27122 Apr 14 15:29 messageListView_tag$messageListView_tagHelper.clas s
-rw-r----- 1 zimbra zimbra 42421 Apr 14 15:29 messageListView_tag.class
-rw-r----- 1 zimbra zimbra 2433 Apr 14 15:29 messageViewToolbar_tag$messageViewToolbar_tagHelpe r.class
-rw-r----- 1 zimbra zimbra 27670 Apr 14 15:29 messageViewToolbar_tag.class
-rw-r----- 1 zimbra zimbra 14155 Apr 14 15:29 messageView_tag$messageView_tagHelper.class
-rw-r----- 1 zimbra zimbra 23435 Apr 14 15:29 messageView_tag.class
I just did the same thing to myself, wiping the jetty/work/ cache... fortunately on a 5.0.5 test system, not our production 5.0.4. So this is a learning opportunity.
What should I do?
I notice that /zimbra/h/calendar, /zimbra/m/mocalendar, /zimbra/m/main, and /zimbra/m/main?action=compose work. /zimbra/m/mosearch, /zimbra/h/search, and /zimbra/h/search?action=compose are broken.
Code:2008-04-23 12:11:41,547 WARN [btpool0-11]  log - /zimbra/h/search: java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/message/displayMessage_tag.class java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/message/displayMessage_tag.class 2008-04-23 12:13:16,144 WARN [btpool0-13]  log - /zimbra/h/search: java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/message/displayMessage_tag.class java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/mobile/moDisplayMessage_tag.class 2008-04-23 12:15:11,851 WARN [btpool0-9]  log - /zimbra/m/mosearch: java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/mobile/moDisplayMessage_tag.class
Last edited by Rich Graves; 04-23-2008 at 10:19 AM.
Damnit. Now it works.
I had restarted mailboxd twice and verified it still broken. Then I turned on zmprov aal for everything but POP and IMAP (which one is relevant? soap? misc?), shut mailboxd down again, emptied mailbox.log to reduce the noise and disk space, and restarted, at which point it worked with nothing of interest logged. If there was anything, it's in the mailbox.log that I deleted. :-(
So unless there's something interesting in the jetty logs, I got nothing. Oh well. Mark, did the support case you opened conclude happily?
We upgraded to 5.0.5 as a "fix".
We ran out of time during the maintenance window to test deleting the jetty/work directory contents and see if they get recreated correctly when restarting the mailbox. We'll try that this weekend and then update here and our support ticket accordingly.
Hope your "fix" sticks as well!
All the best,
I can reproduce the problem with 5.0.5. Stop mailboxd, rm -rf /opt/zimbra/jetty/work/*, start mailboxd, and get random errors in /h/ and /m/.
Wait sometimes as long as 15 minutes, and the errors go away.
I guess the server has better things to do at startup than recompile JSPs.
Can you do an ls > rightafterrestart.txt on the jetty/work directory, and then when the errors go away, do another ls > allisOKnow.txt and do a diff on the two text files?
Would be nice to know if the "lazy loading" feature in ZCS 5 applies to recreating the jar files as well...
All the best,
P.S. If you are running 5.0.5 now shouldn't you update your profile?
Production 5.0.4, testing 5.0.5. I wouldn't be offering to restart production so willy-nilly.P.S. If you are running 5.0.5 now shouldn't you update your profile?
Can you guys open tickets? We really need to figure out why it's happening on like 2% of systems. Very odd.
Usually, wiping the work dir can be a good thing to ensure that you're getting the latest version of the files.
When the work dir is wiped (and as long as perms are correct), the jsp's simply recompile. In the cases on this thread, they're not.
IF you get a chance, please do a diff on your /opt/zimbra/jetty/webapps and a off the shelf webapps dir. I'm getting the feeling that one of our apps/tags aren't getting updated.
We already have a support ticket open and were asked to upgrade to 5.0.5 before proceeding further.
We'll test this weekend during our maintenance window:
and report back what happens.Code:mkdir /opt/zimbra/jetty/work-original rsync -avz /opt/zimbra/jetty/work/ /opt/zimbra/jetty/work-original/ zmcontrol stop rm -rf /opt/zimbra/jetty/work/* zmcontrol start
If we get errors, I'll do a diff on the contents of ~/jetty/work and ~/jetty/work-original.
Anything else I should be doing here?