Upgrade to ECC 6 from 4.6c => the aftermath...so far...

Wed Jan 21 05:15:11 EST 2009

Hi again,

Some more progress and issues to report.

The new 'Standard' Locking behaviour of our Workflows in ECC 6 is falling foul of any DEQUEUE_ALL statements issued inside methods called by Tasks. So far, I might add, no SAP code has been found at fault. We're considering reverting to the 'old' Locking behaviour for the Workflows that call these Tasks by implementing a manual note to enable a few more options in the Locking profile.

We've discovered that SWEQADM has been 'enhanced' and now prevents users from manually re-delivering Events that have found their way on to the 'Linkages with Error' tab if the Event Queue job is running at the same time. Sensible I suppose, but we run the job every minute at the minute...

Our poor old SWWERRE job is still having a hard time. I've asked for a limit cap modification on it so we can give it a chance to process blocks of Work Items rather than trying to process everything, which it tries to do before running out of memory. We might consider turning it into a parallel stream tRFC job as well as we've done for Deadlines.

The useless SWF_RUN 641 'Success' messages in SWWWIRET are still sky-rocketing, 5.8 million and counting :(

Regards,

Mike GT

From: madgambler at hotmail.comTo: sap-wug at mit.eduSubject: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...Date: Tue, 20 Jan 2009 11:39:37 +0000

Hi Eddie, Thanks but we did that already and it helped to squish an IS-U statement that was adding to the noise before we upgraded.  This reduced the frequency a lot, but we still had a residual amount in testing. In Production that residual amount has been magnified into thousands unfortunately. I think the main culprit is actually one of our own custom BOR Methods that invokes a ROLLBACK WORK statement if a certain condition is met. It's a frequent step and is most likely the frequent offender in this case, but we can't seem to find a workaround because without the ROLLBACK in place the implicit COMMIT WORK in BOR will actually save the database changes we would otherwise want to discard. And we can't fully control the LUW in this case because we're calling a SAP FM that wants to update the database on every call. So I think in our case these H0 messages are actually legitimate warnings about locks being lost before Workflow gets around to trying to remove them. I might try to fiddle with the locking settings for the Workflow trying to invoke this step to see if we can avoid this lock deletion wrist-slapping.Regards,Mike GT 

Subject: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...Date: Tue, 20 Jan 2009 12:24:21 +0100From: eddie.morris at sap.comTo: sap-wug at mit.edu

Hi Mike,

In relation to # 6 please apply note 1125717

* 1125717 - Enhanced analysis for DEQUEUE_ALL

With note 1125717 you must create the checkpoint group WF_DEQUEUE_ALL via SAAB and then activate it. If there is an issue with the DEQUEUE_ALL function a short dump is created and we can see where the issue is.

You need to goto transaction SAAB, enter WF_DEQUEUE_ALL and click the Create button. Fill in the following:

Checkpoint Group   WF_DEQUEUE_ALLDescription        Assert for DEQUEUE_ALLPackage            SWW

Now activate the assertion 'WF_DEQUEUE_ALL' via the Activate button on the original screen.Goto the Server Button.Right mouse click on the line 'Global settings' in the column 'Assert'.Select 'Abort'. Confirm. Save this configuration.

Once this is done a shortdump will be created wit the location for the dequeue_all function. 

It is most likely being called from application code and will have to be looked at by the application area responsible in order to remove the call to DEQUEUE_ALL.

Regards,
Eddie

From: sap-wug-bounces at mit.edu [mailto:sap-wug-bounces at mit.edu] On Behalf Of Mike GambierSent: 20 January 20, 2009 11:07To: sap-wug at mit.eduSubject: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...
Hi, More creaks and groans to report from our ECC 6 upgrade I'm afraid. The current issues are: #1 Massive growth of WF-BATCH enqueues in SM12 - briefly these hit 50k+ last night. With the 'new' locking profile that came with ECC 6 as the 'Standard' option on our WF definitions we expected a saw-tooth pattern to locks and hoped this would be a bit more efficient than the lock/unlock frenzy we had in 4.6C. So a spike in numbers was to be expected too, but this amount is just crazy. If we can't cope with this we might have to switch back to the 'Background' locking approach. #2 SWWERRE job is having seriously difficulty clearing the backlog of temporary exceptions. Longest runtime so far has been 26,000 seconds before it was killed and restarted :( #3 Notable increase in the number of events not being delivered first time, i.e. appearing on 'linkages with errors' tab in SWEQADM. #4 Unicode runtime errors with standard SAP code have started to crawl out of the woodwork #5 SWWWIRET table growth is still causing a lot of concern. Total number of new entries added since we upgraded = 5.4 million. Of these, the total number of useless 'Success' messages = 3.1 million, a staggering 57%. #6 Thousands of H0 Messages (Lock Deletion failed) appearing in SM21 - despite numerous OSS Notes applied to prevent them. Guess we have work to do still... Regards, Mike GT

From: madgambler at hotmail.comTo: sap-wug at mit.eduSubject: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...Date: Mon, 19 Jan 2009 15:50:42 +0000

Hi John, As a precaution I had already arranged for our instance linkages to be backed up in a Transport and reapplied as part of our Build process following the upgrade, as we spotted that exact same problem early on. But thanks for reminding me :) We always use 'Do Not Change Linkage' anyway because of our volumes and the risk of massive penalties if we don't progress some Workflows immediately. Oh, and another thing, I'm asking SAP to turn off are these blasted SWF_RUN 641 messages (Method &1->&2 executed successfully) being posted in SWWWIRET every time a step actually works. We've had nearly 1.5 million of these things already and they serve no use whatsoever! Mike GT

Subject: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...Date: Mon, 19 Jan 2009 09:37:03 -0600From: Scheinoha.John at basco.comTo: SAP-WUG at mit.eduCC: coyle.pat at basco.com

Mike,

   When we upgraded from 4.6C to ECC 5.0, we noticed the upgrade changed our Receiver Err Feedback value from "3 Do not Change Linkage" to "1 Deactivation of Linkage".  We did not notice this change until a workflow error occurred, and subsequent workflow instances errored after that.  It happened to our Purchase Requisition application and plenty of workflow errors occurred in a very short period of time.  

   You might want to review SWEQADM > Basic Data tab > Receiver Error Feedback value.

   The SAP Inbox feature of "Double-clicking on an object in the same window is also fully functional in ECC 5.0 and I assume ECC 6.  This checkbox needs to be checked in the Personal workflow settings as a default value in order to view SAP workflow objects.

Hope this helps and thanks for the ECC 6 update.  We'll be upgrading to ECC 6 later this year.

Thanks,
John Scheinoha
Briggs & Stratton 
(414) 256 - 5136

From: sap-wug-bounces at mit.edu [mailto:sap-wug-bounces at mit.edu] On Behalf Of Mike GambierSent: Monday, January 19, 2009 4:39 AMTo: sap-wug at mit.eduSubject: Upgrade to ECC 6 from 4.6c => the aftermath...so far...
Hi, Well it happened...we upgraded over the weekend...and things seem to be creaking a little, but so far so good. :) We had a problem with SAP's XPRA for Event Linkages that needed to be tackled on the day by Oracle DBAs. We couldn't use SAP's single-stream version of the code because we had 8 million table entries to move from SWEINSTCOU to SWFDEVINST so we converted their program to a mutli-stream job to run in parallel. But on the day our Basis people decided to import table statsitics from a Pre-Prod system which skewed the SQL to run with very poorly optimised indexes. Once we spotted why the job was taking so long and producing so little the stats were updated on both the old and the new tables again and the app servers bounced and the job re-run again. It finished in 20 minutes instead of the 6 hours it looked like it might take, yay! :) Then, a few hours later with the system up, a quick scan of SWWWIDH entries with a Status of '02' showed up some issues we weren't aware of. We had about 4k deadlines that stubbornly refused to fire properly. They kept coming back even after using the new Transaction SWF_ADM_SWWWIDH to reset them. After a bit of sleuthing it turned out that they were related to old instances of Workflows (mostly the same template which was handy) that now were suffering from syntax errors because their container definition had become invalid during the upgrade. ECC 6 has introduced a new vulnerability here that we might have to have squished by an OSS Note.  In our case a few versions of a common Workflow had an element that pointed to a SAP-defined structure that had been deleted as part of the upgrade. Luckily it was obvious that the developer had simply wanted to find a simple BOOLE-BOOLE flag but had unwisely chosen something else instead.  In this situation SAP's new code detects that it is dealing with an old WF instance and switches to a 'Persistence' version of its Container ABAP Class inside CL_SWF_CNT_FACTORY, namely CL_SWF_CNT_WF_PERSISTENCE. This Class duly calls old FMs (SWD_GET_WF_CONTAINER_TABLE) to fetch the Container Definition for the version of the instance in question which end up selecting from old tables SWDSWFCONT and SWD_WFCONT. And it's here that the definition of our element was suddenly found to be pointing to a bogus structure and field when the element was instantiated at runtime. But the new code can't cope with this syntax error very well and simply bombs out with an Exception that is eventually caught in the stack:   IF NOT por-bus_key IS INITIAL.    l_retcode = l_pmanager->load( ).                  <== Return Code comes back here...    CASE l_retcode.      WHEN 0.      WHEN 95.                                                           <== Exception bombs out here...        lr_cx_cnt = instance->get_last_exception( ).        RAISE EXCEPTION TYPE cx_swf_utl_obj_create_failed          EXPORTING            t100_msg = lr_cx_cnt->t100_msg            previous = lr_cx_cnt. The upshot of this was that we couldn't even display these Deadline steps in SWI1, let alone reset their deadlines, because of a nasty error crawling out.  In the end I had to resort to surgery and update the old 4.6c tables directly before we could move these on :) We have some SAP guys here on site so I'll nag them to have this situation looked at a bit more. I have a feeling though that they've missed an XPRA or something here because I can imagine other clients suffering from just this sort of thing were SAP have zapped their own DDIC structures and Worklow defintions that were perfectly legitimate in 4.6c suddenly become invalid in ECC 6 and behave erratically like this. Regards, Mike GT

Are you a PC? Upload your PC story and show the world

Choose the perfect PC or mobile phone for you. Click here

What can you do with the new Windows Live? Find out
_________________________________________________________________
Imagine a life without walls.  See the possibilities
http://clk.atdmt.com/UKM/go/122465943/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/sap-wug/attachments/20090121/5461d914/attachment.htm