Upgrade to ECC 6 from 4.6c => the aftermath...so far...

[Florin Wach]

thanks for the update :-)

-------- Original-Nachricht --------
> Datum: Thu, 22 Jan 2009 10:42:40 +0000
> Von: Mike Gambier <madgambler at hotmail.com>
> An: sap-wug at mit.edu
> Betreff: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...

> 
> Morning,
>  
> Progress! :)
>  
> We're implementing SAP note 1122756 to re-introduce the 'old' locking
> behaviour for one of our repeat offender Workflows currently issuing 'H0'
> System Log Messages. This seems to avoid the lock deletion wrist-slapping with
> the DEQUEUE_ALL statement still inside the Method. We'll do the same thing
> for the other dozen Workflows calling methods with the same statement in and
> just take the hit on more enqueue/dequeue calls provided, as soon as the
> evidence is strong enough that this helps.
>  
> SWWERRE performance and memory issues look like they'll be fixed by SAP
> Note 1113338, which I have to admit the Workflow Gods did tell us about a
> while back, but somehow this was missed. 
>  
> I think we're also a little unlucky because some of the tweaks to the
> Workflow service jobs come as standard in Release 14 and we're on 13 at the
> moment :(
>  
> Useless SWWWIRET 'Success' messages at 7.7 million and rising...
>  
> Regards,
>  
> Mike GT
> 
> 
> 
> From: madgambler at hotmail.comTo: sap-wug at mit.eduSubject: RE: Upgrade to ECC
> 6 from 4.6c => the aftermath...so far...Date: Wed, 21 Jan 2009 10:15:11
> +0000
> 
> Hi again, Some more progress and issues to report. The new 'Standard'
> Locking behaviour of our Workflows in ECC 6 is falling foul of any DEQUEUE_ALL
> statements issued inside methods called by Tasks. So far, I might add, no
> SAP code has been found at fault. We're considering reverting to the 'old'
> Locking behaviour for the Workflows that call these Tasks by implementing a
> manual note to enable a few more options in the Locking profile. We've
> discovered that SWEQADM has been 'enhanced' and now prevents users from
> manually re-delivering Events that have found their way on to the 'Linkages with
> Error' tab if the Event Queue job is running at the same time. Sensible I
> suppose, but we run the job every minute at the minute... Our poor old
> SWWERRE job is still having a hard time. I've asked for a limit cap modification
> on it so we can give it a chance to process blocks of Work Items rather
> than trying to process everything, which it tries to do before running out of
> memory. We might consider turning it into a parallel stream tRFC job as
> well as we've done for Deadlines. The useless SWF_RUN 641 'Success' messages
> in SWWWIRET are still sky-rocketing, 5.8 million and counting :( Regards,
> Mike GT
> 
> 
> 
> From: madgambler at hotmail.comTo: sap-wug at mit.eduSubject: RE: Upgrade to ECC
> 6 from 4.6c => the aftermath...so far...Date: Tue, 20 Jan 2009 11:39:37
> +0000
> 
> Hi Eddie, Thanks but we did that already and it helped to squish an IS-U
> statement that was adding to the noise before we upgraded.  This reduced the
> frequency a lot, but we still had a residual amount in testing. In
> Production that residual amount has been magnified into thousands unfortunately. I
> think the main culprit is actually one of our own custom BOR Methods that
> invokes a ROLLBACK WORK statement if a certain condition is met. It's a
> frequent step and is most likely the frequent offender in this case, but we
> can't seem to find a workaround because without the ROLLBACK in place the
> implicit COMMIT WORK in BOR will actually save the database changes we would
> otherwise want to discard. And we can't fully control the LUW in this case
> because we're calling a SAP FM that wants to update the database on every
> call. So I think in our case these H0 messages are actually legitimate
> warnings about locks being lost before Workflow gets around to trying to remove
> them. I might try to fiddle with the locking settings for the Workflow
> trying to invoke this step to see if we can avoid this lock deletion
> wrist-slapping.Regards,Mike GT 
> 
> 
> 
> Subject: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...Date:
> Tue, 20 Jan 2009 12:24:21 +0100From: eddie.morris at sap.comTo:
> sap-wug at mit.edu
> 
> 
> Hi Mike,
>  
> In relation to # 6 please apply note 1125717
>  
> * 1125717 - Enhanced analysis for DEQUEUE_ALL
>  
> With note 1125717 you must create the checkpoint group WF_DEQUEUE_ALL via
> SAAB and then activate it. If there is an issue with the DEQUEUE_ALL
> function a short dump is created and we can see where the issue is.
>  
> You need to goto transaction SAAB, enter WF_DEQUEUE_ALL and click the
> Create button. Fill in the following:
>  
> Checkpoint Group   WF_DEQUEUE_ALLDescription        Assert for
> DEQUEUE_ALLPackage            SWW
>  
> Now activate the assertion 'WF_DEQUEUE_ALL' via the Activate button on the
> original screen.Goto the Server Button.Right mouse click on the line
> 'Global settings' in the column 'Assert'.Select 'Abort'. Confirm. Save this
> configuration.
>  
> Once this is done a shortdump will be created wit the location for the
> dequeue_all function. 
>  
> It is most likely being called from application code and will have to be
> looked at by the application area responsible in order to remove the call to
> DEQUEUE_ALL.
>  
> Regards,
> Eddie
> 
> 
> 
> From: sap-wug-bounces at mit.edu [mailto:sap-wug-bounces at mit.edu] On Behalf
> Of Mike GambierSent: 20 January 20, 2009 11:07To: sap-wug at mit.eduSubject:
> RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...
> Hi, More creaks and groans to report from our ECC 6 upgrade I'm afraid.
> The current issues are: #1 Massive growth of WF-BATCH enqueues in SM12 -
> briefly these hit 50k+ last night. With the 'new' locking profile that came
> with ECC 6 as the 'Standard' option on our WF definitions we expected a
> saw-tooth pattern to locks and hoped this would be a bit more efficient than the
> lock/unlock frenzy we had in 4.6C. So a spike in numbers was to be expected
> too, but this amount is just crazy. If we can't cope with this we might
> have to switch back to the 'Background' locking approach. #2 SWWERRE job is
> having seriously difficulty clearing the backlog of temporary exceptions.
> Longest runtime so far has been 26,000 seconds before it was killed and
> restarted :( #3 Notable increase in the number of events not being delivered
> first time, i.e. appearing on 'linkages with errors' tab in SWEQADM. #4
> Unicode runtime errors with standard SAP code have started to crawl out of the
> woodwork #5 SWWWIRET table growth is still causing a lot of concern. Total
> number of new entries added since we upgraded = 5.4 million. Of these, the
> total number of useless 'Success' messages = 3.1 million, a staggering 57%.
> #6 Thousands of H0 Messages (Lock Deletion failed) appearing in SM21 -
> despite numerous OSS Notes applied to prevent them. Guess we have work to do
> still... Regards, Mike GT
> 
> From: madgambler at hotmail.comTo: sap-wug at mit.eduSubject: RE: Upgrade to ECC
> 6 from 4.6c => the aftermath...so far...Date: Mon, 19 Jan 2009 15:50:42
> +0000
> 
> Hi John, As a precaution I had already arranged for our instance linkages
> to be backed up in a Transport and reapplied as part of our Build process
> following the upgrade, as we spotted that exact same problem early on. But
> thanks for reminding me :) We always use 'Do Not Change Linkage' anyway
> because of our volumes and the risk of massive penalties if we don't progress
> some Workflows immediately. Oh, and another thing, I'm asking SAP to turn
> off are these blasted SWF_RUN 641 messages (Method &1->&2 executed
> successfully) being posted in SWWWIRET every time a step actually works. We've had
> nearly 1.5 million of these things already and they serve no use whatsoever!
> Mike GT
> 
> Subject: RE: Upgrade to ECC 6 from 4.6c => the aftermath...so far...Date:
> Mon, 19 Jan 2009 09:37:03 -0600From: Scheinoha.John at basco.comTo:
> SAP-WUG at mit.eduCC: coyle.pat at basco.com
> 
> 
> Mike,
>  
>    When we upgraded from 4.6C to ECC 5.0, we noticed the upgrade changed
> our Receiver Err Feedback value from "3 Do not Change Linkage" to "1
> Deactivation of Linkage".  We did not notice this change until a workflow error
> occurred, and subsequent workflow instances errored after that.  It happened
> to our Purchase Requisition application and plenty of workflow errors
> occurred in a very short period of time.  
>  
>    You might want to review SWEQADM > Basic Data tab > Receiver Error
> Feedback value.
>  
>    The SAP Inbox feature of "Double-clicking on an object in the same
> window is also fully functional in ECC 5.0 and I assume ECC 6.  This checkbox
> needs to be checked in the Personal workflow settings as a default value in
> order to view SAP workflow objects.
>  
> Hope this helps and thanks for the ECC 6 update.  We'll be upgrading to
> ECC 6 later this year.
>  
> Thanks,
> John Scheinoha
> Briggs & Stratton 
> (414) 256 - 5136
> 
> 
> From: sap-wug-bounces at mit.edu [mailto:sap-wug-bounces at mit.edu] On Behalf
> Of Mike GambierSent: Monday, January 19, 2009 4:39 AMTo:
> sap-wug at mit.eduSubject: Upgrade to ECC 6 from 4.6c => the aftermath...so far...
> Hi, Well it happened...we upgraded over the weekend...and things seem to
> be creaking a little, but so far so good. :) We had a problem with SAP's
> XPRA for Event Linkages that needed to be tackled on the day by Oracle DBAs.
> We couldn't use SAP's single-stream version of the code because we had 8
> million table entries to move from SWEINSTCOU to SWFDEVINST so we converted
> their program to a mutli-stream job to run in parallel. But on the day our
> Basis people decided to import table statsitics from a Pre-Prod system which
> skewed the SQL to run with very poorly optimised indexes. Once we spotted
> why the job was taking so long and producing so little the stats were
> updated on both the old and the new tables again and the app servers bounced and
> the job re-run again. It finished in 20 minutes instead of the 6 hours it
> looked like it might take, yay! :) Then, a few hours later with the system
> up, a quick scan of SWWWIDH entries with a Status of '02' showed up some
> issues we weren't aware of. We had about 4k deadlines that stubbornly refused
> to fire properly. They kept coming back even after using the new
> Transaction SWF_ADM_SWWWIDH to reset them. After a bit of sleuthing it turned out
> that they were related to old instances of Workflows (mostly the same template
> which was handy) that now were suffering from syntax errors because their
> container definition had become invalid during the upgrade. ECC 6 has
> introduced a new vulnerability here that we might have to have squished by an
> OSS Note.  In our case a few versions of a common Workflow had an element
> that pointed to a SAP-defined structure that had been deleted as part of the
> upgrade. Luckily it was obvious that the developer had simply wanted to find
> a simple BOOLE-BOOLE flag but had unwisely chosen something else instead. 
> In this situation SAP's new code detects that it is dealing with an old WF
> instance and switches to a 'Persistence' version of its Container ABAP
> Class inside CL_SWF_CNT_FACTORY, namely CL_SWF_CNT_WF_PERSISTENCE. This Class
> duly calls old FMs (SWD_GET_WF_CONTAINER_TABLE) to fetch the Container
> Definition for the version of the instance in question which end up selecting
> from old tables SWDSWFCONT and SWD_WFCONT. And it's here that the definition
> of our element was suddenly found to be pointing to a bogus structure and
> field when the element was instantiated at runtime. But the new code can't
> cope with this syntax error very well and simply bombs out with an
> Exception that is eventually caught in the stack:   IF NOT por-bus_key IS INITIAL. 
>   l_retcode = l_pmanager->load( ).                  <== Return Code comes
> back here...    CASE l_retcode.      WHEN 0.      WHEN 95.                 
>                                          <== Exception bombs out here...  
>      lr_cx_cnt = instance->get_last_exception( ).        RAISE EXCEPTION
> TYPE cx_swf_utl_obj_create_failed          EXPORTING            t100_msg =
> lr_cx_cnt->t100_msg            previous = lr_cx_cnt. The upshot of this was
> that we couldn't even display these Deadline steps in SWI1, let alone reset
> their deadlines, because of a nasty error crawling out.  In the end I had
> to resort to surgery and update the old 4.6c tables directly before we
> could move these on :) We have some SAP guys here on site so I'll nag them to
> have this situation looked at a bit more. I have a feeling though that
> they've missed an XPRA or something here because I can imagine other clients
> suffering from just this sort of thing were SAP have zapped their own DDIC
> structures and Worklow defintions that were perfectly legitimate in 4.6c
> suddenly become invalid in ECC 6 and behave erratically like this. Regards, Mike
> GT
> 
> Are you a PC? Upload your PC story and show the world
> 
> Are you a PC? Upload your PC story and show the world
> 
> Choose the perfect PC or mobile phone for you. Click here
> 
> What can you do with the new Windows Live? Find out
> 
> Choose the perfect PC or mobile phone for you. Click here
> _________________________________________________________________
> Choose the perfect PC or mobile phone for you
> http://clk.atdmt.com/UKM/go/130777504/direct/01/