Stop the mess in your IDoc processing
Are the IDocs in your system piling up into an ever-increasing chaos? Is your database constantly growing and is it taking longer and longer to process incoming IDocs or to list them in BD87 and similar transactions?
Read here how to clear up this mess.
Recently I saw it again. The IDoc tables... what a mess!
- Millions of IDocs that could be archived - still remaining in the system.
- Lots of IDocs, queued into processing again and again, although they were created years ago.
In the observed SAP system, the IDoc tables occupied ~700 GB in the database (without the indexes). Of course, these ~700 GB were copied to the quality system and another test system as part of a system refresh, so we are faced with a multiplied wastage of space. And as you can imagine, this is not only a waste of storage space, but also of time and money.
But more than this waste of resources and money, I think that constantly processing - or attempting to process - years old IDocs is unnecessary and dangerous. The hit list of these IDocs was led by an Idoc with ~230000 status records. It was created ~8.5 years ago, and from that point in time up to 7.5 years the system tried to process this Idoc again and again. Approx. 1 year ago, someone took pity on the IDoc and set it to status '68' so that no further processing attempts were made.
How much CPU capacity was wasted on this ancient Idoc during this time?
This IDoc topped the "hit list", but it was not an isolated case. Second place went to an IDoc with 170000 status records. And in total there were more than 200 IDocs with more than 10,000 status records. What a waste of processing time and database space!
It is clear, that processing of these old IDocs does not make sense. Ok, so what is the reason for this waste of space, time and money?
In a nutshell, I would say it is just a lack of care. The IDocs seem to me like unloved dirty children within the data categories. Unlike master data for example, where a lot of value and effort is put into correct data maintenance, there is no such awareness with IDocs. So, what went wrong here?
1.: It seems there is no adequate monitoring for the processing of IDocs.
IDocs are a bit of a no-man's land. Since they are used in interfaces, their actual nature is technical. But they mostly contain application data. So who should take care of them - IT support or the user department? If in doubt, then... often no one does.
2.: For the same reason, unfortunately, housekeeping often does not take place.
From my point of view housekeeping for IDocs should contain at least these activities:
From my point of view housekeeping for IDocs should contain at least these activities:
- Set old/obsolete Inbound IDocs to status '68' for avoiding further processing attempts and enabling archiving for these IDocs.The 'quality' of an Idoc deteriorates with age. Should an IDoc that was created 2 weeks ago actually still be processed? Or is its content now outdated? Has the IDoc been replaced by a newer one so that its processing would cause a double booking?
- Archiving!
"IDoc" stands for "Intermediate Document". It is 'only' a container for data to be processed. Once the data is processed, the IDoc usually is not important anymore and can be archived (and deleted in the system afterwards). For this it is important that an IDoc has an appropriate status that allows archiving (according to table STACUST). For most of the Inbound IDocs this should be status '53' (Application document posted), for faulty IDocs status '68'. Notice: status '68' will not be set automatically by the SAP system.
But with only few activities you can achieve a really big improvement for the situation regarding IDocs:
1.: Assign responsibility for IDocs
This could be the most difficult part. The responsibility could take place in the user or in the IT department, depending on the IDoc message type. However, although it might be difficult to find someone who takes over the responsibility, there should be someone for each IDoc type! And the responsibles should be supported with updates about failed and not processed IDocs, e.g. via scheduling RSEBWE02 with suitable selection criteria and mailing the list. Of course a daily check of BD87, WE02 or others is also an option. Anyway, incorrect IDocs should be investigated. It may be that there are missing settings or table entries that prevent processing - this can often be fixed quickly. Or the IDocs contain incorrect data - in this case the sender should be informed and asked to correct the data.
2.: Take old IDocs out of the race
The case mentioned above, in which the system was forced to process an IDoc that was several years old over and over again, is extreme (but true!). Set an age limit after which IDocs are no longer processed. Of course, this limit can also be set in the variants for the SAP standard programs RBDAPP01 and RBDMANI2, but naturally there are many variants that you may not have ongoing control over.
I think 2 - 4 weeks is a reasonable age limit.
Taking old Inbound IDocs out of the race can be done via setting the status to '68' (Error - no further processing). Possibly in your release there is an SAP report RC1_IDOC_SET_STATUS available that can do this, but unfortunately each and every Idoc number has to be specified. So for an automatic solution this report, although it might be available, can not be used.
But it is not necessary to invest time and effort in the development of an ABAP program.
With "Shortcut for SAP systems" it can be done easily by using the "Update table data" function:
First we have to select the connection to the SAP system (1) and the client (2), then we specify the update: the table containing the current IDoc status is "EDIDC" (3), and the field "STATUS" is to be updated with '68' (Error - no further processing) (4).
We restrict the update (5) to the Inbound IDocs (DIRECT = '2') that were created a month ago or earlier (CREDAT LT @( sy-datum - 30 )) and which have a non-archivable status (STATUS in ( select distinct STATUS from STACUST where ARCHFL = ' ' )). Another option would be to select the IDocs with STATUS in ('64','66') ('64' = IDoc ready to be passed to application, '66' = IDoc is waiting for predecessor IDoc (serialization)) - this would be the IDocs selected by SAP standard program for Inbound IDoc processing RBDAPP01 - or STATUS in ('51','64','66') - this would also include the IDocs selected by SAP standard program RBDMANI2 (Repeat processing of IDocs with error status).
Think about it and find an agreement - preferably with the user department or whoever is responsible for the Idocs. A further criteria is the message type, so you can also use different selection criteria depending on the message type.
Once you have finished this, we should think about an automatism - finally we do not want to do this manually again and again.
For this you can either create a variant for the "Update table data" function
or use an XML file:
Personally I prefer using an XML file. With an XML file it is not necessary to create a variant for the "Update table data" function - all necessary information is given in the XML file and it is quite easy to use the XML file also for another system or just extend the XML file for doing the same task in other systems.
Running the "Shortcut for SAP systems" command line tool with the XML file would be done like this:
Here we see a warning ("Sc4SAP command line tool finished with RC 4"). This is because with this execution no IDocs were updated - for today it was already done. But the next days - assuming we turn the execution of the command line tool into automation - IDocs for update will be found and with status '68' there will be no further attempts for processing of IDocs older than 30 days anymore. As the command line tool works on OS level, you can easily add this one-liner into an automatism (simplest way to do this would be the Windows Task Scheduler, but often there are more sophisticated automation tools in use that can do this simple job).
You can also add a list of program RSEIDOC2 in front of the deletion for getting some information about the IDocs to be updated. Just create a variant for RSEIDOC2 with the same selection criteria as you use for the deletion and run the report in front of the deletion. This can be done in an additional task for executing a program in the XML file in front of the update task or via job scheduling in the SAP system.
3.: Implement IDoc archiving
Also this should be done on a regularly basis to avoid the IDoc tables from growing endlessly. Keep in mind that likely the system will be used for system refreshes of your Quality and/or test systems - copying Gigabytes of outdated and archivable Idocs does not make sense, it will cause costs for hardware and it will unnecessarily consume time in the system refresh process.
Find here the SAP documentation about IDoc archiving.
4.: Start now - do it!
Although the 1st task may not be completed yet, the 2nd and 3rd task can be implemented! Every day you postpone this will result in larger Idoc tables and unnecessary space consumption in the database. You will probably never get rid of this! Since there is constant IDoc traffic in the system, reorganizing the IDoc tables resp. the tablespaces to shrink the used space in the database becomes difficult or even impossible.
Once implemented, you will be rewarded with the end of permanent growth in the IDoc tables, which will benefit your SAP system and thus your users and the entire company!
Es gibt noch keine Rezension.