Assume that a message arrives on a queue. It needs to be transformed and processed somehow; then another message should be written to an output queue, a database should be updated and a long-running asynchronous process spawned. If something goes wrong the message should be rolled back to the input queue where it can be retried (or eventually moved to dead letters by the queue manager) and all the other changes should be rolled back as well.

This was possible to do in 10.1.3.3, but it required some black magic. The WSDL file for the input adapter had to be modified and all the BPEL processes had to run in-memory with completionPersistLevel=off and with transaction=participate everywhere. It worked, but the instances would never appear in the BPEL console, making it hard to see what was going on in the system.

With 11g this works much better. It is no longer necessary to use in-memory optmization and the full audit trail can be saved and viewed in the console. The Mediator can be used as well. However, the procedure is still poorly documented and it still requires manual changes in the generated code.

To try it out, create two queues (one inbound and one outbound) and a test table. Create a new composite. Add an inbound AQ or JMS adapter and an outbound AQ or JMS adapter. Add a database adapter for the test table. Create an asynchronous BPEL process that does nothing. Create a synchronous BPEL process that inserts a record into the table, calls the asynhrounous BPEL process and finally enqueues a message into the output queue. Finally create another BPEL process or Mediator component that dequeues a message from the input queue and invokes the synchronous BPEL process.

Enqueue a message on the input queue. The entire flow should appear in the console, a new record should be present in the table and a message should appear in the output queue. So far so good. Now, disable enqueue for the output queue and try again. What happens? Without a global transaction the message will be removed from the input queue, a new record will appear in the table, the asynchronous process will run but nothing appears on the output queue!

Right, do the obvious thing. Edit composite.xml and add bpel.config.transaction=required for all the BPEL processes. Assuming that the application server has been configured correctly with an XA data source this should do the trick, right? Wrong. The changes are rolled back, so there will be no new record in the table and no asynchronous process. However, the message is still removed from the input queue and lost.

Time for some undocumented magic. Edit the WSDL file for the input adapter and add a new type, a new element and a new message. Include the new message as output message, making the WSDL synchronous. For example (assuming an opaque JMS adapter and showing only the relevant parts):

Edit the BPEL process or Mediator component that dequeues the message and add a reply to the adapter. Good to go? Almost. Everything looks good, but for some reason the message will still be lost. As a final step the input adapter must be configured with jca.retry.all=true in composite.xml. With that option in place, deploy the code and try again. This time the message should be returned to the input queue and retried and eventually moved to dead letters, where it can be handled.

In summary:

1) Configure all BPEL processes with bpel.config.transaction=required in composite.xml.
2) Modify the generated WSDL file for the AQ or JMS adapter and add an output message, making it synchronous.
3) Configure the AQ or JMS adapter with jca.retry.all=true in composite.xml.
4) Use an XA data source.

This has been tested with 11.1.1.5.

The compiler will fail with a message similar to “Load of wsdl … failed”. The SOA log (scac.log) may contain:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

In this case the self-signed certificate must be added to the keystore for the JRE used by jDeveloper. Access the server with a browser and open the certificate. Export it as a file. DER format works. Close jDeveloper. Import the certificate:

set JAVA_HOME=c:\oracle\Middleware\jdk160_24
set PATH=%JAVA_HOME%\bin;%PATH%
keytool -import -trustcacerts -file certificate.der -keystore %JAVA_HOME%\jre\lib\security\cacerts

The default password is “changeit”. Start jDeveloper and try again. This time the certificate should be accepted, but there may still be problems. The SOA log (scac.log) may contain:

java.security.cert.CertificateException: No name matching hostname.domain.name found

In this case the SSL certificate uses the wrong CN. There are no options for ignoring hostname verification in JDeveloper (docs), so the names must match. Fix the certificate, there is no workaround.

follow symlinks=yes
wide links=yes
unix extensions=off

They will then be added to the global section of smb.conf. Select “Save and Restart” and the links should work again.

The installation process is a bit involved as the embedded version of FreeNAS keeps the root file system in RAM. A standard installation will simply disappear after the first reboot.

Create a group and a user named svn through the web interface. Note that if you want to su to svn later on, a real shell is required, nologin will not work. Login to the Unix prompt and su to root. Create a directory on one of the mounted disks (not the root file system, which is a RAM disk!):

mkdir -p /mnt/data/apps/Subversion

Install the subversion package to the proper location:

setenv PKG_TMPDIR /mnt/data/apps/Subversion
pkg_add -r subversion -P /mnt/data/apps/Subversion

Include the dynamic libraries in the search path:

ldconfig -Rm /mnt/data/apps/Subversion/lib

Create a repository. Again the location must be on a mounted disk:

/mnt/data/apps/Subversion/bin/svnadmin create /mnt/data/apps/Subversion/svnrep

Configure the repository.

vi /mnt/data/apps/Subversion/svnrep/conf/svnserve.conf

Apart from comments this is an example:

[general]
anon-access = none
auth-access = write
password-db = passwd
realm = FreeNAS

Edit the password file (vi /mnt/data/apps/Subversion/svnrep/conf/passwd) and add users:

[users]
userid1 = password1
userid2 = password2

Create a start script for the Subversion daemon:

vi /mnt/data/apps/Subversion/bin/start_svnserve.sh

The start script needs to include the Subversion libraries before it launches svnserve:

#!/bin/bash
ldconfig -Rm /mnt/data/apps/Subversion/lib
su svn -c '/mnt/data/apps/Subversion/bin/svnserve -d
  --listen-host=n.n.n.n -r /mnt/data/apps/Subversion/svnrep'

The svnserve command should use a single line, wrapped for readability. Without the listen-host option Subversion may use an IPv6 address. Use the option with your IPv4 address to get around that. Make the script executable:

chmod +x /mnt/data/apps/Subversion/bin/start_svnserve.sh

Change ownership of all files properly:

cd /mnt/data/apps/Subversion
chown -R svn:svn svnrep

Test the script. If it works, add it as a PostInit start script (System/Advanced/Command scripts). Voila, FreeNAS is running Subversion!

With Oracle 11g this becomes much easier. With the new interval partitioning scheme the database automatically creates new partitions when they are needed. A template partition is created from the start; the system takes care of the rest.

For example, to create a new partition for each day:

create table test_tab (
  created_date   date default sysdate
 ,partition_key  date generated always as (trunc(created_date)) virtual
 ,test_col_1     number
) partition by range (partition_key)
    interval (NUMTODSINTERVAL(1,'DAY')) (
    partition test_tab_initial_part
      values less than (to_date('20100121', 'YYYYMMDD')
  )
);

Note that a virtual column is used for the partitioning key; another 11g feature. Try it out:

insert into test_tab (test_col_1) values (1);
commit;
select partition_name, high_value
  from user_tab_partitions where table_name = 'TEST_TAB';

PARTITION_NAME        HIGH_VALUE
--------------------- ----------------------------
TEST_TAB_INITIAL_PART TO_DATE(' 2010-01-21 ...
SYS_P42               TO_DATE(' 2010-10-29 ...

A new partition has been created on the fly.

While this is great it solves only half the problem. The old partitions must be dropped eventually. How can we automate that?

First we need to find the partitions that can be dropped. This is a bit tricky as the high_value column in user_tab_partitions is a LONG that contains a SQL expression rather than a value we can use in comparisons. Our first task is to write a function that converts a high_value to a date:

create or replace function get_high_value_as_date(
  p_table_name     in varchar2,
  p_partition_name in varchar2
) return date as
  v_high_value varchar2(1024);
  v_date        date;
begin
  select high_value into v_high_value from user_tab_partitions
    where table_name = upper(p_table_name)
      and partition_name = upper(p_partition_name);
  execute immediate 'select ' || v_high_value || ' from dual' into v_date;
  return v_date;
end;

Armed with the conversion function we can loop over the partitions for our table and drop the obsolete ones:

declare
  c_days_to_keep constant integer := 31;
  x_last_partition exception;
  pragma exception_init(x_last_partition, -14758);
begin
  for rec in (select table_name, partition_name
    from user_tab_partitions
      where table_name = 'TEST_TAB'
        and get_high_value_as_date(table_name, partition_name) <
        sysdate - c_days_to_keep) loop
    begin
      execute immediate 'alter table ' || rec.table_name ||
        ' drop partition ' || rec.partition_name;
    exception
      when x_last_partition then
        null;
    end;
  end loop;
end;

Note that we cannot drop the template partition.

It should be possible to define a network drive as follows:

net use x: "http://192.168.157.23:7777/esb/slide"

That fails with:

System error 67 has occurred.

The same thing happens from the user interface in Windows Explorer. Fortunately there is a solution – add a network place instead. Select “Tools/Map Network Drive”. Click on the link “Sign up for online storage or connect to a network server”:

Click next until prompted for the address. Enter the host and port for the ESB server followed by “/esb/slide”:

Give the new network place a name. This will become the name of a folder where the repository can be found:

Finish the wizard. There will be a new folder present in “My Network Places”. It can be copied or moved somewhere else just like a normal folder. Expand it and voila – there are the ESB files.

Extract from opmn.xml

Failover worked, but sometimes opmn started BOTH instances. Only one should be running. To produce the error:

node 1: opmnctl shutdown
(wait until down)
node 2: opmnctl shutdown
(wait until down)
node 2: opmnctl startall
(wait until up)
node 1: opmnctl startall
(wait until up – running on node 2, stopped on node 1)
node 2: opmnctl stopproc process-type=oc4j_apsa
(wait until down – down on node 2, running on node 1)
node 1: opmnctl stopproc process-type=oc4j_apsa
(wait until down – stopped on both)
node 2: opmnctl startproc process-type=oc4j_apsa
(wait until up – started on BOTH nodes!!!)

The solution was simple, though not obvious. It is not good enough to specify process-type or gid, one has to supply ias-component as well! To start:

opmnctl startproc ias-component=apsa_group process-type=oc4j_apsa

To stop:

opmnctl stopproc ias-component=apsa_group process-type=oc4j_apsa

To debug the odd behaviour opmnctl debug was invaluable.

The fix is straightforward. Shrink space for the two lobs that store payload and wirepayload:

alter table tip_datastorage_rt modify lob (clobvalue_clob) (shrink space);
alter table tip_datastorage_rt modify lob (blobvalue) (shrink space);

This can be scheduled as a recurring job or executed manually by a DBA and it can be done online without stopping B2B. To check the amount of storage used before and after, execute as b2b:

select round(sum(us.bytes/1024/1024)) total_mb
  from user_lobs ul, user_segments us
 where ul.segment_name = us.segment_name
   and ul.table_name = 'TIP_DATASTORAGE_RT';

For one system the storage used went from 5648M to 16M, quite a respectable saving.

According to Oracle Support (3-1865217281) it is safe to do this. It has intentionally been left out of the purge job, as it is considered to be a task for a DBA.

Note! Apparently one excellent reason to leave the shrink out of the purge job is that there is a bug in some old Oracle versions that may cause data corruption in lobs after a shrink. Check that you have 10.2.0.4 or better, or alternatively that you have a one-off patch for the relevant bugs fixed in that release.

select s1.sid as blocker_sid,
       s1.username || '@' || s1.machine as blocker_user,
       wm_concat(s2.sid || ':' || s2.username || '@' || s2.machine) as blocked
  from gv$lock l1, gv$session s1, gv$lock l2, gv$session s2
  where s1.sid=l1.sid and s2.sid=l2.sid
    and l1.block=1 and l2.request > 0
    and l1.id1 = l2.id1
    and l2.id2 = l2.id2
  group by s1.sid, s1.username, s1.machine;

The following SQL shows the most recent SQL executed by sessions holding blocking locks (if available):

select sid as blocker_sid, wm_concat(sql_text) as blocker_sql
  from (select s.sid, txt.sql_text
          from gv$sqltext txt, gv$session s, gv$lock l
          where txt.address = s.sql_address
          and s.sid = l.sid
          and l.block = 1
          order by s.sid, txt.piece)
  group by sid order by sid;

Finally the following SQL shows the SQL executed by the sessions that are waiting for blocking locks:

select sid as waiting_sid, wm_concat(sql_text) as waiting_sql
  from (select s.sid, txt.sql_text
          from gv$sqltext txt, gv$session s, gv$lock lb, gv$lock lw
          where txt.address = s.sql_address
          and s.sid = lw.sid
          and lw.id1 = lb.id1
          and lb.block = 1
          and lw.request > 0
          order by s.sid, txt.piece)
  group by sid order by sid;

Armed with this information the root cause is often obvious. The user can be told to commit or rollback, a session can be killed or perhaps an application issue resolved.