Tuning and Optimizing Red Hat Linux Advanced Server = for Oracle9i Database (RedHat, 9i)

From: Subject: Tuning and Optimizing Red Hat Linux Advanced Server for Oracle9i Database (RedHat, 9i) Date: Tue, 27 Jul 2004 12:22:12 +0100 MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_NextPart_000_000A_01C473D4.58079750"; type="text/html" X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 This is a multi-part message in MIME format. ------=_NextPart_000_000A_01C473D4.58079750 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Location: http://www.puschitz.com/TuningLinuxForOracle.shtml Tuning and Optimizing Red Hat Linux Advanced Server = for Oracle9i Database (RedHat, 9i)          &= nbsp;           &n= bsp; Werner=20 Puschitz

= Red=20 Hat =AE Certified Engineer (RHCE)=20
       Linux=20 Professional since 1997=20
          Homepage of Werner Puschitz =

Tuning and Optimizing Red Hat Linux Advanced = Server 2.1=20 for Oracle9i Database

This article has been published in the September = 2003 Australian Unix = Users Group=20 Newsletter (AUUGN) Journal.

>>>=20 Werner's Oracle - Linux Page <<< =

The=20 following procedure is a step-by-step guide with tips and information = for tuning=20 and optimizing Red Hat Linux Advanced Server 2.1 for Oracle9i. This = summary=20 (HOWTO) shows how I tuned and optimized Red Hat AS 2.1 for Oracle 9iR2 = (9.2.0).=20

A procedure for installing Oracle9iR2 on Red Hat AS 2.1 can be = found at=20 Werner's = Oracle - Linux=20 Page.

This article covers the following subjects and = steps:=20

* = Introduction
*=20 Oracle=20 Limits on Linux
* Why=20 Using Red Hat Advanced Server 2.1
* Upgrading=20 the Linux Kernel
* Sizing=20 Swap Space
    =20 Swap Size Recommendations
     Checking=20 Physical Memory
     Checking=20 Swap Space Size and Usage
* Setting=20 Shared Memory
     Setting=20 SHMMAX Parameter
     Setting=20 SHMMNI Parameter
     Setting=20 SHMALL Parameter
* Setting=20 Semaphores
     The=20 SEMMSL Parameter
     The=20 SEMMNI Parameter
     The=20 SEMMNS Parameter
     The=20 SEMOPM Parameter
     Setting=20 the Semaphore Kernel Parameters
* Setting=20 File Handles
* Setting=20 Shell Limits for the Oracle User
     Setting=20 Limits for the Maximum Number of Open File Descriptors for the Oracle=20 User
     Setting=20 Limits for the Maximum Number of Processes for the Oracle = User
* Setting=20 Asynchronous I/O
     Relinking=20 Oracle to Enable Asynchronous I/O for = Oracle9iR2
    =20 Enabling=20 Asynchronous I/O in init.ora for Raw = Devices
     Enabling=20 Asynchronous I/O in init.ora for Filesystem=20 Files
     Increasing=20 I/O Throughput at the Linux OS Level
* Increasing=20 Space for Larger SGA (2.7 GB) to Fit Into=20 Memory
     Address=20 Mappings on Linux - Shared Memory and Shared Library Mapping on=20 Linux
     Changing=20 the Base Address "mapped base" for Shared Libraries at the Linux OS=20 Level
     Changing=20 the Base Address for Shared Memory at the Oracle=20 Level
     Giving=20 Oracle Users the Privilege to Change the Base Address for Oracle's = Shared=20 Libraries Without Giving them root = Access
     Changing=20 the Base Address for Oracle's Shared Libraries Automatically During an = Oracle=20 Login
     Important=20 Notes
* Using=20 Large Memory Pages (Bigpages)
     Sizing=20 Bigpages
     Configuring=20 Bigpages
* Making=20 Other Performance Related Changes
     Disabling=20 Unneeded Background Processes
* Oracle=20 Errors and Problems
* Useful=20 Linux Performance Utilities
     to= p=20 Utility
     sa= r=20 Utility
     vmstat=20 Utility
* Oracle=20 Linux Management
     Determining=20 Which Semaphore Sets and Shared Memory Segments Belong to Each Oracle = Database=20 or Instance
* Hardware=20 Recommendation
* Re= ferences

Introduction

Please point out every = error you=20 can find. I welcome email from any readers with comments, suggestions, = or=20 corrections. My address is webmaster_at_puschitz.com. I will continue to = update=20 and add new information for this article. So make sure to come back. :)=20

Before you begin making any changes to the Linux systems, = make sure=20 that the Oracle database is down!

Oracle Limits = on=20 Linux

Some limits apply to Red Hat Advanced Server only.=20

Linux supports 64-bit file I/O on 32-bit Intel platforms. =
According=20 to the white paper Oracle9= iR2 on=20 Linux: Performance, Reliability and Manageability Enhancements on Red = Hat Linux=20 Advanced Server 2.1", the limits are as follows:
- Number of = files per=20 database: 64K
- Number of blocks per file: 4 million
- Maximum = block size:=20 16 KB
- Maximum size for a database file is 64 GB
- Maximum = database size=20 is 4 petabytes with 16 KB blocks

On a 4 GB RAM machine, the size = of the=20 SGA (SGA utilizes shared memory) can be increased up to is 2.7 GB. This = requires=20 changes in Linux and Oracle. By default, the maximum size is 1.7 GB. =
On a 8=20 GB RAM machine, the size of the SGA can be increased up to 7 GB by using = the=20 shared memory filesystem "shmfs". A maximum = size of 5.4=20 GB of SGA can be created using the "bigpages" feature for System V = shared memory=20 where the page size is 4 MB vs. the regular 4 KB.
On a machine that = supports=20 Physical Address Extension (PAE), the SGA can theoretically have a size = of 62=20 GB. The PAE mechanism allows addressing using 36 bits on IA-32 systems. = But=20 current hardware limitations and practical consideration limit the = actual size=20 of the SGA on such systems.

The number of local concurrent users = on a 4=20 GB server in non-MTS mode can range from 600 through 1200 without = becoming=20 unacceptable slow. For more information on the tpcc run that measured = the number=20 of concurrent users, see L= inux=20 Virtual Memory in Red Hat Advanced Server 2.1 and Oracle's Memory Usage=20 Characteristics.

Why = Using Red=20 Hat Advanced Server

Red Hat Linux Advanced Server has = several=20 features and enhancements that don't exist in other Red Hat versions. = Among=20 other things, Red Hat AS provides:
- Asynchronous I/O
- Process = scheduler=20 with CPU affinity, cache affinity, and per CPU runqueues and locks that = provide=20 better performance
- "mapped base" (base address for shared libaries) = can be=20 changed dynamically allowing larger sizes for the SGA
- Page frame of = size 4=20 MB as opposed to 4 KB can be used for the SGA which improves performance = for=20 large SGAs
- The kernel can also use the "high memory" pool (physical = memory=20 above 1 GB) for allocating page table entries (PTE) which allow a higher = number=20 of Oracle connections
- Elimination of copy to bounce buffer improves = I/O=20 performance

Upgrading = the Linux=20 Kernel

rpm -q =
kernel-smp kernel-enterprise

To check which kernel is=20 currently running, execute the following command:

uname -a

To install e.g. the enterprise = kernel, download the=20 "kernel-enterprise" RPM and execute the following command:

rpm -ivh =
kernel-enterprise-2.4.9-e.25.i686.rpm

To make sure=20 that the right kernel is booted, check the /etc/grub.conf file = if you=20 use GRUB, and change the "default" attribute if necessary. Here is an = example:

default=3D1
timeout=3D10
splashimage=3D(hd0,1)/boot/grub/splash.xpm.gz
title Red Hat Linux (2.4.9-e.25enterprise)
        root (hd0,1)
        kernel /boot/vmlinuz-2.4.9-e.25enterprise ro root=3D/dev/hda2 =
hdc=3Dide-scsi
        initrd /boot/initrd-2.4.9-e.25enterprise.img
title Red Hat Linux Advanced Server (2.4.9-e.25smp)
        root (hd0,1)
        kernel /boot/vmlinuz-2.4.9-e.25smp ro root=3D/dev/hda2 =
hdc=3Dide-scsi
        initrd /boot/initrd-2.4.9-e.25smp.img
title Red Hat Linux Advanced Server-up (2.4.9-e.25)
        root (hd0,1)
        kernel /boot/vmlinuz-2.4.9-e.25 ro root=3D/dev/hda2 =
hdc=3Dide-scsi
        initrd /boot/initrd-2.4.9-e.25.img

In this example, the "default" attribute is set to = "1"=20 which means that the 2.4.9-e.25smp kernel will be booted. If the=20 "default" attribute would be set to "0", then the = 2.4.9-e.25enterprise=20 kernel would be booted.

After you installed the new kernel = and/or made=20 changes to the /etc/grub.conf file, reboot the = server.

Once you=20 are sure you don't need the old kernel anymore, you can remove the old = kernel by=20 running:

su - root
rpm -e <OldKernelVersion>

When you remove the kernel, = you=20 don't need to make any changes to the /etc/grub.conf file.=20

NOTE: Be very careful when removing a kernel! = Making a=20 mistake could render the server unbootable.

Sizing Swap=20 Space

In order to perform a typical Oracle 9i = installation and to=20 create a simple prototype database, Oracle says that you need a minimum = of 512MB=20 of RAM for the Oracle9i Server, and the amount of swap space should be = equal to=20 twice the amount of RAM or at least 400 MB, whichever is greater. Oracle = also=20 says that the minimum swap space should be at least the same as physical = memory=20 size.

Swap Size = Recommendations=20

To summarize Oracle's recommendation for the database and to = take system=20 configurations into account that were used for workload testings, here = is what I=20 came up with:

0.5 GB RAM   1 GB - 2 GB Swap Space
  1 GB RAM   2 GB - 3 GB Swap Space
  2 GB RAM   2 GB - 3 GB Swap Space
  3 GB RAM   3 GB Swap Space
  4 GB RAM   4 GB Swap Space
  8 GB RAM   4 GB Swap Space
 16 GB RAM   8 GB Swap Space

The swap space will not be utilized until the system runs out of = physical=20 memory. So don't configure too much swap space. Keep in mind that if the = system=20 starts using swap space, it has a negative impact to the performance of = the=20 database. So make sure that the system has always enough physical RAM = and that=20 it doesn't use swap space continuously.

Checking Physical Memory

You = can check=20 the size of physical memory by running the following command:

grep MemTotal /proc/meminfo

You can = find a detailed description=20 of the entries in /proc/meminfo at http://www.redhat= .com/advice/tips/meminfo.html.=20

Alternatively, you can use free(1) to check the memory: =

# free
             total       used       free     shared    buffers     =
cached
Mem:       1031004     734656     296348          0     262404    =
 287388
-/+ buffers/cache:     184864     846140
Swap:      2097144      40184    2056960
#

In this example the total amount of available memory is = 1031004=20 bytes. 184864 bytes are used by programs and 846140 bytes are available = for more=20 programs.
Don't get confused with the first line that shows that = 296348=20 bytes are free! If you look at the usage figures you can see that most = of the=20 increase of memory is for buffers and cache. Linux tries to use all the = memory=20 for disk buffers and cache. It helps the system to run faster because = disk=20 information is already in memory and Linux doesn't have to read it from = disk=20 again. If space is needed by a program or application like Oracle, Linux = will=20 make the space available immediately. So if your system runs for a = while, you=20 will usually see a small number for "free" in the first line, and there = is=20 nothing to be worried about.

Note: If you create a large SGA = (shared=20 memory) and start the database, free won't show all the memory = that has=20 been allocated for SGA as "used" right away. That's because Linux does = not=20 assign page frames to a memory mapping right after it has been created = due to=20 reasons of efficiency.

Checking=20 Swap Space Size and Usage

You can check the size and current = usage=20 of swap space by running the following command:

cat /proc/swaps

If your swap partition = is not large enough, you=20 can add another swap partitions to your system. See "Adding=20 Swap Space" for more information. Adding a permanent swap file to = the system=20 is not recommended due to the performance impact of the filesystem = layer.=20

Setting = Shared=20 Memory

Shared memory allows processes to access common = structures=20 and data by placing them in shared memory segments. It's the fastest = form of IPC=20 (Interprocess Communication) available since no kernel involvement = occurs when=20 data is passed between the processes. In fact, data does not need to be = copied=20 between the processes.
Oracle uses shared memory segments for the = SGA=20 (Shared Global Area) which is an area of memory that is shared by all = Oracle=20 background and foreground processes. The size of the SGA has a major = impact to=20 Oracle's performance since it holds database buffer cache and much more. =

To see all shared memory settings, run:

ipcs -lm

Setting SHMMAX=20 Parameter

This parameter defines the maximum size in bytes = for a=20 shared memory segment. Since the SGA is comprised of shared memory, = SHMMAX can=20 potentially limit the size of the SGA. Ideally, SHMMAX should be large = enough so=20 that SGA can fit into one segment.
The default size on RH 2.1 AS is=20 33554432. With this value, the Oracle Database Configuration Assistant = failed on=20 my server with the following error message:

ORA-27123: unable to attach to shared memory =
segment

Setting SHMMAX to 1 GB always worked for me when I setup a = medium=20 sized database. However, it is suggested that it should be set to 2 GB; = the=20 default maximum size of the SGA is 1.7 GB which requires a larger = SHMMAX. And if=20 the available size of the SGA is set to 2.7 GB by changing "mapped=20 base" at the Linux OS level, then SHMMAX should be set to 3 GB. = The=20 maximum value of SHMMAX can be set to 4GB-1. (A typical 32-bit Linux = system=20 without Physical Address Extension (PAE) is divided into 3 GB user space = and 1=20 GB kernel space.)

The default shared memory limit for SHMMAX can = be=20 changed in the proc file system without reboot:

su - root
echo "2147483648" > =
/proc/sys/kernel/shmmax

Alternatively, you=20 can use sysctl(8) to change it:

sysctl -w kernel.shmmax=3D2147483648

To = make the change=20 permanent, add the following line to the file /etc/sysctl.conf. = This=20 file is used during the boot process.

echo =
"kernel.shmmax=3D2147483648" >> =
/etc/sysctl.conf

Setting SHMMNI Parameter

This = parameter=20 sets the maximum number of shared memory segments system wide. The = default=20 number on RH 2.1 AS is 4096. To my knowledge this value should be = sufficient.

# cat =
/proc/sys/kernel/shmmni
4096

Setting SHMALL Parameter=20

This parameter sets the total amount of shared memory in pages = that can=20 be used at one time on the system. So shmall should always be = at least=20 ceil(SHMMAX/PAGE_SIZE).

The default size for = shmall on=20 RH 2.1 AS is 2097152. This should be sufficient since it means that the = total=20 amount of shared memory available on the system is 2097152*4096 bytes=20 (shmall*PAGE_SIZE). On i386 architectures, the = PAGE_SIZE in=20 RHAS 2.1 (UP and SMP kernel) is 4096 bytes unless you use bigpages=20 which supports the configuration of larger memory pages.

# =
cat /proc/sys/kernel/shmall
2097152

Setting=20 Semaphores

Semaphores can best be described as counters = which are=20 used to provide synchronization between processes or between threads = within a=20 process for shared resources like shared memories. System V semaphores = support=20 semaphore sets where each one is a counting semaphore. So when an = application=20 requests semaphores, the kernel releases them in "sets". The number of=20 semaphores per set can be defined through the kernel parameter SEMMSL.=20

To see all semaphore settings, run:

ipcs -ls

The SEMMSL=20 Parameter

This parameter defines the maximum number of = semaphores=20 per semaphore set.

Oracle recommends to set SEMMSL to the = largest=20 PROCESSES init.ora parameter of any database on the Linux system plus = 10.=20
Oracle also recommends to set SEMMSL to a minimum value of 100. =

The=20 init.ora parameter PROCESSES specifies the maximum number of operating = system=20 processes that can be started by the Oracle instance. In a non MTS = environment,=20 Oracle spawns a system user process for each connection. This means that = in such=20 an environment the PROCESSES parameter defines the maximum number of=20 simultaneous Oracle connections minus sum of all Oracle background = processes.=20
It can also be said that the PROCESSES value should never be greater = than=20 SEMMSL.

The SEMMNI Parameter =

This parameter defines the maximum number of semaphore sets in = the=20 entire Linux system.

Oracle recommends to set SEMMNI to a = minimum value=20 of 100.

The SEMMNS Parameter =

This parameter defines the total number of semaphores (not = semaphore=20 set) in the entire Linux system. A semaphore set can have more than one=20 semaphore, and according to the semget(2) man page, values greater than = SEMMSL *=20 SEMMNI makes it irrelevant.

Setting it to a minimum value of 256 = is for=20 initial Oracle installation only.
Oracle recommends to set SEMMNS to = the sum=20 of the PROCESSES parameter for each database on the system, adding the = largest=20 PROCESSES twice, and then adding 10 for each DB.

The maximum = number of=20 semaphores that can be allocated on a Linux system will be the lesser = of:=20
SEMMNS or (SEMMSL * SEMMNI)

Setting SEMMSL = and=20 SEMMNI to 100 makes sure that SEMMNS semaphores can be allocated as = determined=20 by the above calculation.

The = SEMOPM=20 Parameter

This parameter defines the maximum number of = semaphore=20 operations that can be performed per semop(2) system call. =

The=20 semop(2) function provides the ability to do operations for = multiple=20 semaphores with one semop(2) system call. Since a semaphore set = can=20 have the maximum number of SEMMSL semaphores per semaphore set, it is = often=20 recommended to set SEMOPM equal to SEMMSL.

Oracle recommends to = set=20 SEMOPM to a minimum value of 100.

Setting the Semaphore Kernel=20 Parameters

To determine the values of the four described = semaphore=20 parameters, run:

# cat /proc/sys/kernel/sem
250     32000   32      128

Alternatively, you can run:

ipcs =
-ls

All four described semaphore=20 parameters can be changed in the proc file system without reboot: =

su - root
# echo SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value > =
/proc/sys/kernel/sem
# These are the values I'm using since I don't want to lower Red Hat's =
default=20
# values. The only value I raise is SEMOPM to comply with Oracle's =
minimum=20
# requirement for SEMOPM.
echo 250 32000 100 128 > =
/proc/sys/kernel/sem

Alternatively,=20 you can use sysctl(8) to change it:

sysctl -w kernel.sem=3D"250 32000 100 =
128"

To make the change=20 permanent, add or change the following line in the file=20 /etc/sysctl.conf. This file is used during the boot process. =

echo "kernel.sem=3D250 32000 100 128" >> =
/etc/sysctl.conf

To=20 see the new updated semaphore settings, run:

ipcs -ls

Setting File=20 Handles

The maximum number of file handles denotes the = maximum=20 number of open files that you can have on the Linux system. =

Setting System Wide Limit = for File=20 Handles

The value in /proc/sys/fs/file-max sets the = maximum=20 number of file handles or open files that the Linux kernel will = allocate. When=20 you get error messages about running out of file handles, then you might = want to=20 raise this limit. The default value on RH 2.1AS is 8192.

For an = Oracle=20 server it is recommended that the file handles for the entire system is = set to=20 at least 65536.

To determine the maximum number of file handles = for the=20 entire system, run:

cat =
/proc/sys/fs/file-max

To determine the current usage of=20 file handles, run:

$ cat =
/proc/sys/fs/file-nr
1154    133     8192

The file-nr file displays three=20 parameters:
  - Total allocated file = handles
  -=20 Currently used file handles
  - Maximum file handles that = can be=20 allocted (see also file-max)

The kernel dynamically = allocates=20 file handles whenever a file handle is requested by an application, but = the=20 kernel does not free these file handles when they are released by the=20 application. The kernel recycles these file handles instead. This means = that=20 over time the total number of allocated file handles will increase even = though=20 the number of currently used file handles may be low.

The = maximum number=20 of file handles can be changed in the proc file system without reboot: =

su - root
echo "65536" > /proc/sys/fs/file-max

Alternatively, you = can use=20 sysctl(8) to change it:

sysctl -w =
fs.file-max=3D65536

To make the change permanent, add=20 or change the following line in the file /etc/sysctl.conf. This = file is=20 used during the boot process.

echo =
"fs.file-max=3D65536" >> /etc/sysctl.conf

Setting=20 Shell Limits for the Oracle User

Most shells like Bash = provide=20 control over various resources like the maximum allowable number of open = file=20 descriptors or the maximum number of processes available to a user. =

To=20 see all shell limits, run:

ulimit =
-a

For more information on=20 ulimit for the Bash shell, see man bash and search for = ulimit.

Setting=20 Limits for the Maximum Number of Open File Descriptors for the Oracle = User=20

After you changed and increased /proc/sys/fs/file-max = at Setting=20 File Handles, there is still a per user limit of open file = descriptors which=20 is set to 1024 by default:

$ su - =
oracle
$ ulimit -n
1024
$

To change this, you have to edit the file /etc/security/limits.conf as root and = make the=20 following changes or add the following lines, respectively:

oracle           soft    nofile          4096
oracle           hard    nofile          63536

The "soft limit" in the first line defines the number of = file=20 handles or open files that the Oracle user will have after login. If the = Oracle=20 user gets error messages about running out of file handles, then the = Oracle user=20 can increase the number of file handles like in this example up to 63536 = ("hard=20 limit") by running the following command:

ulimit -n 63536

You can set the "soft" = and "hard" limits higher=20 if necessary. Note that I do not recommend to set the "hard" limit = for=20 nofile for the oracle user equal to=20 /proc/sys/fs/file-max. If you do that and the user uses up all = the file=20 handles, then the system would run out of file handles. This could mean = that you=20 won't be able to initiate new remote logins any more since the system = won't be=20 able to open any PAM modules which are required for performing a login. = That's=20 why I set the hard limit to 63536 and not to 65536.

You also = need to=20 make sure that pam_limits is configured in the file /etc/pam.d/system-auth. This is the PAM = module=20 that will read the /etc/security/limits.conf file. The entry = should=20 read like:

session     required      =
/lib/security/pam_limits.so

Here are the two "session" entries I have in my=20 /etc/pam.d/system-auth file:

session     required      =
/lib/security/pam_limits.so
session     required      /lib/security/pam_unix.so

Now login to the oracle account again since the changes will = become=20 effective for new login sessions only.

$ su - =
oracle
$ ulimit -n
4096
$

Note that the ulimit options are different for other=20 shells.

The default limit for oracle is now 4096 and the = oracle user=20 can increase the number of file handles up to 63536:

$ su - oracle
$ ulimit -n
4096
$ ulimit -n 63536
$ ulimit -n
63536
$

To make this change permanent, add "ulimit -n 63536" = (for Bash)=20 to the ~oracle/.bash_profile file which is the user startup = file for=20 the Bash shell on Red Hat Linux (to verify your shell run: echo $SHELL). To do this you could simply = copy/paste the=20 following commands for the oracle's Bash shell:

su - oracle
cat >> ~oracle/.bash_profile << EOF
ulimit -n 63536
EOF

Settin= g Limits=20 for the Maximum Number of Processes for the Oracle User =

After=20 reading the procedure at Setting=20 Limits for the Maximum Number of Open File Descriptors for the Oracle = User,=20 you should now understand what "soft" and "hard" limits are, how to = configure=20 pam_limits.so, and how to change the limits.

To see the = current=20 limit of the maximum number of processes for the oracle user, = run:

su - oracle
ulimit -u

Note that the ulimit options are different for other=20 shells.

To change the "soft" and "hard" limits for the = maximum=20 number of processes for the oracle user, add the following = lines to the=20 /etc/security/limits.conf file: =

oracle           soft    nproc          2047
oracle           hard    nproc          16384

To make this change permanent, add "ulimit -u = 16384" (for=20 Bash) to the ~oracle/.bash_profile file which is the user = startup file=20 for the Bash shell on Red Hat Linux (to verify your shell run: echo $SHELL). To do this you could simply = copy/paste the=20 following commands for the oracle's Bash shell:

su - oracle
cat >> ~oracle/.bash_profile << EOF
ulimit -u 16384
EOF

Setting = Asynchronous=20 I/O

Red Hat Advanced Server supports asynchronous I/O in = the=20 kernel. Asynchronous I/O permits Oracle to continue processing after = issuing=20 I/Os requests which leads to much higher I/O throughputs. This = enhancement also=20 allows Oracle to issue thousands of simultaneous I/O requests with a = single=20 system call. It also reduces context switch overhead.

According = to a Red=20 Hat webcast I attended, only 2 Oracle dbwriter processes are needed when = asynchronous I/O is being used.

To enable Oracle to use = asynchronous=20 I/O, it is necessary to relink Oracle. Oracle ships Oracle9iR2 with = asynchronous=20 I/O support disabled. According to Oracle, this is necessary to = accommodate=20 other Linux distributions that do not support asynchronous I/O. =

Relinking Oracle to Enable=20 Asynchronous I/O for Oracle9iR2

# shutdown Oracle
SQL> shutdown

su - oracle
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk async_on
make -f ins_rdbms.mk ioracle

# The last step creates a new "oracle" executable =
"$ORACLE_HOME/bin/oracle".
# It backs up the old oracle executable to $ORACLE_HOME/bin/oracleO,
# it sets the correct privileges for the new Oracle executable "oracle",
# and moves the new executable "oracle" into the $ORACLE_HOME/bin =
directory.

If asynchronous I/O needs to be disabled for any reason, run = the=20 following commands:

# shutdown Oracle
SQL> shutdown

su - oracle
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk async_off
make -f ins_rdbms.mk ioracle

Enabling=20 Asynchronous I/O in init.ora for Raw Devices

The=20 disk_asynch_io init.ora parameter needs to be set to true: =

disk_asynch_io=3Dtrue

Note that this init.ora parameter is already set to true by = default:

SQL> select value, isdefault from v$parameter where =
name =3D 'disk_asynch_io';

VALUE                          ISDEFAULT
------------------------------ ---------
TRUE                           TRUE

Enabling=20 Asynchronous I/O in init.ora for Filesystem Files

Make sure = that all=20 Oracle datafiles reside on filesystems that support asynchronous I/O = (e.g.=20 "ext2"). According to Oracle's white paper Oracle9= iR2 on=20 Linux: Performance, Reliability and Manageability Enhancements on Red = Hat Linux=20 Advanced Server 2.1", Oracle9iR2 has been certified with the = standard Linux=20 filesystem "ext2" on RH AS 2.1. In addition, Oracle has also been = certified for=20 raw devices.

The disk_asynch_io init.ora parameter = needs to be=20 set to true (same as for raw devices):

disk_asynch_io=3Dtrue

Note that this init.ora parameter is already set to true by = default:

SQL> select value, isdefault from v$parameter where =
name =3D 'filesystemio_options';

VALUE                          ISDEFAULT
------------------------------ ---------
none                           TRUE

SQL>

The filesystemio_options init.ora parameter needs to be = set to=20 asynch:

filesystemio_options=3Dasynch

This init.ora parameter is platform-specific. By default, = this=20 parameter is set to none for Linux and thus needs to be changed. =

SQL> select value, isdefault from v$parameter where name =3D =
'filesystemio_options';

VALUE                          ISDEFAULT
------------------------------ ---------
none                           TRUE

SQL>

The filesystemio_options can have the following = values=20 with Oracle9iR2:
   asynch: This value enables=20 asynchronous I/O on file system files.
   = directio: This=20 value enables direct I/O on file system files.
   = setall:=20 This value enables both asynchronous and direct I/O on file system=20 files.
   none: This value disables both = asynchronous and=20 direct I/O on file system files.

Increasing I/O Throughput = at the=20 Linux OS Level

The /proc/sys/fs/aio-max-size = parameter can=20 be changed if asynchronous I/O is used for Oracle datafiles residing on=20 filesystems (e.g. "ext2"). To my knowledge, this parameter does not have = any=20 effect to raw devices. According to the Oracle9= iR2 on=20 Linux: Performance, Reliability and Manageability Enhancements on Red = Hat Linux=20 Advanced Server 2.1" document, Oracle9iR2 has been certified with = the=20 standard Linux filesystem "ext2" on RH AS 2.1.

To get better I/O = throughput for Decision Support Systems (DSS) workloads, the=20 /proc/sys/fs/aio-max-size parameter should be increased to > = 1 MB. A=20 typical DSS system queries large amount of data and makes heavy use of = full=20 table scans. Parallel Query is particularly designed for DSS. =

For Online=20 Transaction Processing (OLTP) workloads, the default size of 131072 = would=20 suffice. A typical OLTP system has high throughputs, are insert- and=20 update-intensive, have concurrent access by many users, and have large,=20 continuously growing data volume.

To determine the number of = bytes, run:=20

su - root
# cat /proc/sys/fs/aio-max-size
131072

The maximum number of bytes can be changed for e.g. DSS systems in = the=20 proc file system without reboot:

echo =
"2147483648" > /proc/sys/fs/aio-max-size

Alternatively,=20 you can use sysctl(8) to change it:

sysctl -w =
fs.aio-max-size=3D2147483648

To make the change=20 permanent, add or change the following line in the file=20 /etc/sysctl.conf. This file is used during the boot process. =

echo "fs.aio-max-size=3D2147483648" >> =
/etc/sysctl.conf

Increasing Space=20 for larger SGA (2.7 GB) to Fit Into Memory

If the size of = SGA=20 does not need to be increased from 1.7 GB to 2.7 GB, then the following = steps=20 can be skipped.

By default, the maximum size for SGA is 1.7 GB = on a=20 32-bit system without Physical Address Extension (PAE). You will also be = able to=20 allocate 1.7 GB SGA if you have less than 4 GB RAM. In this case you = have to=20 make sure you have enough swap space, however, this will have an impact = to the=20 performance of the database. I was even able to bring up a database with = a SGA=20 size of 2.64 GB on a test PC that had 256 MB RAM.

Theoretically, = the SGA=20 can have a size of up to 62 GB on a system that supports Physical = Address=20 Extension (PAE). The PAE mechanism allows addressing using 36 bits on = IA-32=20 systems. But current hardware limitations and practical consideration = limit the=20 actual size of the SGA on such a system. Since I do not have such a = system, I=20 will not cover the steps for creating SGAs larger than 2.7 GB via the tmpfs filesystem. =

To=20 increase the size of the SGA to 2.7 GB without using a shared memory = filesystem=20 (tmpfs), the = following needs=20 to be done:
- The base address "mapped base" for = Oracle's shared=20 libraries has to be lowered at the Linux OS level.
- = Oracle=20 needs to be relinked with a lower base address for SGA which uses shared = memory=20 segments.

Address = Mappings on=20 Linux - Shared Memory and Shared Library Mapping on Linux =

Normally,=20 the 4 GB linear address space (also known as virtual address space) for = a 32-bit=20 Linux system is split into 4 equal sized sections for different = purposes:

0GB-1GB  User space   - Used for executable and brk/sbrk =
allocations (malloc uses brk for small chunks).
1GB-2GB  User space   - Used for mmaps (shared memory), shared libraries =
and malloc uses mmap (malloc uses mmap for large chunks).
2GB-3GB  User space   - Used for stack.
3GB-4GB  Kernel Space - Used for the kernel itself.

- The mmaps grow bottom up and the stack grows top down. = The=20 unused space used by the one can be used by the other.
- The = split=20 between userspace and kernelspace can be changed by setting the kernel = parameter=20 PAGE_OFFSET and recompiling the kernel. By default, the PAGE_OFFSET = macro yields=20 the value 0xc0000000.
- The split between brk(2) and=20 mmap(2) can be changed by setting the kernel parameter=20 TASK_UNMAPPED_BASE and recompiling the kernel. However, on Red Hat AS = this=20 parameter can be changed for individual processes dynamically without = reboot or=20 kernel recompilation.

Usually, the portion of address space = available=20 for mapping shared libraries and shared memory segments consists of = virtual=20 addresses in the range of 0x40000000 (1 GB) - 0xc0000000 (3 GB). On Red = Hat AS,=20 0x40000000 is the default base address for shared libraries and shared = memory=20 segments. The default base address for mapping shared memory segments = can be=20 changed and overwritten for programs and applications by non-root users. = The=20 default base address "mapped base" for loading shared libraries for = programs and=20 applications can be changed by the user root only.

The default = base=20 address that Oracle uses for SGA (shared memory segment) is 0x50000000 = and not=20 0x40000000. Oracle uses or keeps the space from 0x40000000-0x50000000 = for=20 loading Oracle shared libraries. As I mentioned before, 0x40000000 is = the=20 default base address on RH AS for loading shared libraries which can = only be=20 changed by the user root. Oracle increased the base address for SGA to = prevent=20 address range conflicts between the segments (shared memory segment and = shared=20 libraries).
If the base address for shared memory segments would be=20 0x15000000 and if the base address for shared libraries would be = 0x40000000,=20 then Oracle cannot create the SGA larger than 0x2b000000 bytes or 688 = MB, even=20 though there is address space available above the shared libraries = portion.=20 (According to Oracle, Oracle binaries will no longer work if the base = address=20 for shared memory segments is lower than the base address shared = libraries like=20 in this example. Even though I didn't experience any problems, I would = not=20 recommend it).
If the base address for shared memory segments is = 0x50000000=20 and if the base address for shared libraries is 0x40000000, then Oracle = can=20 create a SGA that starts at 0x50000000 and ends almost at 0xc0000000; = 0xc0000000=20 is the address where the kernel address space begins. This means that = the SGA=20 can have a size of almost 0x70000000 bytes or 1.792 GB - actually it's = about 100=20 MB less due to stack space and other use of memory.

Once again, = Oracle=20 increased the default base address for SGA to 0x50000000 so that all = shared=20 libraries can be loaded below 0x50000000, and the rest of the space up = to almost=20 0xc0000000 can be used for shared memory.

You can verify the = address=20 mappings of Oracle processes by viewing the proc file=20 /proc/<pid>/maps where <pid> stands for the Oracle = process=20 ID. The default mapping of an Oracle process might look like this: =

08048000-0ab11000 r-xp 00000000 08:09 273078     =
/ora/product/9.2.0/bin/oracle
0ab11000-0ab99000 rw-p 02ac8000 08:09 273078     =
/ora/product/9.2.0/bin/oracle
0ab99000-0ad39000 rwxp 00000000 00:00 0
40000000-40016000 r-xp 00000000 08:01 16         /lib/ld-2.2.4.so
40016000-40017000 rw-p 00015000 08:01 16         /lib/ld-2.2.4.so
40017000-40018000 rw-p 00000000 00:00 0
40018000-40019000 r-xp 00000000 08:09 17935      =
/ora/product/9.2.0/lib/libodmd9.so
40019000-4001a000 rw-p 00000000 08:09 17935      =
/ora/product/9.2.0/lib/libodmd9.so
4001a000-4001c000 r-xp 00000000 08:09 16066      =
/ora/product/9.2.0/lib/libskgxp9.so
...
42606000-42607000 rw-p 00009000 08:01 50         =
/lib/libnss_files-2.2.4.so
50000000-50400000 rw-s 00000000 00:04 163842     /SYSV00000000 =
(deleted)
51000000-53000000 rw-s 00000000 00:04 196611     /SYSV00000000 (deleted)
53000000-55000000 rw-s 00000000 00:04 229380     /SYSV00000000 (deleted)
...
bfffb000-c0000000 rwxp ffffc000 00:00 0

As this address mapping shows, shared libraries start at base = address=20 0x40000000. The address mapping also shows that Oracle uses the base = address=20 0x50000000 for SGA (in this example System V shared memory for SGA). = Here is a=20 summary of all the entries:

The text (code) section is mapped at = 0x08048000:

  08048000-0ab11000 r-xp 00000000 08:09 273078     =
/ora/product/9.2.0/bin/oracle

The data section is mapped at 0x0ab11000:

  0ab11000-0ab99000 =
rw-p 02ac8000 08:09 273078     /ora/product/9.2.0/bin/oracle

The uninitialized data segment .bss is allocated at 0x0ab99000: =

  0ab99000-0ad39000 rwxp 00000000 00:00 0

The base address for shared libraries is 0x40000000:

  =
40000000-40016000 r-xp 00000000 08:01 16         /lib/ld-2.2.4.so

The base address for SGA (System V shared memory) is 0x50000000: =

  50000000-50400000 rw-s 00000000 00:04 163842     /SYSV00000000 =
(deleted)

The stack is allocated at 0xbfffb000:

  bfffb000-c0000000 =
rwxp ffffc000 00:00 0

Now it should become clear what needs to be done to provide more = space for=20 SGA. To increase the space for SGA, two base addresses need to be = changed. The=20 base address "mapped base" for shared libraries needs to be lowered at = the Linux=20 OS level, and the base address for SGA (shared memory) needs to be = lowered at=20 the Oracle level (application level).

Note: Once the base = addresses=20 have been changed at the Linux OS level and at the Oracle level, all = Oracle=20 commands need to be executed with a lower "mapped base"! This means that = every=20 new shell must run with a lowered "mapped base". Further down I will = show you=20 how you can automate this so that every Oracle user gets automatically a = shell=20 with a lowered "mapped base".

Changing the Base Address = "mapped=20 base" for Shared Libraries at the Linux OS Level

The default = base=20 address "mapped base" on RH 2.1AS is TASK_UNMAPPED_BASE =3D 0x40000000 = (decimal=20 1073741824 or 1 GB). This is the address that splits the section between = brk(2) and mmap(2), which defines available space for = shared=20 libraries (if it hasn't been changed and overwritten at the application = level)=20 and for shared memory (e.g. SGA).

To change "mapped base" for a = Linux=20 process, the file /proc/<pid>/mapped_base needs to be = changed=20 where <pid> stands for the process ID. Note that this is not a = system wide=20 parameter! So in order to change "mapped base" for the Oracle database = (i.e.=20 Oracle processes), the parent shell that starts the database needs to be = modified at the Linux OS level to allow it's child processes to inherit = the=20 change. The following procedure shows how this can be done. =

Execute the=20 following command to identify the process ID "pid" of the shell process = used by=20 the Oracle user that will start the database:

echo $$

As root in another shell, = change "mapped base" to=20 0x10000000 (decimal 268435456 bytes or 256 MB) for the Oracle shell with = the pid=20 we identified above:

su - root
echo 268435456 > /proc/<pid>/mapped_base

This will = tell the=20 kernel to load shared libraries at the virtual address portion starting = at=20 0x10000000. Now if Oracle is started with sqlplus in the shell = used by=20 the Oracle user for which we changed "mapped base", the Oracle processes = will=20 inherit the new base address.

Once the base=20 address for shared memory has been changed at the Oracle level as = well, more=20 space will become available for the SGA. To accommodate the increased = space for=20 shared memory allocations by the Oracle processes, the maximum value of = SHMMAX=20 needs to be raised. This value defines the largest shared memory segment = size=20 allowed by the kernel. Since the SGA can be increased up to 2.7 GB with = this=20 method, the maximum size for SHMMAX can be rounded to 3000000000. This = will=20 allow Oracle to allocate one large shared memory segment for the SGA. = This is=20 also what Oracle recommends.

The maximum size SHMMAX for a = shared memory=20 segment can be changed in the proc file system without reboot:

su - root
echo "3000000000" > =
/proc/sys/kernel/shmmax

Alternatively, you=20 can use sysctl(8) to change it:

sysctl -w =
kernel.shmmax=3D3000000000

To make the change=20 permanent, add or change the following line in the file /etc/sysctl.conf. This file is used = during the=20 boot process.

kernel.shmmax=3D3000000000

Changing the Base Address = for Shared=20 Memory at the Oracle Level

The previous steps showed how to = lower=20 the base address "mapped base" for Oracle's shared libraries to = 0x10000000 (256=20 MB). The following steps show how to lower the base address for shared = memory=20 (SGA) for Oracle to 0x15000000 (336 MB).

The base address for = SGA=20 (shared memory) should not be lowered to 0x10000000 at the Oracle level. = As I=20 explained in the section "=20 Address Mappings on Linux - Shared Memory and Shared Library Mapping on=20 Linux", to prevent address range conflicts between the segments = (Oracle=20 shared libraries and Oracle shared memory), the address at which the SGA = should=20 be attached is 0x15000000. It can be lowered to 0x12000000, but this = would=20 require thorough testing. So I would not recommend it.

The = following=20 calculation shows how large the SGA can be created:

   0xc0000000  =
(base address of the kernel space -> 3 GB)
 - 0x15000000  (base address of SGA -> 336 MB)
 -------------
   0xab000000  (decimal 2868903936 or 2.736 GB)
 - stack space
 - other memory allocations
 ------------
 ~ 2.65 to 2.70 GB

To lower the base address at which the SGA (shared memory) = should be=20 attached, Oracle needs to be relinked. Changing the base address for SGA = can be=20 done on Linux with genksms, which is an Oracle utility: =

  # shutdown Oracle
  SQL> shutdown

  su - oracle
  cd $ORACLE_HOME/rdbms/lib

  # Make a backup of the ksms.s file if it exists
  [[ -f ksms.s ]] && cp ksms.s ksms.s_orig

  # Modify the attach address in the ksms.s file before relinking =
Oracle
  genksms -s 0x15000000 > ksms.s

Rebuild the Oracle executable in the=20 $ORACLE_HOME/rdbms/lib directory by entering the following = commands:

  # Create a new ksms object file
  make -f ins_rdbms.mk ksms.o

  # Create a new "oracle" executable =
($ORACLE_HOME/bin/oracle):
  make -f ins_rdbms.mk ioracle

  # The last step will create a new Oracle kernel that loads the =
SGA at
  # the address specified by sgabeg in ksms.s:
  # .set   sgabeg,0X15000000
  # It also backs up the old oracle executable to =
$ORACLE_HOME/bin/oracleO,
  # it sets the correct privileges for the new Oracle executable =
"oracle", and
  # moves the new executable "oracle" into the $ORACLE_HOME/bin =
directory.

Now when Oracle is started, the lowered base addresses for = Oracle's=20 shared library and shared memory (SGA) can be seen with the following = commands:

  # Get the pid of e.g. the Oracle checkpoint process
  su - oracle
  $ pgrep -f -x ora_dbw0_$ORACLE_SID -l
  13519 ora_dbw0_test
  # You can also use /sbin/pidof to get the process ID
  $ /sbin/pidof ora_dbw0_$ORACLE_SID
  13519
  $ DBW0_PID=3D`pgrep -f -x =
ora_dbw0_$ORACLE_SID`
  $ echo $DBW0_PID
  13519

  # Check the base addresses for shared libraries and shared memory for =
the
  # process ID 1049:

  $ grep '.so' /proc/$DBW0_PID/maps |head =
-1
  10000000-10016000 r-xp 00000000 03:02 750738     =
/lib/ld-2.2.4.so

  $ grep 'SYS' /proc/$DBW0_PID/maps |head =
-1
  15000000-24000000 rw-s 00000000 00:04 262150     /SYSV3ecee0b0 =
(deleted)
  $

Now you can increase the init.ora parameters db_cache_size or db_block_buffer to create a larger database buffer = cache. If the=20 size of the SGA is larger than 2.65 GB, then I would test the database = very=20 thoroughly to make sure no other memory allocation problems arise. =

For=20 fun I tried to test these settings on a little test PC with 256 MB RAM = and 4 GB=20 swap space. I wanted to see if I was able to bring up a database on such = a=20 little PC. I set db_block_buffer to 315000 and = db_block_size=20 to 8192 (2580480000 bytes), and I was able to bring up a database with = 2.654 GB=20 (2850033824 bytes) SGA on this PC:

Total System Global Area =
2850033824 bytes
Fixed Size                   450720 bytes
Variable Size             268435456 bytes
Database Buffers         2580480000 bytes
Redo Buffers                 667648 bytes

Giving Oracle = Users the=20 Privilege to Change the Base Address for Oracle's Shared Libraries = Without=20 Giving them root Access

As shown above, only root can change = the=20 base address "mapped base" for shared libraries. Using sudo we = can give=20 Oracle users the privilege to change "mapped base" for their own shells = without=20 giving them full root access. Here is the procedure:

su - root

# E.g. create a script called "/usr/local/bin/ChangeMappedBase"
# which changes the "mapped base" for the parent process,
# the shell used by the Oracle user where the "sudo" program
# is executed (forked). Here is an example:

#/bin/sh
# Lowering "mapped base" to 0x10000000
echo 268435456 > /proc/$PPID/mapped_base

# Make sure that owernship and permissions are correct
chown root.root /usr/local/bin/ChangeMappedBase
chmod 755 /usr/local/bin/ChangeMappedBase

# Allow the Oracle user to execute /usr/local/bin/ChangeMappedBase via =
sudo
echo "oracle   =
ALL=3D/usr/local/bin/ChangeMappedBase" >> /etc/sudoers

Now the Oracle user can run = /usr/local/bin/ChangeMappedBase to=20 change "mapped base" for it's own shell:

$ su =
- oracle
$ cat /proc/$$/mapped_base; echo
1073741824
$ sudo /usr/local/bin/ChangeMappedBase
Password:   # type in the password for the Oracle user account
$ cat /proc/$$/mapped_base; echo
268435456
$

When /usr/local/bin/ChangeMappedBase is executed the = first time=20 after an Oracle login, sudo will ask for a password. The = password that=20 needs to be entered is the password of the Oracle user account.=20

Changing the=20 Base Address for Oracle's Shared Libraries Automatically During an = Oracle=20 Login

The procedure in the previous section asks for a = password each=20 time /usr/local/bin/ChangeMappedBase is executed the first time = after=20 an Oracle login. To have "mapped base" changed automatically during an = Oracle=20 login without a password, the following can be done:

Edit the = /etc/sudoers file with visudo: =

su - root
visudo

Change the entry in /etc/sudoers from:

oracle   ALL=3D/usr/local/bin/ChangeMappedBase

to read:

oracle   =
ALL=3DNOPASSWD: /usr/local/bin/ChangeMappedBase

Make=20 sure bash executes /usr/local/bin/ChangeMappedBase = during the=20 login process. You can use e.g. ~oracle/.bash_profile:

su - oracle
echo "sudo /usr/local/bin/ChangeMappedBase" >> =
~/.bash_profile

The=20 next time you login to Oracle, the base address for shared libraries = will bet=20 set automatically.

$ ssh =
oracle@localhost
oracle@localhost's password:
Last login: Sun Apr  6 13:59:22 2003 from localhost
$ cat /proc/$$/mapped_base; echo
268435456
$

Important Notes

When = the base=20 address "mapped base" for Oracle's processes has changed, then every = Linux shell=20 that spawns Oracle processes (e.g. listener) must have the same "mapped = base" as=20 well. This means that even shells that that are used to connect locally = to the=20 database need to have the same "mapped base". For example, if you run=20 sqlplus to connect to the local database, then you will get the = following error message if "mapped base" of this shell is not the same = as for=20 the Oracle processes:

SQL> connect scott/tiger
ERROR:
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux Error: 12: Cannot allocate memory
Additional information: 1
Additional information: 491524

SQL>

Using Large = Memory=20 Pages (Bigpages)

This feature is very useful for large = SGA sizes.=20 In the following example I will show how to use and configure Linux = bigpage=20 memory area for System V shared memory segments. System V shared memory = segments=20 are allocated for SGA if "shmfs" is not used or = configured for SGA.

A separate Linux memory area can be = allocated to use=20 4 MB memory pages rather than the normal 4 kB pages. Large memory pages=20 "bigpages" are locked in memory and do not get swapped out. This means = that a=20 whole separate pigpage memory area can be allocated for the entire SGA = not to=20 get swapped out of memory. This means that it is very important that = the=20 bigpage memory area is only as large as needed for SGA because unused = memory in=20 the bigpage pool won't be available for other use than for shared memory = allocations, even if the Linux system starts swapping. It is also = important to=20 be aware that if bigpages is set to a high value, then the available = memory for=20 user connection will be low. Using bipages also increases TLB (Translation=20 Lookaside Buffers) cache hits which makes the CPUs to run more = efficiently=20 in particular with large memory configurations.

Sizing Bigpages

Oracle says that the = maximum=20 value of Bigpages should be:

Maximum value of =
Bigpages =3D HighTotal / 1024 * 0.8 MB

The bigpage memory area is only available for shared memory. So if = bigpages is set to a high value, then the available memory for user = connection=20 will be low. If the memory consumption for the maximum number of user=20 connections is known, then Oracle says that bigpages can be calculated = as=20 follows:

Maximum value of Bigpages =3D =
(HighTotal - Memory required by maximum user connections in KB) / 1024 * =
0.8 MB

According to Oracle's white paper L= inux=20 Virtual Memory in Red Hat Advanced Server 2.1 and Oracle's Memory Usage=20 Characteristics, the assumption is that 20% of memory is reserved = for kernel=20 bookkeeping.

The value for "HighTotal" can be obtained with the=20 following command:

grep HighTotal =
/proc/meminfo

Note that highmem is all memory=20 above (approx) 860MB of physical RAM. This means that "HighTotal" is the = the=20 total amount of memory in the high memory region. It should now be clear = that=20 large memory pages should only be configured if enough physical RAM is=20 available. For instance, if the server has only 512 MB RAM, then = "HighTotal"=20 will be 0 kB. And on my 1 GB RAM desktop PC, "HighTotal" shows 130992 = kB.=20

Here are a few examples for bigpage sizes taken from Tips = and=20 Techniques: Install and Configure Oracle9i on Red Hat Linux Advanced = Server:=20

2 GB SGA    2100 MB bigpages
4 GB SGA    4100 MB bigpages

The bigpages feature allows a maximum size of 5.4 GB SGA on a = machine with=20 8 GB RAM.

Configuring=20 Bigpages

The kernel needs to be told to use the bigpages pool = for=20 shared memory allocations. The bigpages feature can be enabled for = System V=20 shared memory in the proc file system without reboot with the following = command:=20

su - root
echo "1" > =
/proc/sys/kernel/shm-use-bigpages

Alternatively, you=20 can use sysctl(8) to change it:

sysctl -w =
kernel.shm-use-bigpages=3D1

To make the change=20 permanent, add the following line to the file /etc/sysctl.conf. = This=20 file is used during the boot process.

echo =
"kernel.shm-use-bigpages=3D1" >> =
/etc/sysctl.conf

Setting=20 kernel.shm-use-bigpages=3D2 will enable bigpages for "shmfs" which I'm not = covering=20 in this article. Setting kernel.shm-use-bigpages=3D0 will = disable the=20 bigpages feature.

The kernel needs to be told how large the = bigpage pool=20 should be. If you use GRUB, add the "bigpages" parameter in the=20 etc/grub.conf file and set the maximum value of bigpages as = follows. In=20 this example I will set bigpages = to 2100 MB=20 for the SMP kernel 2.4.9-e.25 that is started on my database server: =

default=3D1
timeout=3D10
splashimage=3D(hd0,1)/boot/grub/splash.xpm.gz
title Red Hat Linux (2.4.9-e.25enterprise)
        root (hd0,1)
        kernel /boot/vmlinuz-2.4.9-e.25enterprise ro root=3D/dev/hda2 =
hdc=3Dide-scsi
        initrd /boot/initrd-2.4.9-e.25enterprise.img
title Red Hat Linux Advanced Server (2.4.9-e.25smp)
        root (hd0,1)
        kernel /boot/vmlinuz-2.4.9-e.25smp ro root=3D/dev/hda2 =
hdc=3Dide-scsi bigpages=3D2100MB
        initrd /boot/initrd-2.4.9-e.25smp.img
title Red Hat Linux Advanced Server-up (2.4.9-e.25)
        root (hd0,1)
        kernel /boot/vmlinuz-2.4.9-e.25 ro root=3D/dev/hda2 =
hdc=3Dide-scsi
        initrd /boot/initrd-2.4.9-e.25.img

After this change the system needs to be rebooted: =

su - root
shutdown -r now

After a system reboot, the "MemFree" = value (free=20 system memory) in the /proc/meminfo is subtracted by 2100 MB in = this=20 example. The 2100 MB show now up in the "BigPagesFree" which means that = 2100 MB=20 are now in a separate allocation area:

grep =
MemTotal /proc/meminfo
grep BigPagesFree /proc/meminfo

Note that if you configure=20 "bigpages" in the etc/grub.conf file and reboot the system,=20 "BigPagesFree" in /proc/meminfo will be 0 KB if "HighTotal" in=20 /proc/meminfo is 0 KB and if = /proc/sys/kernel/shm-use-bigpages=20 is set to "1".

Making=20 Other Performance Related Changes

Disabling Unneeded = Background=20 Processes

Disable or remove slocate from your = system. The nightly slocate cron job can become a real=20 performance killer for your database!

Every night=20 slocate updates a database for files on your system which match = a given=20 pattern. It helps you to find files on your system very quickly. = However, when=20 the cron job is run at night, it will flush the buffers, it can fragment = your=20 memory, and it could cause your system to do heavy paging.

The = easiest=20 way is to disable updatedb in = /etc/cron.daily/slocate.cron or=20 to remove slocate from your system completely:

su - root
rpm -e slocate

X should not run unless you need to. You = can stop=20 X by switching to runlevel 3 with the following command:

init 3

To switch back to runlevel 5 = that X comes up again, run:=20

init 5

To set the default = runlevel permanently to 3 so that X=20 doesn't come up with the next reboot, change the following line in /etc/inittab:

 id:5:initdefault:

so that it reads:

 =
id:3:initdefault:

You can = check=20 for other unneeded background processes by running the command:

/sbin/chkconfig --list

To temporarely = disable e.g.=20 ypbind, run:

su - root
service ypbind stop

To permanently disable ypbind, run:

chkconfig ypbind off

Oracle = Errors and=20 Problems

The intention of this section is to describe = errors and=20 problems that can occur in connection with the changes covered in this=20 article.
For errors regarding the installation of Oracle software = and=20 regarding the creation of a database, see Oracle = Installation=20 Errors.

ORA-3133 errors and attach = errors

Cause(s):
- Running an Oracle binary that has = a lower=20 SGA base, but /proc/proc/<pid>/maps has not been adjusted = as=20 well.
- SHMMAX value has not been increased large enough. =

SQL> startup
ORA-03113: end-of-file on communication =
channel
SQL>

Cause(s):
- A too large SGA has been configured
- SHMMAX = value has=20 not been increased large enough.
- Oracle has been relinked with a = lower SGA=20 base address but "mapped base" has not been lowered for the shell at the = Linux=20 OS level.

SQL> startup
ORA-27102: out of memory
Linux Error: 12: Cannot allocate memory
Additional information: 1
Additional information: 262148
SQL>

Cause(s):
- This error message comes up if the SGA size if too = large.

ORA-01041: internal error. =
hostdef extension doesn't exist

Cause(s):
- If this error comes up and the database is not up, = then=20 remove all shared memory segments from the Linux OS.

Useful Linux=20 Performance Utilities

top=20 Utility

This utility shows CPU consumption, memory = consumption, and=20 "top" sessions on the Linux server:

top

Load Averages:

The = first line of the=20 top output shows you a series of three "load average" numbers. = These=20 numbers describe the load on the system. The load average is the average = number=20 of processes that are waiting in the queue for CPU time (including = processes=20 that are waiting for I/O) for the past 1, 5 and 15 minutes.

For = example,=20 if a process is sleeping in an uninterruptible state, it will count as a = load of=20 1. Or if you run 3 non-interactive processes that are not waiting for = input,=20 then you can expect the average load to be 3. To illustrate that, run = the=20 following command in 3 different shells on a server that is not being = used:

  while [ 1 ]; do str=3D"x"; done

This loop = will use up all=20 the CPU time that it can get. Now wait for about 2-3 minutes and you = will see=20 that the average load for the last 1 minute will increase to be 3 and = higher. It=20 will be a little bit higher than 3 since there are other processes = running on=20 the system.

In general, a number less than 1 is ideal. A load = average=20 value of 3 is high. And a value of 10 is definitely a heavily loaded = system=20 where you can expect delays.

You can also use the tload = command=20 to display real-time text mode graph on the "load average". =

CPU=20 States:

It shows the load on each processor - the percentage = of CPU=20 time in user mode, system mode, niced tasks, and idle.
The "user" = percentage=20 shows how much processing time the CPU is spending on user processes, = and the=20 "system" percentage shows how much processing time the CPU is spending = in the=20 system (kernel). Niced tasks are only those processes whose nice value = is=20 negative. And note that the processing time for niced processes will = also be=20 counted in system and user time, so the total will be more than=20 100%.
However, the best indicators of a stressed CPU is the load = average=20 which I described above.

Sessions:

This section = shows the=20 top sessions (Linux processes) in terms of CPU utilization. =

sar Utility

"sar" stands for = System Activity=20 Reporter.

CPU Usage:

To check CPU usage over time, = run:=20

sar -u

This command is useful if = you want to see overall CPU=20 consumption over time.
%user shows the percentage of CPU = utilization=20 at the user level (application).
%system shows the = percentage of CPU=20 utilization at the system level (kernel).

To check CPU usage 10 = times=20 with a time interval of 3 seconds, run:

sar =
-u 3 10

Swap Activity:

To check swap=20 activity over time, run:

sar =
-W

This command is useful if you suspect memory shortages.=20
pswpin/s shows the total number of swap pages the system = brought in=20 per second.
pswpout/s shows the total number of swap pages = the=20 system brought out per second.

These numbers should be low. If = not, you=20 need more RAM.

To check swap activity 10 times with a time = interval of 3=20 seconds, run:

sar -W 3 10

I/O = Activity:

To check physical disk=20 I/O activity over time, run:

sar =
-b

This command is useful if you suspect that the database=20 is I/O bound.
See manual pages for more information.

To = check I/O=20 activity 10 times with a time interval of 3 seconds, run:

sar -b 3 10

vmstat=20 Utility

This utility provides a report that covers process = activity,=20 paging, memory usage, disk I/O, and CPU usage.

To create 5 = reports with=20 a time interval of 3 seconds, run:

$  vmstat =
3 5
   procs                      memory    swap          io     system      =
   cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  =
sy  id
 0  0  0 186460   7416   9424  45272   1   4    25    35  126    33   3  =
 0  96
 0  0  0 186460   7416   9432  45272   0   0     0    17  103    18   0  =
 0 100
 0  0  0 186460   7288   9440  45272   0   0     0    73  104    23   4  =
 1  95
 0  0  1 186460   7288   9440  45272   0   0     0     5  102    12   0  =
 0 100
 0  0  0 186460   7288   9440  45272   0   0     0     8  102    14   0  =
 0 100

See man pages for more information.

Oracle = Linux=20 Management

Determining Which = Semaphore Sets=20 and Shared Memory Segments Belong to Each Oracle Database or=20 Instance

When Oracle hangs or crashed or when Oracle was = killed, then=20 sometimes you will see that shared memory segments and/or semaphore sets = have=20 not been released or removed by the Oracle background processes. It is = important=20 to make sure that the semaphore sets and shared memory segments are = released at=20 the Linux OS level before the database or instance is restarted. =

Running=20 ipcs will only show you which semaphore sets and which memory = segments=20 are owned by the Oracle user account. If you have only one database = runnning on=20 your server, then you can simply use the IDs of all shared memory = segments and=20 semaphore sets that belong to the Oracle user account and release them = via=20 ipcrm:

$ su - oracle
$ ipcs

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 0 root 600 196608 2
0x00000001 32769 root 600 655360 2
0x00000000 458755 oracle 660 4194304 0
0x00000000 491524 oracle 660 33554432 0
0x00000000 524293 oracle 660 33554432 0
0x00000000 557062 oracle 660 33554432 0
0x00000000 589831 oracle 660 33554432 0
0x00000000 622600 oracle 660 33554432 0
0x00000000 655369 oracle 660 33554432 0
0x00000000 688138 oracle 660 33554432 0
0x3ecee0b0 720907 oracle 660 4194304 0

------ Semaphore Arrays --------
key semid owner perms nsems status

------ Message Queues --------
key msqid owner perms used-bytes messages

$

To release all shared memory segments that are owned by the Oracle = user as=20 listed above, run:

$ ipcrm shm 458755 491524 =
524293 557062 589831 622600 655369 688138 720907

The=20 command for releasing semaphore sets is:

$ ipcrm sem <semid>...

But if you=20 have more than one database or instance running on the Linux servers, = then=20 ipcs will NOT show you the = semaphore sets=20 and shared memory segments that are owned by each database or instance. = The=20 following steps can be used to find the right IDs for each database or = instance:=20

$ su - oracle
$ sqlplus /nolog

SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL> select value from v$parameter where name =
=3D 'user_dump_dest';

VALUE
-------------------------------------------------------------------------=
-------
/opt/oracle/admin/test/udump

SQL>

On my test server, the oradebug = ipc=20 command created a file called test_ora_6626.trc in the=20 USER_DUMP_DEST directory /opt/oracle/admin/test/udump. = The=20 name of the created trace file is = $ORACLE_SID_ora_<pid>.trc where=20 <pid> stands for the process ID of the Oracle foreground process = in a=20 non-MTS environment that's talking to sqlplus here. If you are = not sure=20 about the name of the file that was created, run ls -lrt to see = the=20 timestamp of the latest trace file created in the = USER_DUMP_DEST=20 directory.

When you open the trace file (in my example=20 test_ora_6626.trc), you can find the semaphore ID for this = database=20 after the line "Semaphore List=3D". Here are the semaphore sets on my = test box for=20 the Oracle database:


/opt/oracle/admin/test/udump/test_ora_6626.trc:
[SKIP]
Maximum processes:               =3D 150
Number of semaphores per set:    =3D 154
Semaphores key overhead per set: =3D 4
User Semaphores per set:         =3D 150
Number of semaphore sets:        =3D 1
Semaphore identifiers:           =3D 1
Semaphore List=3D
98304
-------------- system semaphore information -------------
------ Shared Memory Segments --------
[SKIP]

To release all semaphore sets that are owned by the = database as=20 listed above, run:

$ ipcrm sem =
98304

And here are the=20 shared memory IDs on my test box for the Oracle database:


/opt/oracle/admin/test/udump/test_ora_6626.trc:
[SKIP]
 Area #0 `Fixed Size' containing Subareas 0-0
  Total size 000000000006e078 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      0        0  1671186 0x00000050000000 0x00000050000000
                              Subarea size     Segment size
                          000000000006f000 0000000000400000
 Area #1 `Variable Size' containing Subareas 1-7
  Total size 000000000e000000 Minimum Subarea size 01000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        1  1703955 0x00000051000000 0x00000051000000
                              Subarea size     Segment size
                          0000000002000000 0000000002000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        2  1736724 0x00000053000000 0x00000053000000
                              Subarea size     Segment size
                          0000000002000000 0000000002000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        3  1769493 0x00000055000000 0x00000055000000
                              Subarea size     Segment size
                          0000000002000000 0000000002000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        4  1802262 0x00000057000000 0x00000057000000
                              Subarea size     Segment size
                          0000000002000000 0000000002000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        5  1835031 0x00000059000000 0x00000059000000
                              Subarea size     Segment size
                          0000000002000000 0000000002000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        6  1867800 0x0000005b000000 0x0000005b000000
                              Subarea size     Segment size
                          0000000002000000 0000000002000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        7  1900569 0x0000005d000000 0x0000005d000000
                              Subarea size     Segment size
                          0000000002000000 0000000002000000
 Area #2 `Redo Buffers' containing Subareas 8-8
  Total size 00000000000a3000 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      2        8  1933338 0x0000005f000000 0x0000005f000000
                              Subarea size     Segment size
                          00000000000a3000 0000000000400000
 Area #3 `skgm overhead' containing Subareas 9-9
  Total size 0000000000001000 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      3        9  1933338 0x0000005f0a3000 0x0000005f0a3000
                              Subarea size     Segment size
                          0000000000001000 0000000000400000
[SKIP]

To release all shared memory segments that are owned by = the=20 database as listed above, run:

$ ipcrm shm =
1671186 1703955 1736724 1769493 1802262 1835031 1867800 1900569 =
1933338

To=20 verify if the shared memory segments and semaphore sets have been = released, run:=20

$ ipcs

Hardware=20 Recommendation

It really depends on what kind of database = you=20 want to setup and run, how large the database is etc.

But people = keep=20 asking me what I would recommend. If you want to get a feeling how well = Oracle9i=20 (non-RAC system) runs on Linux/Intel systems, and if you don't want to = spend=20 "too much money", here is what I would buy:
  - 2-way = server,=20 2.4GHz Xeon
  - 4 GB RAM; RAM is cheap and gives you = usually the=20 biggest "bang for the buck".
  - Large Internal Ultra SCSI = disks=20 with a hardware RAID controller card

References

Oracle's Linux = Center=20
= An=20 Overview of Red Hat Advanced Server V2.1 Reliability, Availability, = Scalability,=20 and Manageability (RASM) Features
L= inux=20 Virtual Memory in Red Hat Advanced Server 2.1 and Oracle's Memory Usage=20 Characteristics
Tips = and=20 Techniques: Install and Configure Oracle9i on Red Hat Linux Advanced = Server=20
Oracle9= iR2 on=20 Linux: Performance, Reliability and Manageability Enhancements on Red = Hat Linux=20 Advanced Server 2.1
Delivering Leading TPC-C Figures with Red = Hat Linux=20 Advanced Server (Red Hat Webcast Tuesday, 22nd October, 2002)
Understandin= g the=20 Linux Kernel, 2nd edition

Copyright Notice
This article may not be published, sold, = reproduced or copied in whole or in part without obtaining permission = first. But=20 you are welcome to put links from your site to the article.=20

The information provided in this article shows how I = tuned=20 Linux/Oracle on my server(s) and is distributed AS IS. Every effort has = been=20 made to provide the information as accurate as possible, but no warranty = or=20 fitness is implied. The use of this information described herein is your = responsibility, and to use it in your own environments do so at your own = risk.=20

Comments? =20 webmaster_at_puschitz.com

------=_NextPart_000_000A_01C473D4.58079750 Content-Type: image/jpeg Content-Transfer-Encoding: base64 Content-Location: http://www.puschitz.com/images/ocp_225x37.jpg /9j/4AAQSkZJRgABAgAAZABkAAD/7AARRHVja3kAAQAEAAAAUAAA/+4ADkFkb2JlAGTAAAAAAf/b AIQAAgICAgICAgICAgMCAgIDBAMCAgMEBQQEBAQEBQYFBQUFBQUGBgcHCAcHBgkJCgoJCQwMDAwM DAwMDAwMDAwMDAEDAwMFBAUJBgYJDQsJCw0PDg4ODg8PDAwMDAwPDwwMDAwMDA8MDAwMDAwMDAwM DAwMDAwMDAwMDAwMDAwMDAwM/8AAEQgAJQDhAwERAAIRAQMRAf/EAKUAAQACAgMBAQAAAAAAAAAA AAAGBwUIAgQJAwEBAQACAwEBAQAAAAAAAAAAAAAEBQIDBgEHCBAAAAYBBAIBAgMFCQAAAAAAAQID BAUGBwAREgghEyIxFEEVF1EWN3cJYaEyQiPUlle3EQACAQMDAgEIBgcJAQAAAAABAgARAwQhEgUx BkFRYXGhIjITB4GRsdFSkvBC0iNjFBXhgqKyQ1OTRFUW/9oADAMBAAIRAxEAPwD380iNIjSI0iNI jSI0iNIjSI0iNIjSI0iNIkbtVshKXGNZiwOTtWDyYhoJuqRM6oi+n5NtEME+JAEQA7p4kQTfQoDy NsUBHSJJNIjSI0iNIjSI0iNIjSI0iNIjSI0iNIjSI0iVxlTK1Mw3UXd0vEgdlFN1Ct2rdAntcu3K gGMm3bp7hyOYCiPkQKAAJjGKUBEIuZmW8W2blw6esnyCdD2v2tndyZq4eEu5yKkk0VFHV2PgoqPA kkgAEkCayp9wrc7IVzG9Usrvo9wHsYvSRDgSqpG8kOHBA5fkGw+DCH9o6qhzdw6jHuEeg/dPpLfK PCtnbc5vBVxoR8RdD4jVgdPOB6JMsddsqzb7oxx1caPaMRXSYJ7K/FWtmZoV9vvsRE5wIYDm4jxA xAAw/EphN41vxeZS7cFp0a2x6BhSsp+4flXl8dgNyGHk2MzHQ0drDb9nnYCooK60YkdSANZAmvda SmXc4lU+umQbkxgZRzEO5eEaHfNvuWpgA5BUQRUKU3ExTcRHcAMA/jqMOeZydll2AJFQKiol9d+T VrGS2crlsSw1xFuBbjBG2t0NGYEitRXpUHySXUvtzGTV2rtEvmK7niKVuKotqm6s7E7Zs9chsHpK ZQqZwMYxilLsUwchABEu4b77HNK9xbdy2yFum4dZU8z8p7uNgXc7BzcfMSyK3BZcMyL+KgLCgAJN SDQEgGhlv5tzPWMF0lW5WVNZ8KrlJjDQbQS/cvna24gklyEADiQpjmEfoUB+o7AM3Pzkw7e99fAD xJnJdmdnZfdOeMPGIWilndvdRB4mnlNFA8SfAVIyGIcsVjM9Dir9VznSYvuaL6PcCX3snaI7LN1g KIgBi7gYB/zFEpvobWWFmJl2hcTofUfJI/dvauX21yL4GVQstCGFdrqfdZa+B6eZgV6ia+zHc+Cc T0xC4txVdsyt6+uLWXsFYjzuGBVgEQ4pqJlVMcPA7GEpQN9SiYPOq1+dUuVs23uU6lRpO+xPk7kJ jW73J5uNhG4Kql5wtynnBKgeipI6NQ6TL1/s9Z5Vnb5Sb683qkQ1LrUpZJKWsDczBFRONbmX+2RM uimB1VRLxAAHwG5h8F1nb5Z2DFrLqFUsSdOg6emRM/5aYlh7FqzyuNfuX71uyq2mFwg3GC722saK vUnx0A1M2AxlemmTaFV76xYLRjS0MivUGC5inUSKYxi8TGL4Efj+GrLEyBkWluAUDCs4PuThH4Tk b2C7BmtNtLDQH0VmBzBmij4QrSVkurpwJXq/2kNDMEgXfP3G3L1N0hMQB2DyJjGKUPG47iADrzc6 1iJvufQB1Pok7tLs7kO58o42Go9kbndjtRF8rGh+gAEnwGhmu5e311VKVVv1NywsgoAGRWCIc7HI Pkpg2biHkPPgdVn9aueGPc+o/dPoB+UuApo3OYII6j4i9fzS56vmiWnLNQqvL4qs1XeXiCXnVXLx MBRiiJmWBNu+NxL61TFSATE+pBUTKYNzeJ1nOZ3RGtspYV18Ouh8/wBlROO5Ps6zi4mVlWs2zdWx dFuinW7ULVrfWqAsaN0YI7LoNfj2W/h1XP5q4n/9Dr2rGcNL/wBInl9mXM11bdi8s1iS7KoddK/i iMoLrHMC6imUi1tilgdq/mZnLVZE754UBTBoBWahPWJgUEQ4H5Ily98ch3nHOLqA+oNtd0uTsWSI KAlJlk5i2S35e9SdiumV3MpqskNxTKPsWDiXbcR230iWVSLLOB1fUtI3BxaLIzqcy9C2rv4iWXO9 bFdGATu4UhY9YyChOG6JePx2N8gNpE0q/p8Zq7CX28r17M9vlZWKncTQOQK7G2IkIo7eKSj9dqeT jVoZBIU2Qghw9DoRWKfYRANx3RLyvkZlge42OqhFdgbfB0S61yauL6ltWsKZoiauuohuVgkotHnX 9DkHSgqiZQT7j8Dl0iW/ly7WmvZ36nVSGl1GNeyBYbWzuMaUiZivkI+rSD9sQ5jkMYvrcIkUDgIb iGw7h40ia89vrfncuecEYww1ZLNGIXCuWmUnYerOIFm8dHizsfSp9zPtnCBSpgsbcobCO/jSJc/b e93PGePMayFPsS8XKSeT6RXpaSBNE6jmPkZNJu8SOB0zFD3JiICJQAQ3+O2kSF9+Mi3zHVCxCtQL i8o7225Rja9OS7F3ER6x45eImHKiIPJ1NVihuq2TNzVAA+O2/nSJZ4WeyRPUeduTe2OZm2xWN5eX a25d3FybgZBvHuF01jOYsgR65k1CgHJEvrHj9NImqnSHO2RLxkGDo01mH9dIGbwjXci3CYVbRwOK pbJRZEi0Co5i0UExBRNU6gJLgZYnr8iAb7onqLpEaRGkTRbutHv45TB2UHEM4sVIxbbySV7iG6YK 7NVTtzEcHTEBASp+kxfPjc4APgdw5/nlK/CvEVRGqw82mvq9c+3fJrIt3hyXGLcFvJy8cpZYmntA NVAfK24HTWikjpLbadueuDxq3dp5Yh0iOEyqFScAuiqUDBvsdNRIpiiH4gIb6mLzWGRX4gnKXflR 3TbcocG4SDTTaR9BBoR5xNXc95LpPYe/4Mx5hh1++Nsgbg0sMhbmCKwN4eOb7C45ODFL4MPBQ3Hf YUyl35iBdVPI5VvOu2rVj2mDA1H6o8dfX9E+mdi9uZ/Z/G8lyHML8Gxcx2tLaYjdeuN7vs1PT2lF fB2PugmdPryzzAthrMb3Cs1GsLbGZZsawQ0q2TWbySf2sduiVU4h6VA2HgYfiIjxNsA8i48YuQce 6bBAYXG0I66D6pu7/vcIvO8enM23aw2BYG5GKtbO677VB7y/iHvDqtfdMZw/JZA7I5Qrr/Ot/h62 8wFMHnRxiDI8ZLfdMzJqA4cAqRMgIpqJkE4gc4gACUxScwNrVhNdz76nIcA2jXbSjVHifNLLu7H4 3sviLtvg8W5dXkLYt/zG4XLW1wRtXaSd7KTtG1akghm2lZ33dtsedMsvM+scey+S8QYTkyw2OqlF cffIyJh5KSpUVSj7CpnBNQwAAGKAoeB4K6yN58y+ckIXt2zRQPE/i/TzeQzRa4rF7W4ZeBfKt4uf nJ8S/df3bdvws7gfZLDcoJqD+9/Ekr2Lu9rxPkrICi2ObJhvEef0nUT6rC3Mg2iLA+aqlQdoLFKU pEyrGMYwBtxSEfA+ouoyZFzGvP7DW7d2o1/VYjr9fq9Ev8rhcPn+KxQMuzm5/HlX/dHc13HRl3Iy mpLFQACergaje0vTqhnzEmLMWtsU5FkUsY3ykP36FmipZFVEXKyzlRUHBVQIYph4HKQQEdw4+A4c R1YcPyOPjWPg3TsdSag+nr+n2Tifmn2JzXO8u3KcehysW+qG2yEHaAoXbSoI1BbQU9rU7t0ujJuf sN33FGX61TsgRlhnXlBs6jeMZiodQxEYlyoobyQAAClAR3Ef79Tsvkce9YuIjgnY2n90zje2uxOc 4nmcDJzMV7dpcqwCzUpU3UA8fEyDdcOx+Dapg3GlcseS4eInIiHIhJRq5zgoioBziJTABBDfYdaO L5TGt4yKzgEDUS7+Yfy87gz+4MzIx8O49t7hKsAKEUGo1lBdvbvXcjWXE2Ssc5FM7qWO34s7Va4J maSPW3Mgqmo1fGbnFEDCp9ubj8w8pgADyEoDW81kJfe3dtP7KHUgV2k9DT6PVO7+U3C5XDYudxvI 4lL+Su63buN8P+ZW2CHthhupt3iuh0c+AanYSsJVkk1Sf1KTgVUoHKB6+YhgAwbhyId8Bij+0BDc Nei7X/vf4f7Zg3HlSQe0On8avrFuh+ielWMJiFnKHWnsBblb7HEaFbBclt+cis3/ANJdwYRKUNzK FNvxDiA+A+muqxHV7SlW3CnXy+efnLuXEv4vI3kv2Bjvur8IdLYb2lXx6KR11PU9ZXvZb+HVc/mr if8A9Dr2pEopf+kTSjKfZ3B1PzjD49vOMJmbtFck6zHNslBERL2PhXlvUWJEgVdZ4Egn7FGxwMZB sYCiHkfIaRPw3bXDd5yuhgmw40sj1R1dZakxlgmY2IewLifgUDLuCJpkkHLwpQS3EiqjQpfPkxfO yJzpHbXEErOVmjwWOrFXsY3SzSdFx7ko0XHt6jMTrM7n7pi1Ig6M4IVY6KoJHUakIscqgAPxERRM bUe3eHZL9W52AxJZIuNwDAWok/ZiNK2kJmFOUML5hHoITBnwEOdPdEFEEkRHbkYgiGkTMj3Bxcni We7EWHHtlr8JU1GMZGi9JX3cu9GbVbkRQZGjpd4kkCh1EuZXC6O3gx/Ab6ROVm7cY+r2LqpmG54v tkLJyNxLSKdU5ZlGIyak68SWKBmL9d6WOFoskmoIPCO/SdMDfIR+OkSYxXYOhzdh67pSuO7TW7hn kLK0pCFgiGzORh/yBoo9kE5D2OBVRIum33RM39pFgEhwH1mKfSJCqR29xZl+y0CuqY8s7WpZQkZR LDmRbBHx5oKwP64KijgzQCO13SBgBE5253DdP2AURIPIAAUSO0vur12z9E19AtPmZUkvkxrjttW7 HFRqqrWVdMJB21k1UjO10waKEj3SRFSGMpzIcnr25DpEtuodgMQWTM136s1+HctpugxJlZAgsWqV edJgmzO8YMjkVH2qtiSSHvSFEoF57edh0iTXBVsx9fsfNbrjSqhUK5MSkwzGMMxaR6xnMLJOYdwo okzOomPJVkYSDyERJx34juUES4tIjSI0icFE01UzpKkKqkqUSKJnADFMUwbCAgPgQENCKz1WKkEG hEq9fBeEnSx3DnDtHcLqjuqurXo05zD+0TGbiI6hnj8Y6m0n5R906ZO9+ftqFXPyQB4C9cA/zSW1 ulU2moqt6fUoWqN3GwroQ7BuxIfYREORW6ZAHyI/XW+1Yt2tEUL6AB9kquR5nO5Jg2XfuXiOhuOz kejcTO1A1it1ZB41rFejK41kXaj+QbRbRFmmu7WApVHCpUSEA6hwIUDHH5DsG4+Ne27SWwQigVNd BTXyzVncllZ7K2Tde4VUKpdi5VBWigsTRRU0UaCpmFlsa45n5dWwTtArc1PLNzNFpt/FM3Ls7c6R kDImXVSMoJDJnMQSiOwlES/QdYPi2XbcyKT5SBX65MxO4+UxLIx7GVeS0DuCLcdUDAhg20ECoYBq 0rUA9ZnoGvQFWi28JWIOPrkK0E4tYiLbJM2qQqHFQ4kRRKQheRjCYdg8iIjrZbtJbXagAHkAoJBz s/Jz7xvZNx7lw0qzsXY0FBVmJJoAANek+dhq9at0cMPbK9GWeJMoVY0XLNEXrYVCb8DikuQ5ORd/ A7eNeXbSXRtdQR5CKzLj+Ty+Pu/Gxbr2rlKbkYo1D1G5SDQzAzeLsZ2X7T948dViwfYIkbsfzKIZ O/SkkXimmn7kjcSlL4AA8AGtdzEs3PeRTTygGT8LublsLd/L5d63uJJ2XHSpOpJ2sKknqT1nyhcT 4sraj5au41qsArJs1Y+SVjoZi1M4aLbe1usKKJROmfiHIhtyjt5DXlvDsW67UUVFDQAaeSZZndPL 5oUZGZfuBGDLvuu21x0ZdzGjDwYaiYX9BMF/9L0T/jkZ/t9Yf07F/wBpPyj7pM/+67h/9DK/57v7 Ul8FRKRVox9C1mmwdch5MxjyUTFxzZm2cGOQEzGWRRTIQ4iUAKImAfHjW63j2ralUUAHqAABKnO5 zkM66t7JyLty4vus7s7LQ19lmJI1108ZE1MEYPVUOqrhqjKKqGE6ih67GCYxhHcRERb7iIjrSeOx j/pJ+UfdLVe+O4FAA5DJAH8a5+1LHi4uMhI9nEQ0c1iIqOSKhHxjJEjdugkQNippJJgUhCgH0AA2 1KRFQBVFAPATncnKu5V1rt52d2NWZiWZifEsaknzmUl2RSVWx5XSIpHVOXKWKVBIQomECJ5Br5zm 2D8ClAREfwAN9ZTRL80ia4E6xY4e56tmf7VCxNys8wyrrapJy0W3cKV5aBK7Azli4VFQxVHAuCGE SlKJRSKICI/REreI6VVKByvH5miZxBlfm2T5+/v7ElEpJvnsbPsjs1q8u6TXKodunzFQhjiYANvs mHI26J0qr0rb1qeoLFTKspKYcxTepDIuOcSHjGaQMJl6o8XTKrKkH3LINln650k+BR+Wxzn20iZi s9LMe1jHeeqm0/LDXLPCN2ZTGVyQbRKbbMLmZU52QuCm9y6LU5ymKQyoFMJCjsXxsiRKt9JH8DhS 2YY/VKKbR8+/iJBk+gKJBQbUTxhie1KXjmgemWTeETIRYHA8jAX/ABaROnH9Co2GwwbEsPkozUHW QnGQpQi9ej3VacqukgQUiDVlUwtgjwKQpiIgpuRUPYBvPHSJMca9MYLGZusn5bepCRT63vbhINUX LRIpZRW4NHDZdMhU1ClZothcCKSZAP8AEAKI77mFExmLulDfHFlxSdxleWs2NcCvpyRw3jpeOaNj xq02CxDffSiQiu9K3TcKgmAlJ5EBOJgDYUSG03+nfWKVYuvdojskSIyeD5KSfypCsCJoWMHMtKSr ArlMHAgidiMw6SIp8xMU/wBC/TSJLKL0ahKJkWkZfZ5PscjkmBt1mtNvlXZ1lI2bC1guSRbEiTOT NWImIZuX2Il5D9umJgEQLxRLR694HuuCk5OvucwqXbHRnEu8rVKVgGkeaNdTMqtLLqC/SVUXX2Uc qlAp/Gxvw2ANImzmkRpEaRGkRpEaRGkRpEaRGkRpEaRGkRpEaRGkRpEaRGkRpEaRGkRpEaRGkRpE aRGkRpEaRGkT/9k= ------=_NextPart_000_000A_01C473D4.58079750--