Gopi Desaboyina Solaris Blogs

Just another weblog

Increasing Inodes on UFS filesystem

Weblogic Installer on Solaris fails with following error

WARN  [WizardController] com.bea.plateng.common.util.diskspace.FileSystem – Unsatisfied link error…….
WARN  [WizardController] com.bea.plateng.common.util.diskspace.FileSystem – Unsatisfied link error…….
ERROR [Thread-4] com.bea.plateng.wizard.installer.utils.InstallFiles$ExtractFiles – Error encountered during file installation.
com.bea.plateng.common.util.crlf.CRLFException: /Weblogic/wlserver_10.3/server/lib/consoleapp/consolehelp/en-us/taskhelp/channels/ConfigureListenAddresses.html (No space left on device)

    at Method)

1. Check if you’ve sufficient space on that mount point using df command Ex: df -h /Weblogic
2. If it shows up free space. But still you get that error. You could be running out of Inodes on that filesystem.

    Check them using df -k -o i /Weblogic.

    Below is  the o/p my from my case.

                 Filesystem         iused     ifree     %iused     Mounted on
        /dev/md/dsk/d7     10752         0     100%     /Weblogic

  see from above that almost Inodes are consumed on this filesystem.

3. So How can we resolve this issue ? 
 a) Check if you’ve any unwanted files & hard links have been there on filesystem.If necessary cleanup files.especially if there is no cleanup,log files might filed up from so long. 
 b) Increase the Inodes on filesystem. 
    –> You can’t increase the inodes on filesystem without destroying the filesystem. So full backup required. 
    –> check how the filesystem was created using
           # mkfs -m /dev/md/dsk/d7
            mkfs -F ufs -o
            /dev/md/dsk/d7 20481600

From the above output you need to look at nbpi ( Number of Bytes for Inode). This gets calculated automatically based on the disk size while doing newfs. If you need more Inodes. you need to reduce the size of this value which will increase the inodes available. Below are the default nbpi values
            Disk size                   Density ( Please read man newfs for more details)
            Less than 1GB             2048
            Less than 2GB             4096
            Less than 3GB             6144
            3GB to 1 Tbyte            8192

So to increase the inodes after full backup. run newfs -i nbpi=xxxxxx ( The lesser the value. More Inodes ). Once filesystem is created restore the data from backup.


April 26, 2011 Posted by | Solaris | 3 Comments

Installing Locale in Solaris 10.

Installing Japanese fonts/Locale in Solaris 10.
1. Check what all locales you’ve installed using locale -a
2. localeadm -lt –> Check if you’ve Japanese there. If not you can directly install from DVD localeadm -a
3. localeadm -v -a ja -d /<DVD Mount Path/Solaris_10/Product>
4. localeadm -v -r ja –> This one removes the Japnese Packages ( Only Locale Related Packages).
5. set the LANG & LC_ALL variables then open gedit or appropriate program to check fonts.

Check if your installation supports Japanese Region ?

# localeadm -lt
You do not appear to have created a fresh config file since you began using this application.
If you have a set of Solaris install images available to you, it is recommended that you do so before proceeding.

Do you wish to create a new config file? [y/n]: y

Please select the option that was used to install Solaris

1.  CD installation/net installed CD images
2.  DVD installation/net installed combined image

Please enter your choice:
The following regions are supported by localeadm:


Australasia (aua)
[ Australia, New Zealand ]

Central America (cam)
[ Costa Rica, Guatemala, Nicaragua, Panama, El Salvador ]

Central Europe (ceu)
[ Austria, Czech Republic, Germany, Hungary, Poland, Slovakia, Switzerland (German), Switzerland (French) ]

Eastern Europe (eeu)
[ Albania, Bosnia, Bulgaria, Estonia, Croatia, Kazakhstan, Lithuania, Latvia, Macedonia, Romania, Russia, Serbia, Slovenia, Turkey, Ukraine ]

Middle East (mea)
[ Saudi Arabia, Israel ]

Northern Africa (naf)
[ Egypt ]

North America (nam)
[ Canada (English), Canada (French), United States, Mexico ]

Northern Europe (neu)
[ Denmark, Finland, Iceland, Norway (Bokmal), Norway (Nyorsk),  Sweden ]

South America (sam)
[ Argentina, Bolivia, Brazil, Chile, Colombia, Ecuador, Paraguay, Peru, Uruguay, Venezuela ]

Southern Europe (seu)
[ Cyprus, Italy, Greece, Malta (English), Malta (Maltese), Portugal, Spain (Catalan), Spain (Spanish) ]

Western Europe (weu)
[France, Holland, Belgium (French), Belgium (Flemish), Ireland, England, Luxembourg (French), Luxembourg (German) ]

Japanese (ja)

Korean (korean)

Simplified Chinese (china)

Traditional Chinese (Hong Kong) (hongkong)

Traditional Chinese (taiwan)

Thai (th_th)

Hindi (hi_in)


# locale
# ls -ltr /var/adm/messages*
-rw-r–r–   1 root     root      350534 Feb 28 20:11 /var/adm/messages.2
-rw-r–r–   1 root     root      435091 Mar 28 23:05 /var/adm/messages.1
-rw-r–r–   1 root     root      124690 Apr 16 21:04 /var/adm/messages.0
-rw-r–r–   1 root     root       59680 Apr 22 02:59 /var/adm/messages

# export LANG=ja
# export LC_ALL=ja
# locale

# ls -ltr /var/adm/messages*
-rw-r–r–   1 root     root      350534  2?????? 28??????  20:11 /var/adm/messages.2
-rw-r–r–   1 root     root      435091  3?????? 28??????  23:05 /var/adm/messages.1
-rw-r–r–   1 root     root      124690  4?????? 16??????  21:04 /var/adm/messages.0
-rw-r–r–   1 root     root       59680  4?????? 22??????  02:59 /var/adm/messages

List of packages gets installed by localeadm can be found at  “/usr/sadm/lib/localeadm/Locale_config_S10.txt”

April 22, 2011 Posted by | Solaris | | 1 Comment

Solaris-Linux command comparison

From longtime I’m looking for this comparison. Need to do lot of reading on Oracle Enterprise Linux now.

February 3, 2011 Posted by | Solaris | Leave a comment

My Blog 2010 in review

The stats helper monkeys at mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

Healthy blog!

The Blog-Health-o-Meter™ reads This blog is doing awesome!.

Crunchy numbers

Featured image

A helper monkey made this abstract painting, inspired by your stats.

A Boeing 747-400 passenger jet can hold 416 passengers. This blog was viewed about 8,700 times in 2010. That’s about 21 full 747s.

In 2010, there was 1 new post, growing the total archive of this blog to 32 posts.

The busiest day of the year was August 25th with 68 views. The most popular post that day was Discovering new SAN LUNS in Solaris .

Where did they come from?

The top referring sites in 2010 were,,,, and

Some visitors came searching, mostly for symclone, solaris xdmcp, xdmcp solaris 10, ora-04030: out of process memory when trying to allocate, and solaris 10 xdmcp.

Attractions in 2010

These are the posts and pages that got the most views in 2010.


Discovering new SAN LUNS in Solaris April 2009


Enabling XDMCP in Solaris 10 October 2009


EMC TimeFinder/CLONE [ SYMCLONE ] November 2009


ORA-04030: out of process memory when trying to allocate 840 bytes (kgsp-heap,kglss) December 2009


Enabling device Multipathing using Solaris native mpxio April 2009

January 2, 2011 Posted by | Uncategorized | Leave a comment

tcpListenDrop and DTrace to get PID

Yesterday I had strange issue with one of the application box which is causing lot of 
tcpListenDrops. Even after Increasing the tcp_conn_req_max_q & tcp_conn_req_max_q0 has lot
lot of dropped packets. In the process of troubleshooting this issue I learnt
various DTrace scripts and commands.
Below DTrace script will give the PID which is cause for network tcp listen drops.
Using below DTrace you can find which are processes trying to make 
#!/usr/sbin/dtrace -s
#pragma D option quiet
self->connp = (conn_t *)arg0;
self->tcp = (tcp_t *)self->connp->conn_sqp;
printa("PID:%d \ttcp_conn_req_cnt_q/tcp_conn_req_max: %d/%d Number of times:%@d\n",@backlog);
listen system call and arguments.
dtrace -n syscall::listen:entry'{trace(execname);trace(pid);trace(arg0);trace(arg1);} 
You can use below command for finding portwise TCP Q size. 
Note: even though your system has large queue size. 
But applications while starting the listeners they specify how much 
max connections their app listener can handle.
/usr/sbin/ndd /dev/tcp tcp_listen_hash

BTW issue with my application was, It setting backlog Queue size as 2 there are more clients which are trying to connect to listener and which is causing listendrop. anyway I had to kill all client connections & reboot my app..some jackaxx client is making too many cons.

January 8, 2010 Posted by | Solaris | Leave a comment

ORA-04030: out of process memory when trying to allocate 840 bytes (kgsp-heap,kglss)

Recently we fully populated our Sun Fire E2900 from 2 CPU boards to 3 CPU Boards ( 12 Processors & 96 GB RAM).

After that we increased oracle SGA size to more than 12 GB for some of the instances.

Bam..that’s it !!!  They never come up or sometimes they come up and as soon as we issue any sql commands they gets killed. we saw below errors in alert logs

“ORA-04030: out of process memory when trying to allocate 840 bytes (kgsp-heap,kglss)”

Oracle Version is

Googling that error which came with link which I helped us and also confirmed from metalink.

based on CPU count and which leads to error.

As per investigation made in these bugs, the issue is caused by the fact that the granule size automatically chosen at startup is too large and we ran out of granules. It is not factored that we may want to subdivide some of the components into multiple chunks (typically based on directly or indirectly on cpu_count). For example: the shared pool may split into multiple subpools and the buffer cache into multiple working sets. Hence we may choose a granule size which is too large to cater for the fact that some of the components may require multiple granules.


we’ve fixed the issue by using _ksmg_granule_size=16777216 # ( 16 MB ) in init.ora [OR] as per metalink you could apply the patch 8813366 & 7441663.

See MetaLink Note # 831770.1 for details.


Reference Links on Oracle Shared Pool :

December 4, 2009 Posted by | Oracle | | 4 Comments


1. Creating a new session.

#symclone -f <devices file> create -copy/-precopy -sid <symm id> -nop

If you use -copy copying blocks from source to target LUNS gets copied in the background  as soon as activate them.   If you use –precopy, copy will start immediately and won’t wait for activate to be run. Note : This will be full copy from source to target

2. Activating a previously created session.

#symclone -f <devices file> activate -sid <symm id>consistent  -nop

As soon as session is activated block level copy starts if session is created with -copy option.

3. Query the session to find out how is copy going on. You would see status as “copy in progress” or “Copied”.

#symclone -f <devices file> query -sid <symm id> –nop

4. If copying is complete and you want to create a new session. before to that you’ve to Terminate the existing session.

  #symclone -f <devices file> terminate -sid <symm id>nop

All the above steps does are full copy from source LUN to target LUN. If you don’t want to do full copy & copy the only changed tracks. symclone has that options too.

Differential Copy :

1.Create a first session with -diff option

#symclone –f <devices file> create  -copy/-precopy –diff –sid <symm id> –nop

Note : This session will be full copy too. only thing is subsequent copies can be created with recreate option & they’ll be just differential and fast.

2.   Activate the session.

#symclone -f <devices file> activate -sid <symm id>consistent  -nop

3. for subsequent point in time clone creations. you don’t(shouldn’t) need to terminate the session. instead Recreate the same session using below and activate them using activate.

#symclone -f <devices file> recreate -sid <symm id> –nop

4. Activate it for start copying blocks. This should be fast.

#symclone -f <devices file> activate -sid <symm id>consistent  -nop


Restoring or Reverse Cloning :

Incase if something happens to source LUNS. if you want to restore from target. you could do using following commands. you don’t have to flip the devices. you could use same device file.

#symclone -f <devices file> restore -sid <symm id> –nop

you don’t need to activate as restore will establish in opposite directions and automatically activates them to. when you query you should see the status as “Restore in Progress”. once restore is complete you would see “Restored”.

If you want to normal clone from source to target again after restore is complete. you need to Split the existing reverse clone session by using below command.

#symclone -f <devices file> split -sid <symm id> –nop

Now. you could go-ahead with recreate and activate for normal clones.


Format of the device file :


   1300              1500


Reference Docs :

November 10, 2009 Posted by | EMC Symclone | , | 11 Comments

‘ORA-27102: out of memory’ Error while starting oracle as root in Solaris 10

Did it ever happened to you that you could start oracle DB as oracle but not root and keep throwing error saying out of memory. In Solaris 10, first thing you want to do is find out which project it belongs to using id -p. Once you identify that, bump up the value to required using prctl/projmod. In my case root belongs to project called user.root which has 30GB shared mem and it still not able to start the instance which has SHM size more than 8 GB. My box has 32 GB RAM. If we reduce the shm size to below 8 GB. it’s able to start. later on I found that. oracle binaries have setuid set and when you try to start them as root. by default they are going to project called default ( neither it’s taking nor user.root project. Don’t know why though). By default any project can have max-shared memory segment size 1/4 of physical memory. I’ve to bump up the value to required using.

prctl –n project.max-shm-mem –k –r –v 30G –i project default

projmod –s –K ‘project.max-shm-memory=(priv,30g,deny) default
BTW how did I find out it’s using default project ?
I reduced the shm value to below 8 GB and started the db. then I issued
ps –eo user,pid,args,project |grep –i pmon


Warning: while modifying default project you’ve to make sure you consider settings for other apps/instances also which runs on the same machine. By default all users are assigned to default project

November 4, 2009 Posted by | Solaris | | Leave a comment

Enabling XDMCP in Solaris 10

Solaris 10 is secure by default and starts services on localhost instead of system name or all hosts(*) . Due to that if you try to connect to server over xdmcp using exceed on demand or something you won’t get connection. For that we need to use following method to fix it.

if you issue ps on dtlogin. It might come up like this and listens on udp port 0 instead of 177.

#ps –ef |grep –i dtlogin
/usr/dt/bin/dtlogin -daemon -udpPort 0

You can change the above behaviour by using svccfg and change the properties.

#svccfg -s cde-login setprop 'dtlogin/args=""'
#svcadm restart cde-login # or #svcadm enable -r cde-login

After these system should respond to xdmcp broadcast queries.  Also if you want to disable the default security option you could give a try with netservices { open | limited }

October 7, 2009 Posted by | Solaris | | 4 Comments

Enabling Jumbo Frames in Solaris

Yesterday while setting up oracle RAC cluster we’ve to enable Jumbo Frames which has many advantages while transferring the data through interconnect. RAC advantages are out of scope in this blog. Below are the steps I followed for enabling Jumbo Frames ( MTU size more than 1500 bytes) in Sun E2900 which has CE interfaces.


1. Get the h/w addresses for CE interfaces.

# grep -i ce /etc/path_to_inst
"/ssm@0,0/pci@18,700000/network@1" 0 "ce"
"/ssm@0,0/pci@18,700000/network@3" 1 "ce"
"/ssm@0,0/pci@19,700000/pci@1/network@0" 2 "ce"
"/ssm@0,0/pci@19,700000/pci@1/network@1" 3 "ce"
"/ssm@0,0/pci@19,700000/pci@1/network@2" 4 "ce"
"/ssm@0,0/pci@19,700000/pci@1/network@3" 5 "ce"


2. Make a ce.conf file in /kernel/drv/ce.conf with below format

name="ce" parent=" " unit-address=" " accept-jumbo=1;

parent : It’s the device path with out network@

ex: /ssm@0,0/pci@18,700000 & /ssm@0,0/pci@19,700000/pci@1

unit-address : It’s the number immediately after network@ .

For example "/ssm@0,0/pci@18,700000/network@1" 0 "ce" device entry unit-address is 1   which immediately after network@. Note: it’s not 0 which is prior to ce. this has been mentioned in sun docs clearly.

accept-jumbo=1 ; For enabling jumbo frames on particular interface.

In my case I want to enable Jumbo frames for CE 1 & CE3 which are used for cluster interconnect. so my ce.conf looks like this.

#cat /kernel/drv/ce.conf
name="ce" parent="/ssm@0,0/pci@18,700000" unit-address="3" accept-jumbo=1; # For CE 0
name="ce" parent="/ssm@0,0/pci@19,700000/pci@1" unit-address="1" accept-jumbo=1; # For CE 1


3. Change the mtu in /etc/hostname.ce1 /etc/hostname.ce3 like below.

# cat /etc/hostname.ce1
systemname mtu 9194 group ipmp-private up
# cat /etc/hostname.ce3
mtu 9194 group ipmp-private up

4. Re-configure reboot the box and check the mtu for ce1 & ce3

  ifconfig –a |grep- i mtu

5. If MTU is configured correctly you can check the how many jumbo frames are passed through interface using

kstat –p ce | grep –i jumbo

* Above all steps makes sure that jumbo frames persistent across reboots ce1 & ce3 .

If you want to enable just temporarily you can do that using ndd commands.

#ifconfig ce1 unplumb
#ifconfig ce3 unplumb
# #Above 2 steps are imporant otherwise Jumbo Frames won’t be enable while interface is active.
#ndd -set /dev/ce instance 1
#ndd -set /dev/ce accept-jumbo 1
#ndd -set /dev/ce instance 3
#ndd -set /dev/ce accept-jumbo 1
#ifconfig ce1 plumb mtu 9194
#ifconfig ce3 plumb mtu 9194

September 28, 2009 Posted by | Solaris | | Leave a comment